運(yùn)營(yíng)商網(wǎng)絡(luò)中基于深度強(qiáng)化學(xué)習(xí)的服務(wù)功能鏈遷移機(jī)制
doi: 10.11999/JEIT190545
-
1.
重慶理工大學(xué)計(jì)算機(jī)科學(xué)與工程學(xué)院 重慶 200433
-
2.
電子科技大學(xué)通信抗干擾技術(shù)國(guó)家級(jí)重點(diǎn)實(shí)驗(yàn)室 成都 710077
-
3.
奧本大學(xué)計(jì)算機(jī)科學(xué)與軟件工程學(xué)院 美國(guó)阿拉巴馬州 奧本市 36849
Deep Reinforcement Learning Based Migration Mechanism for Service Function Chain in Operator Networks
-
1.
College of Computer Science and Engineering, Chongqing University of Technology,Chongqing 200433, China
-
2.
National Key Laboratory of Science and Technology on Communications, University of Electronic Science and Technology of China, Chengdu 710077, China
-
3.
Department of Computer Science and Software Engineering, Auburn University, Auburn 36849, United States of America
-
摘要: 為改善運(yùn)營(yíng)商網(wǎng)絡(luò)提供的移動(dòng)服務(wù)體驗(yàn),該文研究服務(wù)功能鏈(SFC)的在線遷移問(wèn)題。首先基于馬爾可夫決策過(guò)程(MDP)對(duì)服務(wù)功能鏈中的多個(gè)虛擬網(wǎng)絡(luò)功能(VNF)在運(yùn)營(yíng)商網(wǎng)絡(luò)中的駐留位置遷移進(jìn)行模型化分析。通過(guò)將強(qiáng)化學(xué)習(xí)和深度神經(jīng)網(wǎng)絡(luò)相結(jié)合提出一種基于雙深度Q網(wǎng)絡(luò)(double DQN)的服務(wù)功能鏈遷移機(jī)制,該遷移方法能在連續(xù)時(shí)間下進(jìn)行服務(wù)功能鏈的在線遷移決策并避免求解過(guò)程中的過(guò)度估計(jì)。實(shí)驗(yàn)結(jié)果表明,該文所提出的策略相比于固定部署算法和貪心算法在端到端時(shí)延和網(wǎng)絡(luò)系統(tǒng)收益等方面優(yōu)勢(shì)明顯,有助于運(yùn)營(yíng)商改善服務(wù)體驗(yàn)和資源的使用效率。
-
關(guān)鍵詞:
- 運(yùn)營(yíng)商網(wǎng)絡(luò) /
- 遷移機(jī)制 /
- 深度強(qiáng)化學(xué)習(xí) /
- 服務(wù)功能鏈
Abstract: To improve the service experience provided by the operator network, this paper studies the online migration of Service Function Chain(SFC). Based on the Markov Decision Process(MDP), modeling analysis is performed on the migration of multiple Virtual Network Functions(VNF) in SFC. By combining reinforcement learning and deep neural networks, a double Deep Q-Network(double DQN) based service function chain migration mechanism is proposed. This method can make online migration decisions and avoid over-estimation. Experimental result shows that when compared with the fixed deployment algorithm and the greedy algorithm, the double DQN based SFC migration mechanism has obvious advantages in end-to-end delay and network system revenue, which can help the mobile operator to improve the quality of experience and the efficiency of resources usage. -
表 1 基于雙DQN的SFC遷移算法的偽碼
輸入:運(yùn)營(yíng)商網(wǎng)絡(luò)拓?fù)?G = \left( {N,E} \right)$,服務(wù)功能鏈集合C,網(wǎng)絡(luò)功
能集合F;輸出:SFC遷移策略; 步驟1:初始化隨機(jī)權(quán)重為$\psi $的神經(jīng)網(wǎng)絡(luò); 步驟2:初始化動(dòng)作值函數(shù)Q; 步驟3:初始化經(jīng)驗(yàn)池(experience replay)存儲(chǔ)器N; 步驟4:for episode = 1, 2, ···, M do, 觀察初始狀態(tài)${s^0}$, for t = 0, 1, ···, N–1 do, 以概率為$\varepsilon $選擇一個(gè)隨機(jī)動(dòng)作${a^t}$, 否則選擇動(dòng)作${a^t} = \arg \max Q({s^t},a;{\psi ^t})$; 在仿真器中執(zhí)行動(dòng)作${a^t}$,并觀察回報(bào)${R_{t + 1}}$和新?tīng)顟B(tài)${s_{t + 1}}$, 存儲(chǔ)中間量$ < {s^t},{a^t},{r^t},{s^{t + 1}} > $到經(jīng)驗(yàn)池存儲(chǔ)器N中, 從經(jīng)驗(yàn)池存儲(chǔ)器N中獲取一組樣本, 計(jì)算損失函數(shù)$L({\psi ^t})$, 計(jì)算關(guān)于${\psi ^t}$的損失函數(shù)的梯度, 更新${\psi ^t} \leftarrow {\psi ^t} - \phi {{\text{?}} _{ {\psi ^t} } }L({\psi ^t})$,其中$\phi $為學(xué)習(xí)率; end end 下載: 導(dǎo)出CSV
-
CHATRAS B and OZOG F F. Network functions virtualization: The portability challenge[J]. IEEE Network, 2016, 30(4): 4–8. doi: 10.1109/MNET.2016.7513857 ZHANG Qixia, LIU Fangming, and ZENG Chaobing. Adaptive interference-aware VNF placement for service-customized 5G network slices[C]. IEEE Conference on Computer Communications, Paris, France, 2019: 2449–2457. doi: 10.1109/INFOCOM.2019.8737660. AGARWAL S, MALANDRINO F, CHIASSERINI C F, et al. Joint VNF placement and CPU allocation in 5G[C]. IEEE Conference on Computer Communications, Honolulu, USA, 2018: 1943–1951. doi: 10.1109/INFOCOM.2018.8485943. KUO T W, LIOU B H, LIN K C J, et al. Deploying chains of virtual network functions: On the relation between link and server usage[C]. The 35th Annual IEEE International Conference on Computer Communications, San Francisco, USA, 2016: 1–9. doi: 10.1109/INFOCOM.2016.7524565. TALEB T, KSENTINI A, and FRANGOUDIS P A. Follow-me cloud: When cloud services follow mobile users[J]. IEEE Transactions on Cloud Computing, 2019, 7(2): 369–382. doi: 10.1109/TCC.2016.2525987 ERAMO V, MIUCCI E, AMMAR M, et al. An approach for service function chain routing and virtual function network instance migration in network function virtualization architectures[J]. IEEE/ACM Transactions on Networking, 2017, 25(4): 2008–2025. doi: 10.1109/TNET.2017.2668470 HOUIDI O, SOUALAH O, LOUATI W, et al. An efficient algorithm for virtual network function scaling[C]. 2017 IEEE Global Communications Conference, Singapore, 2017: 1–7. doi: 10.1109/GLOCOM.2017.8254727. CHO D, TAHERI J, ZOMAYA A Y, et al. Real-time Virtual Network Function (VNF) migration toward low network latency in cloud environments[C]. The 10th IEEE International Conference on Cloud Computing, Honolulu, USA, 2017: 798–801. doi: 10.1109/CLOUD.2017.118. 蘭巨龍, 于倡和, 胡宇翔, 等. 基于深度增強(qiáng)學(xué)習(xí)的軟件定義網(wǎng)絡(luò)路由優(yōu)化機(jī)制[J]. 電子與信息學(xué)報(bào), 2019, 41(11): 2669–2674. doi: 10.11999/JEIT180870LAN Julong, YU Changhe, HU Yuxiang, et al. A SDN routing optimization mechanism based on deep reinforcement learning[J]. Journal of Electronics &Information Technology, 2019, 41(11): 2669–2674. doi: 10.11999/JEIT180870 HUANG Xiaohong, YUAN Tingting, QIAO Guanhua, et al. Deep reinforcement learning for multimedia traffic control in software defined networking[J]. IEEE Network, 2018, 32(6): 35–41. doi: 10.1109/MNET.2018.1800097 LEE J W, MAZUMDAR R R, and SHROFF N B. Non-Convex optimization and rate control for multi-class services in the Internet[J]. IEEE/ACM Transactions on Networking, 2005, 13(4): 827–840. doi: 10.1109/TNET.2005.852876 李晨溪, 曹雷, 陳希亮, 等. 基于云推理模型的深度強(qiáng)化學(xué)習(xí)探索策略研究[J]. 電子與信息學(xué)報(bào), 2018, 40(1): 244–248. doi: 10.11999/JEIT170347LI Chenxi, CAO Lei, CHEN Xiliang, et al. Cloud reasoning model-based exploration for deep reinforcement learning[J]. Journal of Electronics &Information Technology, 2018, 40(1): 244–248. doi: 10.11999/JEIT170347 MNIH V, KAVUKCUOGLU K, SILVER D, et al. Human-level control through deep reinforcement learning[J]. Nature, 2015, 518(7540): 529–533. doi: 10.1038/nature14236 GHAZNAVI M, KHAN A, SHAHRIAR N, et al. Elastic virtual network function placement[C]. The 4th IEEE International Conference on Cloud Networking, Niagara Falls, Canada, 2015: 255–260. doi: 10.1109/CloudNet.2015.7335318. SUGISONO K, FUKUOKA A, and YAMAZAKI H. Migration for VNF instances forming service chain[C]. The 7th IEEE International Conference on Cloud Networking, Tokyo, Japan, 2018: 1–3. doi: 10.1109/CloudNet.2018.8549194. LIN Tachun, ZHOU Zhili, TORNATORE M, et al. Demand-aware network function placement[J]. Journal of Lightwave Technology, 2016, 34(11): 2590–2600. doi: 10.1109/JLT.2016.2535401 -