運(yùn)營(yíng)商網(wǎng)絡(luò)中基于深度強(qiáng)化學(xué)習(xí)的服務(wù)功能鏈遷移機(jī)制

陳卓; 馮鋼; 何穎; 周楊

doi:10.11999/JEIT190545

運(yùn)營(yíng)商網(wǎng)絡(luò)中基于深度強(qiáng)化學(xué)習(xí)的服務(wù)功能鏈遷移機(jī)制

doi: 10.11999/JEIT190545

陳卓^{1, 2, ,},
馮鋼²,
何穎²,
周楊³

1.
重慶理工大學(xué)計(jì)算機(jī)科學(xué)與工程學(xué)院重慶 200433
2.
電子科技大學(xué)通信抗干擾技術(shù)國(guó)家級(jí)重點(diǎn)實(shí)驗(yàn)室成都 710077
3.
奧本大學(xué)計(jì)算機(jī)科學(xué)與軟件工程學(xué)院美國(guó)阿拉巴馬州奧本市 36849

基金項(xiàng)目: 國(guó)家自然科學(xué)基金(61471089, 61401076)

詳細(xì)信息

作者簡(jiǎn)介:
陳卓：男，1980年生，副教授，博士，碩士生導(dǎo)師，研究方向?yàn)榫W(wǎng)絡(luò)虛擬化和軟件定義網(wǎng)絡(luò)

馮鋼：男，1964年生，教授，博士生導(dǎo)師，研究方向?yàn)橛?jì)算機(jī)網(wǎng)絡(luò)和無(wú)線接入網(wǎng)技術(shù)

何穎：女，1993年生，碩士生，研究方向網(wǎng)絡(luò)虛擬化技術(shù)

通訊作者:
陳卓　chenzhuo@cqut.edu.cn

中圖分類號(hào): TN915; TP393
計(jì)量
- 文章訪問(wèn)數(shù): 1732
- HTML全文瀏覽量: 829
- PDF下載量: 107
- 被引次數(shù): 0
出版歷程
- 收稿日期: 2019-07-18
- 修回日期: 2020-06-14
- 網(wǎng)絡(luò)出版日期: 2020-07-14
- 刊出日期: 2020-09-27

Deep Reinforcement Learning Based Migration Mechanism for Service Function Chain in Operator Networks

Zhuo CHEN^{1, 2
, ,},
Gang FENG²,
Ying HE²,
Yang ZHOU³

1.
College of Computer Science and Engineering, Chongqing University of Technology,Chongqing 200433, China
2.
National Key Laboratory of Science and Technology on Communications, University of Electronic Science and Technology of China, Chengdu 710077, China
3.
Department of Computer Science and Software Engineering, Auburn University, Auburn 36849, United States of America

Funds: The National Natural Science Foundation of China (61471089, 61401076)

摘要

摘要: 為改善運(yùn)營(yíng)商網(wǎng)絡(luò)提供的移動(dòng)服務(wù)體驗(yàn)，該文研究服務(wù)功能鏈(SFC)的在線遷移問(wèn)題。首先基于馬爾可夫決策過(guò)程(MDP)對(duì)服務(wù)功能鏈中的多個(gè)虛擬網(wǎng)絡(luò)功能(VNF)在運(yùn)營(yíng)商網(wǎng)絡(luò)中的駐留位置遷移進(jìn)行模型化分析。通過(guò)將強(qiáng)化學(xué)習(xí)和深度神經(jīng)網(wǎng)絡(luò)相結(jié)合提出一種基于雙深度Q網(wǎng)絡(luò)(double DQN)的服務(wù)功能鏈遷移機(jī)制，該遷移方法能在連續(xù)時(shí)間下進(jìn)行服務(wù)功能鏈的在線遷移決策并避免求解過(guò)程中的過(guò)度估計(jì)。實(shí)驗(yàn)結(jié)果表明，該文所提出的策略相比于固定部署算法和貪心算法在端到端時(shí)延和網(wǎng)絡(luò)系統(tǒng)收益等方面優(yōu)勢(shì)明顯，有助于運(yùn)營(yíng)商改善服務(wù)體驗(yàn)和資源的使用效率。
- 運(yùn)營(yíng)商網(wǎng)絡(luò) /
- 遷移機(jī)制 /
- 深度強(qiáng)化學(xué)習(xí) /
- 服務(wù)功能鏈
Abstract: To improve the service experience provided by the operator network, this paper studies the online migration of Service Function Chain(SFC). Based on the Markov Decision Process(MDP), modeling analysis is performed on the migration of multiple Virtual Network Functions(VNF) in SFC. By combining reinforcement learning and deep neural networks, a double Deep Q-Network(double DQN) based service function chain migration mechanism is proposed. This method can make online migration decisions and avoid over-estimation. Experimental result shows that when compared with the fixed deployment algorithm and the greedy algorithm, the double DQN based SFC migration mechanism has obvious advantages in end-to-end delay and network system revenue, which can help the mobile operator to improve the quality of experience and the efficiency of resources usage.
- Operator network /
- Migration mechanism /
- Deep Reinforcement Learning (DRL) /
- Service Function Chain (SFC)

HTML全文

圖 1 基于神經(jīng)網(wǎng)絡(luò)的強(qiáng)化學(xué)習(xí)決策

下載: 全尺寸圖片幻燈片

圖 2 基于雙DQN的SFC遷移方法流程圖

下載: 全尺寸圖片幻燈片

圖 3 移動(dòng)業(yè)務(wù)端到端時(shí)延對(duì)比

下載: 全尺寸圖片幻燈片

圖 4 基于雙DQN方法在不同長(zhǎng)度VNF下的延遲對(duì)比

下載: 全尺寸圖片幻燈片

圖 5 不同移動(dòng)次數(shù)下系統(tǒng)收益對(duì)比

下載: 全尺寸圖片幻燈片

圖 6 系統(tǒng)累積收益對(duì)比

下載: 全尺寸圖片幻燈片

表 1 基于雙DQN的SFC遷移算法的偽碼

輸入：運(yùn)營(yíng)商網(wǎng)絡(luò)拓?fù)?G = \left( {N,E} \right)$，服務(wù)功能鏈集合C，網(wǎng)絡(luò)功　　　能集合F;
輸出：SFC遷移策略;
步驟1：初始化隨機(jī)權(quán)重為$\psi $的神經(jīng)網(wǎng)絡(luò);
步驟2：初始化動(dòng)作值函數(shù)Q;
步驟3：初始化經(jīng)驗(yàn)池(experience replay)存儲(chǔ)器N;
步驟4：for episode = 1, 2, ···, M do,
觀察初始狀態(tài)${s^0}$,
for t = 0, 1, ···, N–1 do,
以概率為$\varepsilon $選擇一個(gè)隨機(jī)動(dòng)作${a^t}$,
否則選擇動(dòng)作${a^t} = \arg \max Q({s^t},a;{\psi ^t})$;
在仿真器中執(zhí)行動(dòng)作${a^t}$，并觀察回報(bào)${R_{t + 1}}$和新?tīng)顟B(tài)${s_{t + 1}}$,
存儲(chǔ)中間量$ < {s^t},{a^t},{r^t},{s^{t + 1}} > $到經(jīng)驗(yàn)池存儲(chǔ)器N中,
從經(jīng)驗(yàn)池存儲(chǔ)器N中獲取一組樣本,
計(jì)算損失函數(shù)$L({\psi ^t})$,
計(jì)算關(guān)于${\psi ^t}$的損失函數(shù)的梯度,
更新${\psi ^t} \leftarrow {\psi ^t} - \phi {{\text{?}} _{ {\psi ^t} } }L({\psi ^t})$，其中$\phi $為學(xué)習(xí)率;
end
end

下載: 導(dǎo)出CSV

參考文獻(xiàn)(16)

CHATRAS B and OZOG F F. Network functions virtualization: The portability challenge[J]. IEEE Network, 2016, 30(4): 4–8. doi: 10.1109/MNET.2016.7513857

ZHANG Qixia, LIU Fangming, and ZENG Chaobing. Adaptive interference-aware VNF placement for service-customized 5G network slices[C]. IEEE Conference on Computer Communications, Paris, France, 2019: 2449–2457. doi: 10.1109/INFOCOM.2019.8737660.

AGARWAL S, MALANDRINO F, CHIASSERINI C F, et al. Joint VNF placement and CPU allocation in 5G[C]. IEEE Conference on Computer Communications, Honolulu, USA, 2018: 1943–1951. doi: 10.1109/INFOCOM.2018.8485943.

KUO T W, LIOU B H, LIN K C J, et al. Deploying chains of virtual network functions: On the relation between link and server usage[C]. The 35th Annual IEEE International Conference on Computer Communications, San Francisco, USA, 2016: 1–9. doi: 10.1109/INFOCOM.2016.7524565.

TALEB T, KSENTINI A, and FRANGOUDIS P A. Follow-me cloud: When cloud services follow mobile users[J]. IEEE Transactions on Cloud Computing, 2019, 7(2): 369–382. doi: 10.1109/TCC.2016.2525987

ERAMO V, MIUCCI E, AMMAR M, et al. An approach for service function chain routing and virtual function network instance migration in network function virtualization architectures[J]. IEEE/ACM Transactions on Networking, 2017, 25(4): 2008–2025. doi: 10.1109/TNET.2017.2668470

HOUIDI O, SOUALAH O, LOUATI W, et al. An efficient algorithm for virtual network function scaling[C]. 2017 IEEE Global Communications Conference, Singapore, 2017: 1–7. doi: 10.1109/GLOCOM.2017.8254727.

CHO D, TAHERI J, ZOMAYA A Y, et al. Real-time Virtual Network Function (VNF) migration toward low network latency in cloud environments[C]. The 10th IEEE International Conference on Cloud Computing, Honolulu, USA, 2017: 798–801. doi: 10.1109/CLOUD.2017.118.

蘭巨龍, 于倡和, 胡宇翔, 等. 基于深度增強(qiáng)學(xué)習(xí)的軟件定義網(wǎng)絡(luò)路由優(yōu)化機(jī)制[J]. 電子與信息學(xué)報(bào), 2019, 41(11): 2669–2674. doi: 10.11999/JEIT180870

LAN Julong, YU Changhe, HU Yuxiang, et al. A SDN routing optimization mechanism based on deep reinforcement learning[J]. Journal of Electronics &Information Technology, 2019, 41(11): 2669–2674. doi: 10.11999/JEIT180870

HUANG Xiaohong, YUAN Tingting, QIAO Guanhua, et al. Deep reinforcement learning for multimedia traffic control in software defined networking[J]. IEEE Network, 2018, 32(6): 35–41. doi: 10.1109/MNET.2018.1800097

LEE J W, MAZUMDAR R R, and SHROFF N B. Non-Convex optimization and rate control for multi-class services in the Internet[J]. IEEE/ACM Transactions on Networking, 2005, 13(4): 827–840. doi: 10.1109/TNET.2005.852876

李晨溪, 曹雷, 陳希亮, 等. 基于云推理模型的深度強(qiáng)化學(xué)習(xí)探索策略研究[J]. 電子與信息學(xué)報(bào), 2018, 40(1): 244–248. doi: 10.11999/JEIT170347

LI Chenxi, CAO Lei, CHEN Xiliang, et al. Cloud reasoning model-based exploration for deep reinforcement learning[J]. Journal of Electronics &Information Technology, 2018, 40(1): 244–248. doi: 10.11999/JEIT170347

MNIH V, KAVUKCUOGLU K, SILVER D, et al. Human-level control through deep reinforcement learning[J]. Nature, 2015, 518(7540): 529–533. doi: 10.1038/nature14236

GHAZNAVI M, KHAN A, SHAHRIAR N, et al. Elastic virtual network function placement[C]. The 4th IEEE International Conference on Cloud Networking, Niagara Falls, Canada, 2015: 255–260. doi: 10.1109/CloudNet.2015.7335318.

SUGISONO K, FUKUOKA A, and YAMAZAKI H. Migration for VNF instances forming service chain[C]. The 7th IEEE International Conference on Cloud Networking, Tokyo, Japan, 2018: 1–3. doi: 10.1109/CloudNet.2018.8549194.

LIN Tachun, ZHOU Zhili, TORNATORE M, et al. Demand-aware network function placement[J]. Journal of Lightwave Technology, 2016, 34(11): 2590–2600. doi: 10.1109/JLT.2016.2535401

相關(guān)文章

施引文獻(xiàn)

資源附件(0)

訪問(wèn)統(tǒng)計(jì)