面向三維高效視頻編碼的深度圖錯誤隱藏

周洋; 吳佳憶; 陸宇; 殷海兵

doi:10.11999/JEIT180926

面向三維高效視頻編碼的深度圖錯誤隱藏

doi: 10.11999/JEIT180926

杭州電子科技大學(xué)通信工程學(xué)院杭州 310018

基金項目: 浙江省自然科學(xué)基金(LY17F020027)，國家自然科學(xué)基金(61401132, 61572449)

詳細信息

作者簡介:
周洋：男，1979年生，副教授，碩士生導(dǎo)師，研究方向為視頻分析和三維視頻編解碼

吳佳憶：男，1994年生，碩士生，研究方向為視頻差錯控制技術(shù)

陸宇：男，1977年生，講師，研究方向為視頻壓縮技術(shù)

殷海兵：男，1974年生，教授，研究方向為視頻編碼與壓縮芯片設(shè)計

通訊作者:
周洋　zhouyang_hz@126.com

中圖分類號: TP391
計量
- 文章訪問數(shù): 2206
- HTML全文瀏覽量: 1025
- PDF下載量: 43
- 被引次數(shù): 0
出版歷程
- 收稿日期: 2018-09-30
- 修回日期: 2019-04-05
- 網(wǎng)絡(luò)出版日期: 2019-05-21
- 刊出日期: 2019-11-01

Depth Map Error Concealment for 3D High Efficiency Video Coding

Institute of Telecommunication Engineering, Hangzhou Dianzi University, Hangzhou 310018, China

Funds: Zhejiang Province Natural Science Fundation (LY17F020027), The National Natural Science Foundation of China (61401132, 61572449)

摘要

摘要: 基于多視點視頻序列視點內(nèi)、視點間存在的相關(guān)性，并結(jié)合視點間運動矢量共享技術(shù)，該文提出一種面向3維高效視頻編碼中深度序列傳輸丟包的錯誤隱藏算法。首先，根據(jù)3D高效視頻編碼(3D-HEVC)的分層B幀預(yù)測(HBP)結(jié)構(gòu)和深度圖紋理特征，將深度圖丟失塊分成運動塊和靜止塊；然后，對于受損運動塊，使用結(jié)合紋理結(jié)構(gòu)的外邊界匹配準則來選擇相對最優(yōu)的運動/視差矢量進行基于位移矢量補償?shù)腻e誤掩蓋，而對受損靜止塊采用參考幀直接拷貝進行快速錯誤隱藏；最后，使用參考幀拆分重組來獲取新的運動/視差補償塊對修復(fù)質(zhì)量較差的重建塊進行質(zhì)量提升。實驗結(jié)果表明：相較于近年提出的對比算法，該文算法隱藏后的深度幀平均峰值信噪比(PSNR)能提升0.25～2.03 dB，結(jié)構(gòu)相似度測量值(SSIM)能提升0.001～0.006，且修復(fù)區(qū)域的主觀視覺質(zhì)量與原始深度圖更接近。
- 3維高效視頻編碼 /
- 多視視頻加深度 /
- 匹配準則 /
- 視差矢量 /
- 錯誤隱藏
Abstract: By using the intra-view and inter-view correlations and the motion vector-sharing, a depth map error concealment approach is proposed for 3D video coding based on the High Efficiency Video Coding (3D-HEVC) to combat the packet loss of the depth video transmission. Based on the Hierarchical B-frame Prediction (HBP) structure in 3D-HEVC and textured features of the depth map, all the lost coding units are firstly categorized into two classes, i.e., motion blocks and static blocks. Then, according to the outer boundary matching criterion combining the texture structure, the optimal motion/disparity vector is chosen for the damaged motion blocks to conduct the motion/disparity compensation based error concealment. Whereas, the direct copy is applied to concealling the damaged static blocks quickly. Finally, for the concealed blocks whose qualities are not ideal, the new motion/disparity compensation blocks reconstructed by the reference frames recombination are applied to improning the qualities of those blocks. The experimental results show that the repaired depth map concealed by the proposed approach can achieve 0.25～2.03 dB gain in term of the Peak-Signal-to-Noise Ratio (PSNR) and 0.001～0.006 gain in term of Structural Similarity Index Measure(SSIM). Moreover, the subjective visual quality of the repaired area is better in lines with the original depth maps.
- 3D-High Efficiency Video Coding(3D-HEVC) /
- Multi-view video plus depth /
- Matching criterion /
- Disparity vector /
- Error concealment

HTML全文

圖 1 算法總體流程圖

下載: 全尺寸圖片幻燈片

圖 2 丟失塊周圍鄰塊位移矢量的選擇

下載: 全尺寸圖片幻燈片

圖 3 PSNR逐幀測試結(jié)果曲線圖

下載: 全尺寸圖片幻燈片

圖 4 Poznanstreet第3視點第3幀錯誤隱藏結(jié)果

下載: 全尺寸圖片幻燈片

圖 5 Kendo第3視點第3幀主觀測試結(jié)果

下載: 全尺寸圖片幻燈片

表 1 組合參考塊對應(yīng)表

參考列表	參考幀號	L0				L1
參考列表	參考幀號	R0₀	R0₁	…	R0_K	R1₀	R1₁	…	R1_K
L0	R0₀	P₀	–	–		B₀₀	B₀₁	B₀_i	B₀_K
	R0₁	–	P₁	–		B₁₀	B₁₁	B₁_i	B₁_K
	∶	–	–	P_i		B_i₀	B_i₁	B_ii	B_iK
	R0_K				P_K	B_K₀	B_K₁	B_Ki	B_KK
L1	R1₀	B₀₀	B₁₀	B_i₀	B_K₀	P_K₊₁	–	–
	R1₁	B₀₁	B₁₁	B_i₁	B_K₁	–	P_K₊₂	–
	∶	B₀_i	B₁_i	B_ii	B_Ki	–	–	P_K₊_i
	R1_K	B₀_K	B₁_K	B_iK	B_KK				P₂_K

下載: 導(dǎo)出CSV

表 2 QP=30, 25時各測試序列重建圖像平均PSNR計算結(jié)果(dB)

QP	序列	原始	錯誤率50%				錯誤率25%				錯誤率10%
QP	序列	原始	文獻[6]	文獻[7]	MVC	本文算法	文獻[6]	文獻[7]	MVC	本文算法	文獻[6]	文獻[7]	MVC	本文算法
30	Kendo	42.761	36.406	36.122	35.894	37.103	37.896	37.359	36.786	38.310	39.449	38.925	38.025	39.833
	Bookarrival	39.923	36.932	35.911	35.656	37.525	37.865	37.171	36.526	38.286	38.338	37.699	37.389	38.747
	Poznanstreet	44.028	39.593	38.834	36.662	40.864	41.155	41.028	40.799	41.890	41.450	41.219	40.983	42.171
	Balloon	40.574	38.713	38.512	37.691	39.269	39.514	39.456	38.347	39.839	39.706	39.638	38.485	39.958
25	Kendo	45.874	36.736	36.548	36.096	37.499	39.227	38.058	37.287	39.655	41.368	39.688	39.133	41.501
	Bookarrival	41.625	37.981	36.733	36.354	38.270	38.411	37.488	36.947	39.921	39.728	39.196	38.463	40.187
	Poznanstreet	46.122	39.955	38.928	37.066	41.561	41.723	41.178	40.802	42.319	42.113	41.797	41.207	42.622
	Balloon	44.022	40.829	39.434	39.156	41.247	41.899	41.277	40.894	42.149	42.234	42.158	41.084	42.612

下載: 導(dǎo)出CSV

表 3 QP=30, 25時各測試序列重建圖像平均SSIM計算結(jié)果

QP	序列	原始	錯誤率50%				錯誤率25%				錯誤率10%
QP	序列	原始	文獻[6]	文獻[7]	MVC	本文算法	文獻[6]	文獻[7]	MVC	本文算法	文獻[6]	文獻[7]	MVC	本文算法
30	Kendo	0.9867	0.9708	0.9688	0.9647	0.9721	0.9780	0.9776	0.9733	0.9785	0.9819	0.9816	0.9766	0.9822
	Bookarrival	0.9565	0.9482	0.9457	0.9422	0.9495	0.9503	0.9481	0.9452	0.9510	0.9525	0.9518	0.9477	0.9534
	Poznanstreet	0.9795	0.9732	0.9720	0.9714	0.9759	0.9746	0.9740	0.9738	0.9767	0.9751	0.9750	0.9744	0.9769
	Balloon	0.9767	0.9738	0.9733	0.9720	0.9743	0.9745	0.9742	0.9736	0.9751	0.9747	0.9746	0.9743	0.9757
25	Kendo	0.9874	0.9722	0.9696	0.9660	0.9751	0.9789	0.9783	0.9765	0.9796	0.9828	0.9824	0.9795	0.9832
	Bookarrival	0.9701	0.9577	0.9547	0.9539	0.9598	0.9634	0.9627	0.9613	0.9641	0.9643	0.9642	0.9633	0.9655
	Poznanstreet	0.9878	0.9818	0.9803	0.9796	0.9824	0.9825	0.9821	0.9816	0.9848	0.9843	0.9841	0.9824	0.9849
	Balloon	0.9855	0.9810	0.9803	0.9796	0.9821	0.9834	0.9833	0.9828	0.9839	0.9841	0.9836	0.9835	0.9844

下載: 導(dǎo)出CSV

表 4 各測試序列解碼運行時間(s)

序列	原解碼	文獻[6]	文獻[7]	MVC	本文
Kendo	188.71	231.45	198.83	192.07	194.19
Balloon	172.35	216.14	178.13	173.82	174.69
Bookarrival	175.86	219.35	183.19	177.18	179.13
Poznanstreet	483.67	558.15	498.68	484.68	486.07

下載: 導(dǎo)出CSV

參考文獻(20)

LI Yufeng and CHEN Ruining. Motion vector recovery for video error concealment based on plane fitting[J]. Computer Engineering &Applications, 2018, 54(2): 182–187. doi: 10.3778/j.issn.1002-8331.1608-0085

XU Dawen and WANG Rangding. Two-dimensional reversible data hiding-based approach for intra-frame error concealment in H.264/AVC[J]. Journal of Visual Communication& Image Representation, 2016, 47(2): 369–379. doi: 10.1016/j.jvcir.2013.12.008

ZABIHI S M, GHANEI Y H, and MEHRSHAD N. Adaptive temporal error concealment method based on the MB behavior estimation in the video[C]. The 7th International Conference on Computer and Knowledge Engineering. Mashhad, Iran, 2017: 193–198.

EL-SHAFAI W, EL-RABAIE S, EL-HALAWANY M M, et al. Recursive Bayesian filtering-based error concealment scheme for 3D video communication over severely lossy wireless channels[J]. Circuits Systems & Signal Processing, 2018(20): 1–32. doi: 10.1007/s00034-018-0786-8

周洋, 蔣剛毅, 郁梅, 等. 面向HBP編碼格式的立體視頻B幀整幀丟失分層錯誤隱藏算法[J]. 電子與信息學(xué)報, 2014, 36(2): 377–383. doi: 10.3724/SP.J.1146.2013.00528

ZHOU Yang, JIANG Gangyi, YU Mei, et al. A hierarchical error concealment algorithm for entire B-frame loss in stereoscopic video coding with hierarchical B pictures[J]. Journal of Electronics &Information Technology, 2014, 36(2): 377–383. doi: 10.3724/SP.J.1146.2013.00528

SONG K, CHUNG T, OH Y, et al. Error concealment of multi-view video sequences using inter-view and intra-view correlations[J]. Journal of Visual Communication & Image Representation, 2009, 20(4): 281–292. doi: 10.1016/j.jvcir.2009.02.002

PENG Yantsung and COSMAN P C. Weighted boundary matching error concealment for HEVC using block partition decisions[C]. The 48th Asilomar Conference on Signals, Systems & Computers, Grove, USA, 2014: 921–925. doi: 10.1109/ACSSC.2014.7094587.

KHATTAK S, MAUGEY T, HAMZAOUI R, et al. Temporal and inter-view consistent error concealment technique for multiview plus depth video[J]. IEEE Transactions on Circuits & Systems for Video Technology, 2016, 26(5): 829–840. doi: 10.1109/TCSVT.2015.2418631

LIU Shujie, CHEN Ying, WANG Yekui, et al. Frame loss error concealment for multiview video coding[C]. IEEE International Symposium on Circuits and Systems (ISCAS 2008), Washington, USA, 2008: 3470–3473.

ZHOU Yuan, XIANG Wei, and WANG Gengkun. Frame loss concealment for multiview video transmission over wireless multimedia sensor networks[J]. IEEE Sensors Journal, 2015, 15(3): 1892–1901. doi: 10.1109/JSEN.2014.2366511

HUANG Pincheng, LIN Jieru, LI Gwolong, et al. Improved depth-assisted error concealment algorithm for 3D video transmission[J]. IEEE Transactions on Multimedia, 2017, 19(11): 2625–2632. doi: 10.1109/TMM.2017.2694218

TAI Shenchuan, WANG Chuenching, HONG Chienshiang, et al. An efficient full frame algorithm for object-based error concealment in 3D depth-based video[J]. Multimedia Tools & Applications, 2016, 75(16): 9927–9947. doi: 10.1007/s11042-015-2899-4

時琳, 劉榮科, 李君燁. 基于深度信息的立體視頻錯誤隱藏方法[J]. 電子與信息學(xué)報, 2012, 34(7): 1678–1684. doi: 10.3724/SP.J.1146.2011.01107

SHI Ling, LIU Rongke, and LI Junye. Error concealment based on depth information for stereoscopic video coding[J]. Journal of Electronics &Information Technology, 2012, 34(7): 1678–1684. doi: 10.3724/SP.J.1146.2011.01107

YANG Meng, LAN Xuguang, ZHENG Nanning, et al. Depth-assisted temporal error concealment for intra frame slices in 3-D video[J]. IEEE Transactions on Broadcasting, 2014, 60(2): 385–393. doi: 10.1109/TBC.2014.2321676

YAN Bo and ZHOU Jie. Efficient frame concealment for depth image-based 3-D video transmission[J]. IEEE Transactions on Multimedia, 2012, 14(3): 936–941. doi: 10.1109/TMM.2012.2184743

MERKLE P, MULLER K, MARPE D, et al. Depth intra coding for 3D video based on geometric primitives[J]. IEEE Transactions on Circuits & Systems for Video Technology, 2016, 26(3): 570–582. doi: 10.1109/TCSVT.2015.2407791

HEWAGE C T E R, WORRALL S, DOGAN S, et al. Frame concealment algorithm for stereoscopic video using motion vector sharing[C]. 2008 IEEE International Conference on Multimedia and Expo, Hannover, Germany, 2008: 485–488. doi: 10.1109/ICME.2008.4607477.

ASSUNCAO P, MARCELINO S, SOARES S, et al. Spatial error concealment for intra-coded depth maps in multiview video-plus-depth[J]. Multimedia Tools &Applications, 2017, 76(12): 13835–13858. doi: 10.1007/s11042-016-3766-7

HTM Reference Software 14.0. http://hevc.kw.bbc.co.uk/git/w/jctvc-3de.git, 2016.

WANG Zhou, BOVIK A C, SHEIKH H R, et al. Image quality assessment: From error visibility to structural similarity[J]. IEEE Transactions on Image Processing, 2004, 13(4): 600–612. doi: 10.3969/j.issn.1006-8961.2006.12.00.003

相關(guān)文章

施引文獻

資源附件(0)

訪問統(tǒng)計