K插值單純形法核極限學(xué)習(xí)機(jī)的研究

蘇一丹; 李若愚; 覃華; 陳琴

doi:10.11999/JEIT171104

K插值單純形法核極限學(xué)習(xí)機(jī)的研究

doi: 10.11999/JEIT171104

廣西大學(xué)計(jì)算機(jī)與電子信息學(xué)院南寧 530004

基金項(xiàng)目: 國家自然科學(xué)基金(61762009)

詳細(xì)信息

作者簡介:
蘇一丹：男，1962年生，教授，研究方向?yàn)樽匀挥?jì)算、數(shù)據(jù)挖掘

李若愚：男，1989年生，碩士生，研究方向?yàn)橹悄軆?yōu)化及在數(shù)據(jù)挖掘中的應(yīng)用

覃華：男，1972年生，教授，研究方向?yàn)榱孔佑?jì)算理論與近似動(dòng)態(tài)規(guī)劃最優(yōu)化方法、數(shù)據(jù)挖掘

陳琴：女，1974年生，副教授，研究方向?yàn)閿?shù)據(jù)挖掘

通訊作者:
覃華 1694520545@qq.com

中圖分類號(hào): TP181
計(jì)量
- 文章訪問數(shù): 1745
- HTML全文瀏覽量: 631
- PDF下載量: 45
- 被引次數(shù): 0
出版歷程
- 收稿日期: 2017-11-24
- 修回日期: 2018-04-23
- 網(wǎng)絡(luò)出版日期: 2018-06-07
- 刊出日期: 2018-08-01

Kernel Extreme Learning Machine Based on K Interpolation Simplex Method

College of Computer and Electronic Information, Guangxi University, Nanning 530004, China

Funds: The National Natural Science Foundation of China (61762009)

摘要

摘要: 針對(duì)核極限學(xué)習(xí)機(jī)高斯核函數(shù)參數(shù)選優(yōu)難，影響學(xué)習(xí)機(jī)訓(xùn)練收斂速度和分類精度的問題，該文提出一種K插值單純形法的核極限學(xué)習(xí)機(jī)算法。把核極限學(xué)習(xí)機(jī)的訓(xùn)練看作一個(gè)無約束優(yōu)化問題，在訓(xùn)練迭代過程中，用Nelder-Mead單純形法搜索高斯核函數(shù)的最優(yōu)核參數(shù)，提高所提算法的分類精度。引入K插值為Nelder-Mead單純形法提供合適的初值，減少單純形法的迭代次數(shù)，提高了新算法的訓(xùn)練收斂效率。通過在UCI數(shù)據(jù)集上的仿真實(shí)驗(yàn)并與其它算法比較，新算法具有更快的收斂速度和更高的分類精度。
- 核極限學(xué)習(xí)機(jī) /
- 核參數(shù) /
- Nelder-Mead單純形法 /
- K插值法
Abstract: The kernel Extreme Learning Machine (ELM) has a problem that the kernel parameter of the Gauss kernel function is hard to be optimized. As a result, training speed and classification accuracy of kernel ELM are negatively affected. To deal with that problem, a novel kernel ELM based on K interpolation simplex method is proposed. The training process of kernel ELM is considered as an unconstrained optimal problem. Then, the Nelder-Mead Simplex Method (NMSM) is used as an optimal method to search the optimized kernel parameter, which improves the classification accuracy of kernel ELM. Furthermore, the K interpolation method is used to provide appropriate initial values for the Nelder-Mead simplex to reduce the number of iterations, and as a result, the training speed of ELM is improved. Comparative results on UCI dataset demonstrate that the novel ELM algorithm has better training speed and higher classification accuracy.
- Kernel Extreme Learning Machine (KELM) /
- Kernel parameter /
- Nelder-Mead Simplex Method (NMSM) /
- K Interpolation method

HTML全文

圖 1 高斯核參數(shù)取值對(duì)訓(xùn)練精度和測(cè)試精度的影響

下載: 全尺寸圖片幻燈片

圖 2 K插值法對(duì)單純形核ELM訓(xùn)練收斂的影響

下載: 全尺寸圖片幻燈片

圖 3 本文算法的時(shí)間復(fù)雜度特征曲線

下載: 全尺寸圖片幻燈片

表 1 1維單純形子算法

輸入：初始搜索點(diǎn) \small${\delta _0}$，數(shù)據(jù)集X，目標(biāo)函數(shù)的計(jì)算公式 \small$F(\delta )$，最大迭代次數(shù)MT，單純形的最小半徑SR。
輸出：最優(yōu)核參數(shù) \small${\delta ^*}$。
步驟1?根據(jù)給定的初始化點(diǎn) \small${\delta _0}$構(gòu)造初始單純形；
步驟2?對(duì)于1維單純形的兩個(gè)頂點(diǎn)A和B，計(jì)算出目標(biāo)函數(shù)值 \small$F(\delta _{\rm A})$和 \small$F(\delta _{\rm B})$，函數(shù)值較小的為best點(diǎn)，較大的為worst點(diǎn)；
步驟3?計(jì)算反射點(diǎn)R和反射點(diǎn)處的目標(biāo)函數(shù)值 \small$F(\delta _{\rm R})$，如果反射點(diǎn)的目標(biāo)函數(shù)值優(yōu)于best點(diǎn)，則進(jìn)入步驟4；如果差于best點(diǎn)，則轉(zhuǎn)步驟5；如果反射點(diǎn)與best點(diǎn)處目標(biāo)值相等，則轉(zhuǎn)步驟6；
步驟4?計(jì)算擴(kuò)展點(diǎn)E和擴(kuò)展點(diǎn)處的目標(biāo)函數(shù)值 \small$F(\delta _{\rm E})$，比較反射點(diǎn)R處和擴(kuò)展點(diǎn)E的目標(biāo)函數(shù)值，選擇目標(biāo)函數(shù)值較小的點(diǎn)替換單純形中的 worst點(diǎn)，然后轉(zhuǎn)步驟7；
步驟5?比較反射點(diǎn)R與worst點(diǎn)處的目標(biāo)函數(shù)值，根據(jù)比較結(jié)果確定壓縮點(diǎn)M的位置，若壓縮點(diǎn)M的目標(biāo)函數(shù)值 \small$F({\delta _M})$與worst點(diǎn)相比更小，則用M點(diǎn)替換worst點(diǎn)，然后轉(zhuǎn)步驟7；否則，轉(zhuǎn)步驟6；
步驟6?執(zhí)行單純形收縮操作；
步驟7?若達(dá)到最大迭代次數(shù)MT，或單純形半徑小于SR，則迭代結(jié)束，best頂點(diǎn)對(duì)應(yīng)的 \small$\delta $值即為最優(yōu)核參數(shù)值 \small${\delta ^*}$；否則，轉(zhuǎn)步驟2進(jìn)行下一次迭代。

下載: 導(dǎo)出CSV

表 2 K插值單純形法核極限學(xué)習(xí)機(jī)訓(xùn)練算法

輸入：訓(xùn)練數(shù)據(jù)集X，測(cè)試數(shù)據(jù)集Y。
輸出：最優(yōu)核參數(shù) \small${\delta ^*}$、分類函數(shù) \small$\tilde {{f}}(\cdot)$，測(cè)試數(shù)據(jù)集的分類結(jié)果。
步驟1?初始化參數(shù)，給定插值數(shù)目K，正則化參數(shù)C，最大迭代次數(shù)MT，單純形最小半徑SR；
步驟2?數(shù)據(jù)預(yù)處理；
步驟3?計(jì)算K個(gè)插值核參數(shù)在訓(xùn)練數(shù)據(jù)集X上的訓(xùn)練精度，并從中找到合適的初始搜索點(diǎn) \small${\delta _0}$：
(1)構(gòu)造核參數(shù) \small$\delta $的K個(gè)插值點(diǎn) \small${\delta _i}$。 \small${\delta _i} = {2^{i - 1}}$，其中 \small$i = 1,2, ·\!·\!· ,K$；
(2)用式(8)分別計(jì)算K個(gè)插值點(diǎn) \small${\delta _i}$下相應(yīng)的分類參數(shù) \small${\tilde {β} _i}$，其中i=1,2,···,K；
(3)用式(9)計(jì)算K個(gè)插值點(diǎn) \small${\delta _i}$下的分類結(jié)果和訓(xùn)練精度 \small${P_{{\delta _i}}}$，其中i=1,2,···,K； (4)構(gòu)造訓(xùn)練精度矩陣 \small${S} = \left[ {\begin{array}{*{20}{c}} {{\delta _1}}&{{P_{{\delta _1}}}} \\ {{\delta _2}}&{{P_{{\delta _2}}}} \\ \vdots & \vdots \\ {{\delta _k}}&{{P_{\delta_k}}} \end{array}} \right]$；
(5)刪除訓(xùn)練精度矩陣 \small${S}$中 \small${P_{{\delta _i}}} \ge 0.999$的行，得到 \small${S}'$；
(6)取 \small${S}'$矩陣中的 \small${S}'\left( {1,1} \right)$元素作為初始搜索點(diǎn) \small${\delta _0}$；
步驟4?調(diào)用單純形法優(yōu)化核參數(shù)子算法，進(jìn)行迭代搜索，傳入搜索初值 \small${\delta _0}$，數(shù)據(jù)集X，目標(biāo)函數(shù)計(jì)算公式 \small$F(\delta )$，最大迭代次數(shù)MT和單純形最小半徑SR，得到最優(yōu)核參數(shù) \small${\delta ^*}$；
步驟5?根據(jù)最優(yōu)核參數(shù) \small${\delta ^*}$，用式(8)計(jì)算核ELM的分類參數(shù) \small$\tilde {β} $；
步驟6?對(duì)于測(cè)試數(shù)據(jù)集Y中的待分類樣本 \small${{x}_p}$，根據(jù)最優(yōu)核參數(shù)和步驟5的 \small$\tilde{β} $，用式(9)計(jì)算 \small${{x}_p}$的類別。

下載: 導(dǎo)出CSV

表 3 實(shí)驗(yàn)數(shù)據(jù)集列表

數(shù)據(jù)集	樣本數(shù)目	屬性數(shù)目	簇的數(shù)目	數(shù)據(jù)集	樣本數(shù)目	屬性數(shù)目	簇的數(shù)目
DNA	1967	180	3	Segment	2310	19	7
Letter	20000	16	26	Satellite	6435	36	6
Msplice	3044	240	3	Vowel	990	14	11
Musk	5692	166	2	Liver	345	7	2
Cnae	1080	857	9	Wilt	4839	5	2
Chess	28056	6	18	D31	3100	2	31

下載: 導(dǎo)出CSV

表 4 兩種核極限學(xué)習(xí)機(jī)算法的分類精度比較

數(shù)據(jù)集	傳統(tǒng)核ELM(%)									KS-KELM
數(shù)據(jù)集	\small$\delta $=0.01	\small$\delta $=0.1	\small$\delta $=1	\small$\delta $=10	\small$\delta $=50	\small$\delta $=100	最好精度	最差精度	平均精度	\small$\delta $值	精度(%)
DNA	21.41	24.63	80.94	94.65	66.38	55.03	94.65	21.41	57.17	4.5	95.72
Letter	4.19	83.32	89.04	77.88	54.22	49.69	89.04	4.19	59.72	2.4521	96.92
Msplice	54.86	59.59	79.53	94.24	75.52	54.86	94.24	54.86	69.77	6	94.82
Musk	84.08	86.09	89.96	95.22	95.94	96.13	96.13	84.08	91.24	224	96.99
Cnae	9.27	80.39	91.38	88.58	71.55	49.35	91.38	9.27	65.09	1.6631	92.67
Chess	0.17	58.55	62.10	32.82	24.35	22.08	62.10	0.17	33.35	0.832	63.21
D31	89.33	95.67	96.67	92.33	23.50	16.33	96.67	16.33	68.97	1.7471	97.17

下載: 導(dǎo)出CSV

表 5 5種極限學(xué)習(xí)機(jī)算法分類精度的比較

數(shù)據(jù)集	WMOS-2017^[19]	M-2016^[20]	IC-2017^[21]	H-2016^[22]	KS-KELM
Segment	94.12	95.56	—	—	95.12
Satellite	88.54	88.25	80.03	90.9	91.30
Letter	86.48	72.30	76.84	95.82	96.92
Vowel	88.78	59.74	—	64.94	92.12
Liver	—	—	66.56	72.17	80.26
Wilt	96.18	97.35	96.52	—	98.06

下載: 導(dǎo)出CSV

參考文獻(xiàn)(22)

HUANG Guangbing, ZHU Qinyu, and SIEW C K. Extreme learning machine: Theory and applications[J]. Neurocomputing, 2006, 70(1): 489–501.DOI: 10.1016/j.neucom.2005.12.126.

SALCEDO S S, CASANOVA M C, PASTOR S A, et al. Daily global solar radiation prediction based on a hybrid coral reefs optimization – extreme learning machine approach[J]. Solar Energy, 2014, 105: 91–98.DOI: 10.1016/j.solener.2014.04.009.

SONG Yuedong and ZHANG Jiaxiang. Discriminating preictal and interictal brain states in intracranial EEG by sample entropy and extreme learning machine[J]. Journal of Neuroscience Methods, 2016, 257: 45–54.DOI: 10.1016/j.jneumeth.2015.08.026.

SURI M and PARMAR V. Exploiting intrinsic variability of filamentary resistive memory for extreme learning machine architectures[J]. IEEE Transactions on Nanotechnology, 2015, 14(6): 963–968.DOI: 10.1109/TNANO.2015.2441112.

DING Shifei, ZHAO Han, ZHANG Yanan, et al. Extreme learning machine: Algorithm, theory and applications[J]. Artificial Intelligence Review, 2015, 44(1): 103–115.DOI: 10.1007/s10462-013-9405-z.

HUANG Guangbing, ZHOU Hongming, DING Xiaojian, et al. Extreme learning machine for regression and multiclass classification[J]. IEEE Transactions on Systems Man & Cybernetics Part B, 2012, 42(2): 513–529.DOI: 10.1109/TSMCB.2011.2168604.

HUANG Guangbing, DING Xiaojian, and ZHOU Hongming. Optimization method based extreme learning machine for classification.[J]. Neurocomputing, 2010, 74(1): 155–163.DOI: 10.1016/j.neucom.2010.02.019.

VAN G T, SUYKENS J A, LANCKRIET G, et al. Bayesian framework for least-squares support vector machine classifiers, gaussian processes, and kernel Fisher discriminant analysis[J]. Neural Computation, 2014, 14(5): 1115–1147.DOI: 10.1162/089976602753633411.

ADANKON M M, CHERIEL M, and AYAT N F. Optimizing resources in model selection for support vector machines[J]. Pattern Recognition, 2007, 40(3): 953–963.DOI: 10.1016/j.patcog.2006.06.012.

XU Yitian, PAN Xianli, ZHOU Zhijian, et al. Structural least square twin support vector machine for classification[J]. Applied Intelligence, 2015, 42(3): 527–536.DOI: 10.1007/s10489-014-0611-4.

XIAO Yingchao, WANG Huangang, ZHANG Lin, et al. Two methods of selecting Gaussian kernel parameters for one-class SVM and their application to fault detection[J]. Knowledge-Based Systems, 2014, 59(2): 75–84.DOI: 10.1016/j.knosys.2014.01.020.

XIAO Yingchao, WANG Huangang, and XU Weili. Parameter selection of gaussian kernel for one-class SVM.[J]. IEEE Transactions on Cybernetics, 2015, 45(5): 941–953.DOI: 10.1109/TCYB.2014.2340433.

TIAN Meng and WANG Wenjian. An efficient Gaussian kernel optimization based on centered kernel polarization criterion[J]. Information Sciences, 2015, 322: 133–149.DOI: 10.1016/j.ins.2015.06.010.

LAGARIAS J C, WRIGHT M H, WRIGHT P E, et al. Convergence properties of the Nelder-Mead simplex method in low dimensions[J]. SIAM Journal on Optimization, 1998, 9(1): 112–147.DOI: 10.1137/S1052623496303470.

馮增喜, 任慶昌, 彭彥平, 等. 基于單純形法的MFAC參數(shù)尋優(yōu)[J]. 控制工程, 2016, 23(3): 405–410.DOI: 10.14107/j.cnki.kzgc.150456.

FENG Zengxi, REN Qingchang, PENG Yanping, et al. Optimizing the parameters of MFAC based on the simplex method[J]. Control Engineering of China, 2016, 23(3): 405–410.DOI: 10.14107/j.cnki.kzgc.150456.

鄭偉博, 張紀(jì)會(huì). 基于Nelder-Mead單純形法的改進(jìn)量子行為粒子群算法[J]. 復(fù)雜系統(tǒng)與復(fù)雜性科學(xué), 2016, 13(2): 97–104.DOI: 10.13306/j.1672-3813.2016.02.012.

ZHENG Weibo and ZHANG Jihui. A improved quantum behaved particle swarm optimization algorithm using nelder and mead’s simplex algorithm[J]. Complex Systems & Complexity Science, 2016, 13(2): 97–104.DOI: 10.13306/j.1672-3813.2016.02.012.

KEERTHI S S and LIN C J. Asymptotic behaviors of support vector machines with gaussian kernel[J]. Neural Computation, 2003, 15(7): 1667–1689.DOI: 10.1162/089976603321891855.

張小云, 劉允才. 高斯核支撐向量機(jī)的性能分析[J]計(jì)算機(jī)工程, 2003, 29(8): 22–25.DOI: 10.3969/j.issn.1000-3428.2003.08.009.

ZHANG Xiaoyun and LIU Yuncai. Performance analysis of support vector machines with Gauss kernel[J]. Computer Engineering, 2003, 29(8): 22–25.DOI: 10.3969/j.issn.1000-3428.2003.08.009.

羅家祥, 羅丹, 胡躍明. 帶權(quán)重變化和決策融合的ELM在線故障檢測(cè)[J]. 控制與決策, 2017.DOI: 10.13195/j.kzyjc.2017.0274.該文獻(xiàn)為網(wǎng)絡(luò)優(yōu)先出版, 暫時(shí)沒有期卷號(hào)和頁碼

LUO Jiaxiang, LUO Dan, and HU Yueming. A new online extreme learning machine with varying weights and decision level fusion for fault detection[J]. Control and Decision, 2017.DOI: 10.13195/j.kzyjc.2017.0274.

ZHANG Yongshan, WU Jia, CAI Zhihua, et al. Memetic extreme learning machine[J]. Pattern Recognition, 2016, 58: 135–148.doi: 10.1016/j.patcog.2016.04.003.

ZHANG Yongshan, WU Jia, Zhou Chuan, et al. Instance cloned extreme learning machine[J]. Pattern Recognition, 2017, 68: 52–65.DOI: 10.1016/j.patcog.2017.02.036.

TANG Jiexiong, DENG Chenwei, and HUANG Guangbing. Extreme learning machine for multilayer perceptron[J]. IEEE Transactions on Neural Networks & Learning Systems, 2016, 27(4): 809–821.DOI: 10.1109/TNNLS.2015.2424995.

相關(guān)文章

施引文獻(xiàn)

資源附件(0)

訪問統(tǒng)計(jì)