K插值單純形法核極限學(xué)習(xí)機(jī)的研究
doi: 10.11999/JEIT171104
-
廣西大學(xué)計(jì)算機(jī)與電子信息學(xué)院 南寧 530004
Kernel Extreme Learning Machine Based on K Interpolation Simplex Method
-
College of Computer and Electronic Information, Guangxi University, Nanning 530004, China
-
摘要: 針對(duì)核極限學(xué)習(xí)機(jī)高斯核函數(shù)參數(shù)選優(yōu)難,影響學(xué)習(xí)機(jī)訓(xùn)練收斂速度和分類精度的問題,該文提出一種K插值單純形法的核極限學(xué)習(xí)機(jī)算法。把核極限學(xué)習(xí)機(jī)的訓(xùn)練看作一個(gè)無約束優(yōu)化問題,在訓(xùn)練迭代過程中,用Nelder-Mead單純形法搜索高斯核函數(shù)的最優(yōu)核參數(shù),提高所提算法的分類精度。引入K插值為Nelder-Mead單純形法提供合適的初值,減少單純形法的迭代次數(shù),提高了新算法的訓(xùn)練收斂效率。通過在UCI數(shù)據(jù)集上的仿真實(shí)驗(yàn)并與其它算法比較,新算法具有更快的收斂速度和更高的分類精度。
-
關(guān)鍵詞:
- 核極限學(xué)習(xí)機(jī) /
- 核參數(shù) /
- Nelder-Mead單純形法 /
- K插值法
Abstract: The kernel Extreme Learning Machine (ELM) has a problem that the kernel parameter of the Gauss kernel function is hard to be optimized. As a result, training speed and classification accuracy of kernel ELM are negatively affected. To deal with that problem, a novel kernel ELM based on K interpolation simplex method is proposed. The training process of kernel ELM is considered as an unconstrained optimal problem. Then, the Nelder-Mead Simplex Method (NMSM) is used as an optimal method to search the optimized kernel parameter, which improves the classification accuracy of kernel ELM. Furthermore, the K interpolation method is used to provide appropriate initial values for the Nelder-Mead simplex to reduce the number of iterations, and as a result, the training speed of ELM is improved. Comparative results on UCI dataset demonstrate that the novel ELM algorithm has better training speed and higher classification accuracy. -
表 1 1維單純形子算法
輸入:初始搜索點(diǎn) \small${\delta _0}$,數(shù)據(jù)集X,目標(biāo)函數(shù)的計(jì)算公式 \small$F(\delta )$,最大迭代次數(shù)MT,單純形的最小半徑SR。 輸出:最優(yōu)核參數(shù) \small${\delta ^*}$。 步驟1?根據(jù)給定的初始化點(diǎn) \small${\delta _0}$構(gòu)造初始單純形; 步驟2?對(duì)于1維單純形的兩個(gè)頂點(diǎn)A和B,計(jì)算出目標(biāo)函數(shù)值 \small$F(\delta _{\rm A})$和 \small$F(\delta _{\rm B})$,函數(shù)值較小的為best點(diǎn),較大的為worst點(diǎn); 步驟3?計(jì)算反射點(diǎn)R和反射點(diǎn)處的目標(biāo)函數(shù)值 \small$F(\delta _{\rm R})$,如果反射點(diǎn)的目標(biāo)函數(shù)值優(yōu)于best點(diǎn),則進(jìn)入步驟4;如果差于best點(diǎn),則轉(zhuǎn)步驟5;如 果反射點(diǎn)與best點(diǎn)處目標(biāo)值相等,則轉(zhuǎn)步驟6; 步驟4?計(jì)算擴(kuò)展點(diǎn)E和擴(kuò)展點(diǎn)處的目標(biāo)函數(shù)值 \small$F(\delta _{\rm E})$,比較反射點(diǎn)R處和擴(kuò)展點(diǎn)E的目標(biāo)函數(shù)值,選擇目標(biāo)函數(shù)值較小的點(diǎn)替換單純形中的 worst點(diǎn),然后轉(zhuǎn)步驟7; 步驟5?比較反射點(diǎn)R與worst點(diǎn)處的目標(biāo)函數(shù)值,根據(jù)比較結(jié)果確定壓縮點(diǎn)M的位置,若壓縮點(diǎn)M的目標(biāo)函數(shù)值 \small$F({\delta _M})$與worst點(diǎn)相比更小,則 用M點(diǎn)替換worst點(diǎn),然后轉(zhuǎn)步驟7;否則,轉(zhuǎn)步驟6; 步驟6?執(zhí)行單純形收縮操作; 步驟7?若達(dá)到最大迭代次數(shù)MT,或單純形半徑小于SR,則迭代結(jié)束,best頂點(diǎn)對(duì)應(yīng)的 \small$\delta $值即為最優(yōu)核參數(shù)值 \small${\delta ^*}$;否則,轉(zhuǎn)步驟2進(jìn)行下一次 迭代。 下載: 導(dǎo)出CSV
表 2 K插值單純形法核極限學(xué)習(xí)機(jī)訓(xùn)練算法
輸入:訓(xùn)練數(shù)據(jù)集X,測(cè)試數(shù)據(jù)集Y。 輸出:最優(yōu)核參數(shù) \small${\delta ^*}$、分類函數(shù) \small$\tilde {{f}}(\cdot)$,測(cè)試數(shù)據(jù)集的分類結(jié)果。 步驟1?初始化參數(shù),給定插值數(shù)目K,正則化參數(shù)C,最大迭代次數(shù)MT,單純形最小半徑SR; 步驟2?數(shù)據(jù)預(yù)處理; 步驟3?計(jì)算K個(gè)插值核參數(shù)在訓(xùn)練數(shù)據(jù)集X上的訓(xùn)練精度,并從中找到合適的初始搜索點(diǎn) \small${\delta _0}$: (1)構(gòu)造核參數(shù) \small$\delta $的K個(gè)插值點(diǎn) \small${\delta _i}$。 \small${\delta _i} = {2^{i - 1}}$,其中 \small$i = 1,2, ·\!·\!· ,K$; (2)用式(8)分別計(jì)算K個(gè)插值點(diǎn) \small${\delta _i}$下相應(yīng)的分類參數(shù) \small${\tilde {β} _i}$,其中i=1,2,···,K; (3)用式(9)計(jì)算K個(gè)插值點(diǎn) \small${\delta _i}$下的分類結(jié)果和訓(xùn)練精度 \small${P_{{\delta _i}}}$,其中i=1,2,···,K;
(4)構(gòu)造訓(xùn)練精度矩陣 \small${S} = \left[ {\begin{array}{*{20}{c}} {{\delta _1}}&{{P_{{\delta _1}}}} \\ {{\delta _2}}&{{P_{{\delta _2}}}} \\ \vdots & \vdots \\ {{\delta _k}}&{{P_{\delta_k}}} \end{array}} \right]$;(5)刪除訓(xùn)練精度矩陣 \small${S}$中 \small${P_{{\delta _i}}} \ge 0.999$的行,得到 \small${S}'$; (6)取 \small${S}'$矩陣中的 \small${S}'\left( {1,1} \right)$元素作為初始搜索點(diǎn) \small${\delta _0}$; 步驟4?調(diào)用單純形法優(yōu)化核參數(shù)子算法,進(jìn)行迭代搜索,傳入搜索初值 \small${\delta _0}$,數(shù)據(jù)集X,目標(biāo)函數(shù)計(jì)算公式 \small$F(\delta )$,最大迭代次數(shù)MT和單純形 最小半徑SR,得到最優(yōu)核參數(shù) \small${\delta ^*}$; 步驟5?根據(jù)最優(yōu)核參數(shù) \small${\delta ^*}$,用式(8)計(jì)算核ELM的分類參數(shù) \small$\tilde {β} $; 步驟6?對(duì)于測(cè)試數(shù)據(jù)集Y中的待分類樣本 \small${{x}_p}$,根據(jù)最優(yōu)核參數(shù)和步驟5的 \small$\tilde{β} $,用式(9)計(jì)算 \small${{x}_p}$的類別。 下載: 導(dǎo)出CSV
表 3 實(shí)驗(yàn)數(shù)據(jù)集列表
數(shù)據(jù)集 樣本數(shù)目 屬性數(shù)目 簇的數(shù)目 數(shù)據(jù)集 樣本數(shù)目 屬性數(shù)目 簇的數(shù)目 DNA 1967 180 3 Segment 2310 19 7 Letter 20000 16 26 Satellite 6435 36 6 Msplice 3044 240 3 Vowel 990 14 11 Musk 5692 166 2 Liver 345 7 2 Cnae 1080 857 9 Wilt 4839 5 2 Chess 28056 6 18 D31 3100 2 31 下載: 導(dǎo)出CSV
表 4 兩種核極限學(xué)習(xí)機(jī)算法的分類精度比較
數(shù)據(jù)集 傳統(tǒng)核ELM(%) KS-KELM \small$\delta $=0.01 \small$\delta $=0.1 \small$\delta $=1 \small$\delta $=10 \small$\delta $=50 \small$\delta $=100 最好精度 最差精度 平均精度 \small$\delta $值 精度(%) DNA 21.41 24.63 80.94 94.65 66.38 55.03 94.65 21.41 57.17 4.5 95.72 Letter 4.19 83.32 89.04 77.88 54.22 49.69 89.04 4.19 59.72 2.4521 96.92 Msplice 54.86 59.59 79.53 94.24 75.52 54.86 94.24 54.86 69.77 6 94.82 Musk 84.08 86.09 89.96 95.22 95.94 96.13 96.13 84.08 91.24 224 96.99 Cnae 9.27 80.39 91.38 88.58 71.55 49.35 91.38 9.27 65.09 1.6631 92.67 Chess 0.17 58.55 62.10 32.82 24.35 22.08 62.10 0.17 33.35 0.832 63.21 D31 89.33 95.67 96.67 92.33 23.50 16.33 96.67 16.33 68.97 1.7471 97.17 下載: 導(dǎo)出CSV
-
HUANG Guangbing, ZHU Qinyu, and SIEW C K. Extreme learning machine: Theory and applications[J]. Neurocomputing, 2006, 70(1): 489–501.DOI: 10.1016/j.neucom.2005.12.126. SALCEDO S S, CASANOVA M C, PASTOR S A, et al. Daily global solar radiation prediction based on a hybrid coral reefs optimization – extreme learning machine approach[J]. Solar Energy, 2014, 105: 91–98.DOI: 10.1016/j.solener.2014.04.009. SONG Yuedong and ZHANG Jiaxiang. Discriminating preictal and interictal brain states in intracranial EEG by sample entropy and extreme learning machine[J]. Journal of Neuroscience Methods, 2016, 257: 45–54.DOI: 10.1016/j.jneumeth.2015.08.026. SURI M and PARMAR V. Exploiting intrinsic variability of filamentary resistive memory for extreme learning machine architectures[J]. IEEE Transactions on Nanotechnology, 2015, 14(6): 963–968.DOI: 10.1109/TNANO.2015.2441112. DING Shifei, ZHAO Han, ZHANG Yanan, et al. Extreme learning machine: Algorithm, theory and applications[J]. Artificial Intelligence Review, 2015, 44(1): 103–115.DOI: 10.1007/s10462-013-9405-z. HUANG Guangbing, ZHOU Hongming, DING Xiaojian, et al. Extreme learning machine for regression and multiclass classification[J]. IEEE Transactions on Systems Man & Cybernetics Part B, 2012, 42(2): 513–529.DOI: 10.1109/TSMCB.2011.2168604. HUANG Guangbing, DING Xiaojian, and ZHOU Hongming. Optimization method based extreme learning machine for classification.[J]. Neurocomputing, 2010, 74(1): 155–163.DOI: 10.1016/j.neucom.2010.02.019. VAN G T, SUYKENS J A, LANCKRIET G, et al. Bayesian framework for least-squares support vector machine classifiers, gaussian processes, and kernel Fisher discriminant analysis[J]. Neural Computation, 2014, 14(5): 1115–1147.DOI: 10.1162/089976602753633411. ADANKON M M, CHERIEL M, and AYAT N F. Optimizing resources in model selection for support vector machines[J]. Pattern Recognition, 2007, 40(3): 953–963.DOI: 10.1016/j.patcog.2006.06.012. XU Yitian, PAN Xianli, ZHOU Zhijian, et al. Structural least square twin support vector machine for classification[J]. Applied Intelligence, 2015, 42(3): 527–536.DOI: 10.1007/s10489-014-0611-4. XIAO Yingchao, WANG Huangang, ZHANG Lin, et al. Two methods of selecting Gaussian kernel parameters for one-class SVM and their application to fault detection[J]. Knowledge-Based Systems, 2014, 59(2): 75–84.DOI: 10.1016/j.knosys.2014.01.020. XIAO Yingchao, WANG Huangang, and XU Weili. Parameter selection of gaussian kernel for one-class SVM.[J]. IEEE Transactions on Cybernetics, 2015, 45(5): 941–953.DOI: 10.1109/TCYB.2014.2340433. TIAN Meng and WANG Wenjian. An efficient Gaussian kernel optimization based on centered kernel polarization criterion[J]. Information Sciences, 2015, 322: 133–149.DOI: 10.1016/j.ins.2015.06.010. LAGARIAS J C, WRIGHT M H, WRIGHT P E, et al. Convergence properties of the Nelder-Mead simplex method in low dimensions[J]. SIAM Journal on Optimization, 1998, 9(1): 112–147.DOI: 10.1137/S1052623496303470. 馮增喜, 任慶昌, 彭彥平, 等. 基于單純形法的MFAC參數(shù)尋優(yōu)[J]. 控制工程, 2016, 23(3): 405–410.DOI: 10.14107/j.cnki.kzgc.150456.FENG Zengxi, REN Qingchang, PENG Yanping, et al. Optimizing the parameters of MFAC based on the simplex method[J]. Control Engineering of China, 2016, 23(3): 405–410.DOI: 10.14107/j.cnki.kzgc.150456. 鄭偉博, 張紀(jì)會(huì). 基于Nelder-Mead單純形法的改進(jìn)量子行為粒子群算法[J]. 復(fù)雜系統(tǒng)與復(fù)雜性科學(xué), 2016, 13(2): 97–104.DOI: 10.13306/j.1672-3813.2016.02.012.ZHENG Weibo and ZHANG Jihui. A improved quantum behaved particle swarm optimization algorithm using nelder and mead’s simplex algorithm[J]. Complex Systems & Complexity Science, 2016, 13(2): 97–104.DOI: 10.13306/j.1672-3813.2016.02.012. KEERTHI S S and LIN C J. Asymptotic behaviors of support vector machines with gaussian kernel[J]. Neural Computation, 2003, 15(7): 1667–1689.DOI: 10.1162/089976603321891855. 張小云, 劉允才. 高斯核支撐向量機(jī)的性能分析[J]計(jì)算機(jī)工程, 2003, 29(8): 22–25.DOI: 10.3969/j.issn.1000-3428.2003.08.009.ZHANG Xiaoyun and LIU Yuncai. Performance analysis of support vector machines with Gauss kernel[J]. Computer Engineering, 2003, 29(8): 22–25.DOI: 10.3969/j.issn.1000-3428.2003.08.009. 羅家祥, 羅丹, 胡躍明. 帶權(quán)重變化和決策融合的ELM在線故障檢測(cè)[J]. 控制與決策, 2017.DOI: 10.13195/j.kzyjc.2017.0274.該文獻(xiàn)為網(wǎng)絡(luò)優(yōu)先出版, 暫時(shí)沒有期卷號(hào)和頁碼LUO Jiaxiang, LUO Dan, and HU Yueming. A new online extreme learning machine with varying weights and decision level fusion for fault detection[J]. Control and Decision, 2017.DOI: 10.13195/j.kzyjc.2017.0274. ZHANG Yongshan, WU Jia, CAI Zhihua, et al. Memetic extreme learning machine[J]. Pattern Recognition, 2016, 58: 135–148.doi: 10.1016/j.patcog.2016.04.003. ZHANG Yongshan, WU Jia, Zhou Chuan, et al. Instance cloned extreme learning machine[J]. Pattern Recognition, 2017, 68: 52–65.DOI: 10.1016/j.patcog.2017.02.036. TANG Jiexiong, DENG Chenwei, and HUANG Guangbing. Extreme learning machine for multilayer perceptron[J]. IEEE Transactions on Neural Networks & Learning Systems, 2016, 27(4): 809–821.DOI: 10.1109/TNNLS.2015.2424995. -