- Visibility 134 Views
- Downloads 50 Downloads
- Permissions
- DOI 10.18231/j.ijn.2021.041
-
CrossMark
- Citation
An advance of complex multifaceted root learning for parkinson behavior using subthalamic nucleus biomarkers
- Author Details:
-
Venkateshwarla Rama Raju *
-
Kavitha Rani Balmuri
-
Konda Srinivas
-
G. Madhukar
Abstract
Though DBS is suiting well to Parkinson`s still its current version being modified now and often as suggested by the neuroscientists. The technique has gained an effectual management surgical therapy concerned to PD, explicitly whilst growing idea as an effectual method to alleviate Parkinson’s disease and other movement disorders. The present ver.1 indirect DBS is an open loop based and its parameters are changeable manually and there is no provision to adjust automatically or online based on Parkinson diseased behavior and performance. Hence, supervised classification of patient behavior is a major and significant step towards the design of next generation DBS systems which are adaptive closed loop based. The work in this study demonstrates a supervised classification machine learning (CML, i.e., multiple kernel learning M-K-L) method to distinguish such cognitive behavioral tasks by using the subthalamic nuclei (STN) biomarkers, i.e., biomedical data of microelectrode recording (MER) bio signals (or local field potential LFP). We applied the time domain and frequency domain representation spectrograms of the raw data acquired from right and left hemisphere brain`s STNs as the feature vectors. Following the feature extractions, we combined those features via support vector machines (SVMs) with complex multifaceted root learning, i.e., C-M-L or multi kernel learning (M-K-L) formulation. The C-M-L based classification techniques were applied to a class and categorize different tasks such as switch (push-pull button), movement of jaws, vocalizations, plus movement of arm due to the tremor. Our experiments show that the l p - n o r m C-M-L/M-K-L radically smash distinct kernel SVM-based classifiers in classifying behavioral tasks of five subjects even using signals acquired with a low sampling rate of 10 Hz. This leads to a lower computational cost.
Introduction
Parkinson disease (or Parkinson`s disease designated as PD) is a complex neuro degenerative disease usually occur in the brains central nervous system – the CNS.[1] Even though, the core and key cause of this malady phenomenon is not yet known. Studies shows that the contact of diverse and distinctive processing-circuits of the basal-ganglia and motor-cortex might have concerned and implicated.[1], [2] The features, i.e., signs and symptoms of PD appear by the malfunction and death of dopamine-generating cells in an important brain area – the substantia-nigra. The lack of these vital neurons causes various motor disorders including tremor, rigidity, bradykinesia, and postural instability.[3]
Although, there is currently no certain cure for PD, there are different kinds of treatment options such as medication and surgery to alleviate the disorder manifestations. In recent years, Deep Brain Stimulation (DBS) has been considered as drug therapy.[1], [2], [3], [4] Using high frequency (160Hz-200Hz) square pulses (electrical). The stimulus procedure is done during implanted microelectrodes and the current is supplied by a battery-powered implanted pulse generators (IPGs), i.e., the battery like pacemaker is implanted at chest or at abdomen and the IPG at the neck.[4] In spite of DBS remarkable performance in providing relief to Parkinson`s, the stimulations may cause some side effects, for instance, cognitive impairment/dementia and postural instabilities, disruption.[5] This is mainly because of the existing open-loop DBS systems and lack of understanding about the mechanism of its action. In other words, altered physiological dynamics from adjusting the stimulation parameters, e.g., voltage, stimulus intensity, pulse-width duration measurement and frequency, remains uncertain/and vague.[5]
Therefore, designing an adaptive closed loop to be able to automatically adjust the stimulation parameters is currently an important research area. Contrary to the direct open loop system that provides a time invariant stimulus current pulse, adaptive/closed loop device would generate a customized stimuli based on the patients’ current behavior, reducing the undesirable dysarthrias which are side effects of the DB stimulation therapy.[1], [2], [3], [4], [5]
To design an efficient closed-loop device, recognition of dissimilar and unusual PD behavioral tasks rooted in feed-back-signals of PD brain is a main issue/problem to address. DBS provides a chance and an opportunity to gain the access to different brain-waves includes bio signals or biomarkers like beta β-oscillations/local field potentials L.F.Ps, plus acquire unswervingly from the ganglia, i.e., the basal-ganglia.[6] Dissimilar activities of PD brain are coded in these waveforms, such that these activities might be differentiated by processing the collected-unruffled signal/waveform. Rooted in different kinds of brain waveforms, hitherto, there have been a number of studies concerning to the detection and classification of singular behavioral tasks of PD brain. Many algorithms have been developed based on the processing of EEG and electro cortico graphy E.Co.G, for instance, epileptic-seizure detections,[7] brain switch rooted in motor fantasy or pretend,[8] and P-300/speller-paradigm,[9], [10] artificial-neural-nets[11], [12] and S V M based classifiers mutually by means of feature extractions, pattern matching templates, through wavelets have been successful in perceiving the Parkinson behavior.[1], [2], [3], [4], [5], [13], [14] Latterly, S.T.N-β-oscillations (L.F.Ps) have been considered as a useful neural feed-back-biomarker, i.e., bio- signal to identify behavioral activities of the PD brain. The time-series analysis of motor-cortex acquired by E.Cortico.G and S.T.N-L.F.Ps has shown that PD exhibits oscillatory behavior modulated by motor activities, which results in suppression of β-oscillations (~13Hertz-~35Hertz) frequency/spectral-power in motor-tasks. [1], [2] Considering this point into account, a support vector machine -based classifier was proposed by the authors.[15] A motor task detection method by applying the L.F. potentials and non linear-regression was shown by authors in.[14]
An adaptive learning approach based on biomarkers, i.e., the L.F.Ps/signal was suggested by authors in their study,[16] in which a model, i.e., hybrid for merging coalescing the support vector machine and probabilistic hidden-Markov-models were applied for the analysis of behavioral clustering of Parkinson`s brain, i.e., the PD brain.
In this study, we demonstrate a l p – n o r m complex Kernel learning C.K.L. or Multi Kernel Learning - the M.K.L.[17] approach for the identification of unusual behavioral activities of PD brain by employing the sub thalamic nuclei 30Hz β-oscillations – the L.F.Ps acquired from PD diseased subjects underwent for the therapeutic-surgery of deep brain stimulation. The spectrogram of the raw/unrefined L F P s correlated to every-event was employed to extort the feature-vectors. On the contrary to the formerly demonstrated single kernel S V M - based classifiers, the proposed C.K.L MKL approach utilizes the equivalent and matching biomarkers from PD brains left hemisphere and right hemisphere sub thalamic nuclei’s concurrently. Hence, it will yeild a privileged and superior performance by the proposed-technique/method. Our experimentally investigated findings show that the classification accuracy of the demonstrated method is vigorous even if the feature vectors are largely down-sampled. So, this leads to a lower computational burden. The sections in this study are as follows:
Firstly, signals data acquisition by MER recording-procedure in section 2, secondly, section 3 gives the l p – n o r m C.K.L./M.K.L method and also proposed approach, thirdly, section 4 yields the contrast and quantitative findings, and lastly, section 5 gives the, inferences drawn from this study followed by some observations.
Signal Acquisition
Five 5 Parkinson`s underwent therapeutic deep brain stimulation functional neurosurgery were participated in this bilateral STN-DBS study. The participants had given their informed consent which was approved by the institute ethical committee following the Helsinki principles. The participants performed bilateral MER signal acquisitions using microelectrodes embedded in their brains two sides, i.e., right and left hemispheres subthalamic nucleuses. MER signal acquisitions were done while the PD subjects awakened and all were DBS state off medication.[1], [2], [3], [4], [5] The field potentials/biomarkers were acquired from all four contacts of each DBS electrodes. The acquired signals were amplified, and with a 5 kilo Hertz (5kHz) sampling frequency they were digitized, and filtered with band pass filters with 0.5 Hertz lower cutoff frequency and 100 Hertz upper cutoff frequencies, and combined with event markers and subject responses. In the interim, a linked-mastoid common reference was employed for signal acquisitions. And, lastly, the local field potential channels were bi polar re referenced, i.e., between 0 to 1, 1 to 2, and between 2 to 3 prior to the inferences drawn.[1], [2], [3], [4], [5], [6], [9] Four kinds of dissimilar tasks were included experimentally investigative: push the button, i.e., press-the-button, movement-of-jaw/the mouth, the vocalizations, and the movement of arm. For every undertaking, a block of numerous repetitions were designed. In terms of the task-initiation, the PD-subjects were cued by an audiosignal. For the press-button, the PD subjects were requested to push the button by adapting their right and/or left thumb. The task of vocalizations comprised of narrating easy and elegant names objects exhibited on the monitor. For the task of the movement of the arm, the Parkinson`s diseased subjects had to raise their arm to contact a tinted target-objective. Lastly the vocalizations were just consisted poignant/touching the jaws devoid of dialogue or idiom or tongue as a contrast to trials of vocalizations.
Materials anMethods
Firstly, a concise examination of the l p – n o r m C.K.L./M.K.L classifier was explained. Followed by the approach of the feature extraction and also the planned and projected classification method is demonstrated.
2.1. L p – n o r m complex Kernel Learning
The CKL has gained to a large extent fame in the pattern matching recognition and templates in artificial intelligence machine learning Faculty owing to its enviable classification ability. Latest studies[17], [18] show that C.K.L.-M.K.L shall progress the discriminate control of the support vector machine classifier. The concept behind the C.K.L is to optimally combine the matrix/matrices computed rooted in multi features by means of complex conjugate kernels in support vector machines optimally. [17] Considerably, the Kernel functions plan the features to a new space where they can linearly be separable. The C.K.L-based support vector machines aims to study the resolution decision-boundaries amid singular, i.e., diverse classes and the K e r n e l mishmash permutation weights in a sole and solitary optimization hitch.[17], [18] We use the understanding of the C.K.L. classifier referred to as/ termed as a l p – n o r m C.K.L., which proved to be further flexible-supple in choosing distinct K e r n e l mishmashes/permutations. An l p – n o r m C.K.L, wherein-p≥1 is termed as follows.[1], [2], [3], [4], [5], [16], [17], [18]
minω, ω0,ξω, ω0,ξ= 12||ω||2, p2+ C ∑i=1Nξi ξi≥0, p≥1
yi ∑m=1MωTϕm (xi)+ω0 ≥1- ξi , 1= 1, 2, 3, …, N
where, ϕm(·) maps the feature vector xi to another space based on which the kernel function k(· , ·) = ‹ϕm(·) , ϕm(·)› is defined. The {ωm}`s are the parameters of the decision hyper-planes. M and N are the number of kernels and training samples respectively. C is the penalty parameter and ξi is the slack parameter. Where, ϕm (.) maps the feature vector xi to another space based on which the kernel function k(.,.) = k(· , ·) = ‹ϕm(·) , ϕm(·)› is defined. The biomarkers, i.e., the bio signals/or local field potentials were acquired from 4 contacts of each D B S microelectrodes. The collected data were then amplified, digitized (5 kHz), band passed filtered (1-100 ϕm(·) maps the feature vector xi to another space based on which the kernel function k(· , ·) = ‹ϕm(·) , ϕm(·)› is termed as follows. {ωm}s are the parameters of the decision hyper-planes. M and N are the number of kernels and training samples respectively. C is the penalty parameter and ξi is the slack parameter.

This convex optimization problem[17] is solved using its dual form as follows:
|
|
3 Tasks |
5 Tasks |
||
|
Freq. (sampling) |
S.V.M.- Linear |
C.K.L./M.K.L |
S.V.M..-Linear |
C.K.L./M.K.L. |
|
5kHz |
66.89 |
67.99 |
53.17 |
54.86 |
|
0.5kHz |
65.99 |
67.86 |
53.43 |
54.19 |
|
50Hz |
65.44 |
69.99 |
53.33 |
56.56 |
|
25Hz |
65.45 |
69.89 |
53.27 |
57.44 |
|
10Hz |
66.40 |
70.83 |
51.29 |
58.34 |
|
2 Hz |
58.15 |
65.33 |
44.05 |
53.53 |
mindmaxdL(d, α)=1T α-(1/2) α T YK dYα, p ∈(1, 2)
mindmaxdL(d, α)=1T α-(1/2) α T YK dYα, p ∈(2+ α) equation (2)
Such that K d = ∑m=1MdmKm,∑i=1Nαiyi=0,0≤αiC, ∑m=1Mdmp/(2-m)≤1, dm ≥ 0
where, α=(α1, …, αN)T is Lagrange-vector dual-variables, Y = diag(y1, …, yN) is an NxN diagonal-matrix (yi, label of each sample-xi), K(m) – kernel-matrix corresponding to the mth kernel-function, d=(d1, …, dN)T is the kernel mixture/grouping-vector which prevents the weight of (||ωm||2) in the objective-function of Equation(1), and the other parameters are defined precisely the similar as given for Equation (1).
Feature Extraction
As discussed previously, the time and frequency spatio temporal representation/spectro-gram of the acquired subthalamic nuclei local field potentials STN—DBS as our features are applied here. It was showed that singular behavioral tasks yield different representations in the time-frequency domain.[1] Thus, it is an appropriate measure to differentiate various Parkinson`s behaviors. To obtain the spectrogram of a continuous signal, we apply the continuous wavelet transform (CWT) on the bilaterally re-referenced LFPs. We employ the complex Morlet (C-Morlet) mother wavelet which is proven to be a suitable choice for biomedical signal processing: [1], [15]
is the C-Morlet mother wavelet. fc and fb are respectively the wavelet center frequency and bandwidth parameter. Since we are interested in analyzing the β frequency components, fc is set here in the range of 13Hz-35Hz.
X (a, b) ∫-∞+∞x(t)αΨt-badt, and Ψt=e-t2/fbπfb ej2πfct
where, Xω(a,b) is the CWT of the function x(t) with two variables a (scaling parameter) and b (shift parameter), and Ψ is the C-Morlet mother wavelet. fc and fb are respectively the wavelet center frequency and bandwidth parameter. Since we are interested in analyzing the β frequency components, fc is set here in the range of 13Hz-35Hz interval. Afterwards, the corresponding wavelet coefficients in the 𝛽 frequency range are calculated using Eq. (3).
To drive the l p – n o r m MKL classifier, we consider the right and left spectrograms for each event as two feature vectors. First, the two dimensional spectrograms are converted to vectors, and low-pass filtered by an anti-aliasing Butterworth filter of order 10. Then, these feature vectors are down-sampled to keep the computational cost low. Finally, the MKL classifier is applied on these pre-processed data to recognize the behavioral task related to each event. [Figure 1] shows the aforementioned procedure graphically.
Results
To evaluate the accuracy of the presented approach, we use the raw LFP data from five unusual/special PD subjects, as demonstrated in the section 2. The computational results of our method are contrasted differentiated via a newly proposed single kernel S-V-M-based method [15] plus we also evaluate the result of 3 distinct kernel-functions on the S-V-M classifier disjointedly and singly: 1. Radial Basis Function (RBF) k(x,y) =exp(γ||x–y||2), 2. Linear function k(x,y)=xTy+c, and 3. Polynomial function k(x,y) = (xTy+c)d.
Note that, x and y are two feature vectors, and γ, c, where, Xω(a,b) is the CWT of the function x(t) with two variables a (scaling parameter) and b (shift parameter). In terms of the lp-norm MKL, we set the parameter p=1.8, and for the SVM-based classifier all the parameters are set so as to achieve the best performance. A10-fold cross validation is implemented in all experiments. Furthermore, in all cases, principal component analysis (PCA) is applied to reduce the dimensionality of data (in each case, 95% of the eigen-values corresponding to the maximum variance direction is kept).
Classification Scheme
The first step to acquire the training and test samples is to specify those parts of the raw LFP data that are related to different events. A time window ranging from 1 sec before to 1 sec after each onset is used to determine the relevant time using Eq. (3).
To evaluate the accuracy of the presented approach, we use the raw LFP data from five different subjects, as described in Section II. The quantitative results of our method are compared with a recently proposed single kernel SVM-based method.[15] We also assess the effect of three different kernel functions on the SVM classifier separately: 1. Radial Basis Function (RBF) k(x, y) = exp(γ||x–y||2), 2. Linear function k(x, y) = xTy+c, and 3. Polynomial function k(x, y) = (xTy+c)d. Note that, x and y are two feature vectors, and γ, c, and d are optional constants. In terms of the lp-norm MKL, we set the parameter p=1.8, and for the SVM-based classifier and d are optional constants. In terms of the l p – n o r m MKL, we set the parameter p=1.8, and for the SVM-based classifier all the parameters are set so as to achieve the best performance. A10-fold cross validation is implemented in all experiments. Furthermore, in all cases, principal component analysis (PCA) is applied to reduce the dimensionality of data (in each case, 95% of the eigen-values corresponding to the maximum variance direction is kept).
Figure. 2 depicts the classification accuracy of different classifiers vs the sampling frequency (from 5kHz to 2Hz). As seen, the presented l p – n o r m MKL classifier outperforms the other compared classifiers. In the experiments, we observed that the results of the MKL classifier are robust even when the sampling frequency of the feature vectors is drastically low. To measure the robustness of different methods against the size of the feature vectors, all the experiments are redone for different down-sampling rates. [Table 1] gives the average classification accuracy of all five subjects for different scenarios (i.e., three and five-task classification and different sampling frequencies). As shown, regardless of the sampling frequency and the number of tasks, the presented MKL-based classifier returns the best results. Figure. 3 provides the average confusion matrix of all subjects with sampling frequency of 10Hz and five-task classification, which summarizes the identification results.

In terms of execution time of the algorithms, using MATLAB 2013a (Mathworks Inc) on a PC with Intel Ci5 CPU (3.4GHz) and 8GB memory, the average training time of each fold of the MKL method for the sampling frequency of 5kHz and 10Hz are about 450s and 6s respectively. The execution time in the test phase for each sample is about 1.5ms and 1ms respectively for the 5kHz and 10Hz sampling rates. For the single kernel SVM classifier, however, the average training time of each fold is about 50s and 0.2s respectively for the aforementioned sampling frequencies, and testing each sample takes about 2ms and 0.3ms. Note that, while 6GB RAM is needed for the 5kHz sampling rate, only 0.25GB memory is used for the 10Hz rate.
Conclusions
In this paper, an l p – n o r m MKL approach for classification of different human behavioral tasks using STN-LFP signal was presented. A feature extraction method based on the time-frequency analysis (spectrogram) of the collected signal was developed. We used the left and right LFPs acquired from the corresponding STNs as two feature vectors. This led to a higher performance with the MKL classifier. The experiments were conducted on five dataset recorded from patients undergoing DBS surgery. The quantitative results confirmed the superiority of the proposed method in almost all cases. In addition, to evaluate the robustness of the classifiers with respect to the size of feature vectors, different down- sampling rates were tested. In contrast to the single kernel SVM-based methods, the proposed MKL approach shows promising results even for very low sampling frequencies. This led to a lower computational burden. Note that, a more accurate classification of the human behavioral tasks would be a precursor for developing future closed-loop DBS systems, which is a cutting-edge research area. Developing a more robust feature extraction method is an interesting extension to this work. Moreover, evaluating the connectivity of different bipolar channels across the DBS leads can be another relevant topic for future research.
![Average confusion matrix of five subjects. The results are reported for the MKL classifier and sampling frequency of 10 Hz. All the values are normalized in the range of [0 100].](https://s3-us-west-2.amazonaws.com/typeset-prod-media-server/3213258a-0803-4410-8788-052c9883524bimage3.png)
Acknowledgments
The author wish to thank the DST-CSRI funded project, Dept of Science & Technology (DST), Ministry of Science & Technology (MST), Government of India, New Delhi.
Conflict of Interest
The authors declare that there are no conflicts of interest in this paper.
Source of Funding
Dept. of Science & Technology (DST), Ministry of Science & Technology (MST), Government of India, New Delhi.
References
- Hebb AO, Darvas F, Miller KJ. Transient and state modulation of beta power in human subthalamic nucleus during speech production and finger movement. Neuroscience. 2012;202:218-33. [Google Scholar] [Crossref]
- Darvas F, Hebb AO. Task specific inter-hemispheric coupling in human subthalamic nuclei. Front Human Neurosci. 2014. [Google Scholar] [Crossref]
- Powell D, Hanson N, Threlkeld AJ, Fang X, Xia R. Enhancement of parkinsonian rigidity with contralateral hand activation. Clin Neurophysiol. 2011;122(8):1595-601. [Google Scholar] [Crossref]
- Katzner S, Nauhaus I, Benucci A, Bonin V, Ringach DL, Carandini M. Local origin of field potentials in visual cortex. Neuron. 2009;61(1):35-41. [Google Scholar]
- Santaniello S, Fiengo G, Gtielmo L, Grill W. Closed- loop control of deep brain stimulation: A stimulation study. IEEE Trans Neural Syst Reh Eng. 2011;19(1):15-24. [Google Scholar]
- Kempf F, Kühn AA, Kupsch A, Brücke C, Weise L, Schneider GH. Premovement activities in the subthalamic area of patients with Parkinson’s disease and their dependence on task. Eur J Neurosci. 2007;25(10):3137-45. [Google Scholar] [Crossref]
- Celka P, Colditz P. A computer-aided detection of EEG seizures in infants: a singular-spectrum approach and performance comparison. IEEE Trans Bio Eng. 2002;49(5):455-62. [Google Scholar]
- Muller-Putz GR, Kaiser V, Solis-Escalante T, Pfurtscheller G. Fast set-up asynchronous brain-switch based on detection of foot motor imagery in 1-channel EEG. Med Biol Eng Comput. 2010;48(3):229-33. [Google Scholar]
- Donchin E, Spencer KM, Wijesinghe R. The mental prosthesis: Assessing the speed of a P300-based brain-computer interface. IEEE Trans. Neural Syst Reh Eng. 2000;8(2):174-9. [Google Scholar]
- Panicker RC, Puthusserypady S, Sun Y. An asynchronous p300 bci with ssvep-based control state detection. IEEE Trans Bio Eng. 2011;58(6):1781-8. [Google Scholar]
- Subasi A, Ercelebi E. Classification of EEG signals using neural network and logistic regression. Computer Methods Programing Biomed. 2005;78(2):87-99. [Google Scholar]
- Wu T, Yang B, Sun H. EEG classification based on artificial neural network in brain-computer interface. Life System Modeling Intelligent Comput. 2010;97:154-62. [Google Scholar]
- Li J, Zhang L, Tao D, Sun H, Zhao Q. A prior Neurologistic knowledge free tensor-based scheme for single trial EEG classification. IEEE Trans. Neural Syst Reh Eng. 2009;17(2):107-15. [Google Scholar]
- Niketeghad S, Hebb AO, Nedrud J, Hanrahan SJ, Mahoor MH. Motor task event detection using subthalamic nucleus local field potentials. . 2015. [Google Scholar]
- Niketeghad S, Hebb AO, Nedrud J, Hanrahan SJ, Mahoor MH. Single trial behavioral task classification using subthalamic nucleus local field potential signals. . 2014. [Google Scholar]
- Zaker N, Dutta A, Maurer A, Zhang JJ, Hanrahan SJ, Hebb AO. Adaptive Learning of Behavioral Tasks for Patients with Parkinson’s Disease Using Signals from Deep Brain Stimulation. . 2014. [Google Scholar]
- Zhang X, Mahoor MH, Mavadati SM. Facial expression recognition using lp-norm MKL multiclass-SVM. Machine Vision Appl. 2015;26:467-82. [Google Scholar]
- Gönen M, Alpaydin E. Multiple kernel learning algorithms. J Mach Learn Res. 2011;12:2211-8. [Google Scholar]
How to Cite This Article
Vancouver
Raju VR, Balmuri KR, Srinivas K, Madhukar G. An advance of complex multifaceted root learning for parkinson behavior using subthalamic nucleus biomarkers [Internet]. IP Indian J Neurosci. 2021 [cited 2025 Nov 09];7(3):231-236. Available from: https://doi.org/10.18231/j.ijn.2021.041
APA
Raju, V. R., Balmuri, K. R., Srinivas, K., Madhukar, G. (2021). An advance of complex multifaceted root learning for parkinson behavior using subthalamic nucleus biomarkers. IP Indian J Neurosci, 7(3), 231-236. https://doi.org/10.18231/j.ijn.2021.041
MLA
Raju, Venkateshwarla Rama, Balmuri, Kavitha Rani, Srinivas, Konda, Madhukar, G.. "An advance of complex multifaceted root learning for parkinson behavior using subthalamic nucleus biomarkers." IP Indian J Neurosci, vol. 7, no. 3, 2021, pp. 231-236. https://doi.org/10.18231/j.ijn.2021.041
Chicago
Raju, V. R., Balmuri, K. R., Srinivas, K., Madhukar, G.. "An advance of complex multifaceted root learning for parkinson behavior using subthalamic nucleus biomarkers." IP Indian J Neurosci 7, no. 3 (2021): 231-236. https://doi.org/10.18231/j.ijn.2021.041