Decoding of Kinetic and Kinematic Information from Electrocorticograms in Sensorimotor Cortex

Brain-machine interfaces (BMI) are useful technologies to provide assistance to disabled individuals, allowing them interaction with their environments. A number of prominent brain-machine interface studies have arisen over the past two decades. These BMI systems translate brain signals into commands for controlling devices such as cursors [1], spelling devices [2], and neural prosthetics [3-9]. This new communication has not only the potential to help to disabled persons but also provide insight into the motor system of the brain [10-14].

and viscosity of joints. Decoding the kinematic and kinetic information from the neural activity is necessary to implement a human-like BMI system. The schematic outline of this concept and achieved studies are shown in Figure 1.
This paper introduces the preprocessing algorithm to decode the kinetic and kinematic information from ECoG signals in time series. Using this novel method, we could predict muscle activities (kinetic) and joint angles (kinematic) of shoulder and elbow joints. We also discuss three questions: which locations are most effective area for decoding, how different numbers of effective electrocorticography signals affect decoding performance, and which frequency band is most effective?

Ethics statement
We got the monkey ECoG data from the National Institutes of Natural Sciences and the human ECoG data from Osaka University Hospital in Japan. The local ethics committee of the National Institutes of Natural Sciences (Approval No.: 11A157) and Osaka University Hospital (Approval No.08061) approved each experiment. The monkeys' welfare and steps taken to ameliorate suffering were in accordance with the recommendations of the Weather all report, "The use of non-human primates in research. " Human experiment conducted in accordance with the Declaration of Helsinki. ECoG electrodes were embedded not for our experiments but for patients' medical treatments. The ECoG arrays were implanted in the intracranium for two weeks to determine the optimum site for effective pain reduction (patients1 and 2) or epileptic foci localization (patient 3). All patients or their guardians gave written informed consent for the use of their data in the academic study. Experiment 1: Monkey data: Two Japanese macaques (Monkey A: male, at 8.9 kg; Monkey B: female, at 4.7 kg) were trained to perform reaching and grasping tasks with the right hand as shown in Figure  2A. The monkeys performed these tasks repeatedly and continuously for over 700 s. Monkey A performed a total of 134 trials, and monkey B performed 248 trials. We chronically implanted a platinum ECoG array (Unique Medical Corporation, Tokyo, Japan) over the left M1, which contained 15 (monkey A: 5×3 grid) and 16 (monkey B: 4×4 grid) channel electrodes.
We recorded ECoG signals with 4 kHz sampling using an acquisition processor system (Plexon MAP System; Plexon, Inc., Dallas, US) and EMG activities of the right forelimb muscles implanted pairs of multistranded stainless steel wires (Cooner Wire, Chatsworth, CA, USA). The 3D-positions of various points of the right arm were recorded using reflective markers tracked with an optical motion capture system (Eagle Digital System; Motion Analysis Corporation, Santa Rosa, CA). The neural data were down-sampled to 500 samples per second, and the motion data were up-sampled to 500 samples per second to match the neural data. The previous work showed the detail experimental information [36]. Experiment 2: Human data: All patients were seated upright on a chair at a table and were asked to perform the tasks using their left hands as shown in Figure 1B. They asked to replace three blocks to vacant corners of the square around a 25 cm × 25 cm, one by one in a clockwise fashion (patient 1), random choose (patient 2), and an arbitrary positioning (patient 3). Patients 1 and 2 were implanted with two 5 × 6 electrode arrays, and patient 3 was implanted with a 3×5 array. ECoG signals were recorded inside an electromagnetically shielded room with a 128-channel digital EEG system (EEG 2000; Nihon Koden Corporation, Tokyo, Japan) set at a sampling rate of 1000 Hz. 3D arm motions were recorded at a sampling rate of 100 Hz with an optical motion capture system (Eagle Digital System; Motion Analysis Corporation, Santa Rosa, CA). Nakanishi et al. [32] showed the experimental setup in detail.
Decoding method: ECoG signals were pre-processed with our previously proposed method [32,33,36]. Firstly, the signal data were rereferenced with a common average reference (CAR) and divided into A) Monkeys performed sequential right arm and hand movements, which consisted of reaching to a knob, grasping the knob with a lateral grip, pulling the knob closer, releasing the knob, and returning the hand to the home position, in a 3-D workspace [36]. During the task, ECoG and EMG signals were recorded simultaneously. B) Patient 1 replaced three blocks one by one and clockwise (green arrows) at the corners of a 25 cm × 25 cm square [32]. ECoG signals were obtained with planar-surface platinum grid electrodes placed on the right sensorimotor cortex. Half-closed circles represent3D markers for the motion capture system. The angles q1, q2, q3, and q4 are defined as an abduction/adduction angle, a flexion/extension angle, an external/internal rotation at the left shoulder joint, and a flexion/extension angle at the left elbow joint, respectively. When he lowered his arm toward the -z direction and turned his palm to the y direction with the elbow extended, q1, q2, and q3 were all zero, and q4 was � radians. rectified and smoothed with a Gaussian filter (width: 0.1 s, σ: 0.04 s), which changed high oscillations into low frequency features. Thirdly, the signals were down sampled to 100 Hz, i.e., the sampling rate of the motion capture recordings. Finally, the obtained signals x i (t)(i=1, 2, … , n×7 or n×9) at time t were normalized to the standard z-score z i (t) as follows.   Dotted blue lines are actual muscle activities from EMG signals and solid red lines are decoded muscle activities from ECoG signals over a 50 s time interval [36]. Both lines were normalized to the ranges of actual muscle activities. The normalized root mean square error (nRMSE) and coefficient of determine (R 2 ) are also shown.
where m i , s i and n denote the mean value of x i (t), the standard deviation of x i (t), and the number of ECoG channels, respectively. These z-scores calculated from ECoG signals were utilized as training data to construct a decoder. Figure 3 shows an example trial including frequency band features of the ECoG signals, rectified raw EMG signals, grip force, and logical signals. We used the sparse linear regression (SLiR) or the Partial least squares regression (PLS) algorithm to determine the weight for prediction.

Decoding of muscle activities from ECoG signals
The neuromuscular system naturally modulates mechanical stiffness and viscosity of arm to achieve proper interaction force to the environments. Stiffness, viscosity and force of joints change with muscle activation. Therefore, decoding muscle activities are key components for realizing neuro-prosthesis capable of the interaction with environments. We verified that ECoG signals are effective for predicting muscle activities in time varying series when performing sequential movements [36]. We used sparse linear regression to find the best fit between frequency bands of ECoG and electromyographic activity. We applied the prediction model to continuous data from an additional session by monkey B. One example of continuous prediction is shown in Figure 4, where the prediction was stable even for repetitive trials over 50s. In the results of the 5-cross validation, Mean and standard deviation (STD) of the coefficient of determination (R 2 ) and nRMSE for each muscle ranged from 0.02 ± 0.006 to 0.63 ± 0.003 (R 2 ) and 0.13 ± 0.005 to 0.18 ± 0.01 (nRMSE). These results could demonstrate the feasibility of predicting muscle activity from ECoG signals in an online fashion. Recently, we succeeded in decoding grasp force profile during reaching and grasping tasks [37].

Decoding of hand trajectory from ECoG signals
We also succeed in decoding 3 dimensional hand positioning from ECoG signals using the proposed preprocessing algorithm and PLS regression [33]. To determine the most effective areas for prediction, we calculated performance values (R 2 ) using only individual electrode. Performance details of two electrode selection methods are shown in Figure 5. For both monkeys, performance was improved quickly as the number of electrodes used increased from 1 to 6. The performance curves fluctuated only slightly when using 9 electrodes and above. The best R 2 values were achieved using 15 and 11 electrodes for monkeys A and B, respectively. Higher performance electrodes are concentrated at the lateral areas and near areas of central sulcus (CS). Our results  Figure 6: Examples of the predicted and actual 3D trajectories [32]. Markers (circles, triangles, squares, and diamonds) represent 2 s time intervals. Circles and diamonds indicate the earliest and the latest positions, respectively. The red trajectories were computed using predicted data q1~q4 and patient 1's actual arm length. The timings (positions of the markers) and trajectory curves of the predicted data were similar to those of the actual data. indicated that 3D hand trajectories can be predicted using nine or ten ECoG signals and that ECoG electrodes with higher performance were concentrated at the lateral areas and areas close to CS.

Decoding of joint angles from ECoG signals
We also predicted 3D angle trajectories in time series from ECoG signals in humans using theproposed preprocessing method and a sparse linear regression [32]. Figure 6 is an example of the comparison between predicted (red lines) and actual 3D trajectories (blue lines) for six seconds in the 10th trial of session 2 by patient 1.

Most effective location for decoding
Carmena et al. [10] reported that neuron activity recorded from Ml showed greater efficacy than that from dorsal premotor cortex, supplementary motor cortex, posterior parietal cortex, and primary somatosensory cortex. In our previous work [33], it is clearly shown that the electrodes in primary motor area are most contributing to decode among the premotor area, primary sensory area, and primary motor areaas shown in Figure 4A-4D. Within primary motor area, however, we could not find experimental evidences to explain the most effective site for force prediction according to anatomical knowledge. Our results just found that ECoG signals from the lateral areas and near areas of CS showed greater efficacy in prediction [33,36]. It might be needed the micro-sized ECoG electrode to find the most effective location within primary motor area.

Most effective number of electrodes
For both monkeys, performance improved quickly as the number of electrodes used increased from 1 to 9 as shown in figure 4E. The performance curves fluctuated only slightly when using 10 electrodes and above. Best decoding performance was achieved using a relatively small number of electrodes, 13 and 10 electrodes in the performancebased selection for monkey A and monkey B, respectively. These trends are similar to the results of a previous neuron activity-based study [38], which selected different numbers of high sensitivity neurons in decoding kinematic variables. We note that decoding performance is not simply related to the number of electrode but may more closely depend on the higher density electrodes within the effective areas. Nevertheless, a small number of electrodes would allow for lower power consumption, extending the usage time for wireless ECoG-based BMIs [39,40].

Most effective frequency band for decoding
Most EEG-based BMI studies have used one or two sensorimotor rhythms such as μ (8~12 Hz) orβ (14~30 Hz) oscillations because the γ (>30 Hz) rhythm is often inconspicuous and neglected with a low pass filter. In ECoG-based BMIs, however, the γ rhythm has been widely used. We identified the useful ECoG frequency bands to decode kinetic and kinematic information. Analysis of the weight values for the frequency bands showed that contributions by the δ, γ, and β bands were significantly larger than those of the other bands [33,36]. This result corresponds to previous studies as well [27,[41][42][43][44][45]. Especially, the γ band was most effective than any other bands because γ band activity of ECoG signals reflect the unit activity in layers V/VI in primary motor area [46].

Conclusion
This study introduced the novel attempt to decode muscle activities, hand trajectories, and joint angles from a small number of ECoG signals. This approach offers important insight regarding the presence of kinetic and kinematic information in ECoG signals to predict timevarying their information, whereas previous ECoG-based studies have tried to classify direction or intention of movement. The primary advantage of the proposed method is that it can predict muscle activities and joint angle during sequential movement tasks. If we can predict muscle activities, joint torque and stiffness can also be predicted using previously proposed methods [47,48]. This creates remarkable benefits, which would contribute to the realization of ECoG-based prosthetics. We foresee this method contributing to future advancements in neuroprosthesis and neuro-rehabilitation technology.