Linear and Nonlinear Kinematic Synergies in the Grasping Hand

Kinematic synergies in human hand movements have shown promising applications in dexterous control of robotic and prosthetic hands. We and others have previously derived kinematic synergies from human hand grasping movements using a widely used linear dimensionality reduction method, Principal Component Analysis (PCA). As the human biomechanical system is inherently nonlinear, using nonlinear dimensionality reduction methods to derive kinematic synergies might be expected to improve the representation of human hand movements in reduced dimensions. In this paper, we derived linear and nonlinear kinematic synergies from linear (PCA), globally nonlinear (Isomap, Stochastic Proximity Embedding (SPE), Sammon Mapping (SaM), and Stochastic Neighbor Embedding (SNE)) and locally nonlinear (Local Linear Embedding (LLE), LaplacianEigenmaps (LaE), and Local Tangent Space Alignment (LTSA)) dimensionality reduction methods. Synergies derived from linear PCA and nonlinear SaMwere able to capture multiple functional postures and physiological patterns. Results from natural hand grasping movements indicated that PCA performed better than all nonlinear dimensionality reduction methods used in the paper. Results from ASL postural movements indicated that PCA, SaM, and SPE better generalized over ASL postural movements when compared to other methods. Overall, our results show that PCA derived synergies offer qualitative and quantitative advantages over nonlinear methods as a limited number of kinematic synergies begin to be implemented in human prosthetics. Linear and Nonlinear Kinematic Synergies in the Grasping Hand


Introduction
The hand is tasked with creating a multitude of postures in everyday life in order to grasp, use, and manipulate objects. With more than 25 degrees of freedom (DoF) [1], the hand is well equipped to dexterously achieve these tasks. It is hypothesized that the central nervous system (CNS) calls upon a few key movement patterns when controlling the hand; this has been supported by neural [2], muscle [3,4], and kinematic studies [5,6,7] of reach and grasp movements [8]. For example, a cup is often grasped with a similar hand motion irrespective of position or size. These few movement patterns are known as synergies.
Synergies, or movement primitives, are viewed as fundamental building blocks of movement [9,10]. Using synergies, the CNS can simplify control of high DoF limbs: instead of controlling individual DoF, the CNS controls a few synergies that each encodes multiple DoF. Synergies, especially in the human hand present a unique testing environment for two reasons: (1) the human hand has the most DoFs in the body and (2) we and others have observed that hand movements in activities of daily living (ADL) involving hand grasping can be reconstructed with 90% accuracy using six synergies [7,11]. These six synergies were computed using principal component analysis (PCA) and accommodated for more than 90% of the variance in the joint kinematics. This means that high dimensional control (25 DoF) control can potentially be reduced to low dimensional control (6 functional DoF).
In the above example, PCA was used as a form of dimensionality reduction in order to determine synergies. A comparison of performance was made between PCA and unsupervised linear discriminant analysis (ULDA) we found that PCA outperformed ULDA [12]. There are other dimensionality reduction methods, both linear and nonlinear, that may extract the functional patterns of human hand motion. While a number of dimensionality reduction techniques have been comparatively evaluated on extraction of muscle synergies [13], their efficiency in a different task space, kinematics, has not been evaluated. Inspired by the above studies, in this paper, linear and nonlinear dimensionality reduction methods were used to derive kinematic synergies in hand movements. The aims of this paper were twofold (i) To compare the performance of linear synergies (i.e., synergies derived from linear dimensionality reduction methods) with the performance of nonlinear synergies (i.e., synergies derived from nonlinear dimensionality reduction methods) in representing and generalizing hand movement kinematics. To this end we used linear (PCA), globally nonlinear (Isomap, Stochastic Proximity Embedding (SPE), Sammon Mapping (SaM), and Stochastic Neighbor Embedding (SNE)) and locally nonlinear (Local Linear Embedding (LLE), LaplacianEigenmaps (LaE), and Local Tangent Space Alignment (LTSA)) dimensionality reduction methods. (ii) To see whether the subtle nonlinearities of human hand grasping kinematics can be captured by nonlinear methods with increased precision when compared to linear methods. Comparatively exploring multiple dimensionality reduction methods allows us to determine which techniques are appropriate for identifying patterns in experimental datasets. The analysis also highlights important physiological patterns based on the assumptions of each method.
Researchers mimicking the CNS in various applications like brain-computer interfaces (BCI) [14], humanoid robotics [15,16], and neuroprosthetics [17] have turned to synergies as a means of operating high-DoF machines with simpler control. This is accomplished by identifying common patterns of joint velocities, or kinematic synergies, which can be invoked using a single control signal. However, the useful application of kinematic synergies for biomimetic prosthesis control relies on their correct derivation. This derivative analysis further establishes which synergies, or properties of synergies, should be implemented in biomimetic prosthesis.

Experiment and materials
Ten healthy, right handed subjects participated in this experiment after signing the consent forms approved by the institutional review board (IRB) of the University of Pittsburgh. The experiment consisted of three phases. In the first phase, subjects were instructed to rapidly grasp 50 objects, one at a time. Objects were placed 40 cm in front of the subject. This was repeated for the same 50 objects, and thus the whole training phase obtained 100 rapid grasps. A typical task consisted of grasping an object. Start and stop times of each task were signaled by computer-generated beeps. In each task, the subject was in a seated position, resting his/her right hand at a corner of a table and upon hearing the beep, grasped the object placed on the table. At the time of the start beep, the hand was in rest posture, and then the subject grasped the object and held it until the stop beep. In the second phase, subjects were instructed to grasp 50 objects naturally (slower than the rapid grasps) twice each. The same setup as Phase I was used. In the third phase, to test the generalizability of the synergies over a broad range of postures, subjects were also asked to pose 36 (10 numbers and 26 alphabets) American Sign Language (ASL) postures. Here, subjects started from an initial posture and stopped at one ASL posture. The 100 rapid grasps from phase 1 were used in deriving synergies using linear and nonlinear dimensionality reduction methods. The 100 natural grasps from phase 2 and the 36 ASL postures from phase three were used to evaluate the derived synergies.
In the experiment we used a CyberGlove (CyberGlove Systems LLC, San Jose, CA, USA) equipped with 22 sensors that captured hand movements at a sampling frequency of 86 Hz. In this paper, we only considered 10 of the sensors which measure the angles of the metacarpophalangeal (MCP) and interphalangeal (IP) joints of the thumb and the MCP, and proximal interphalangeal (PIP) joints of the other four fingers. Distal interphalangeal joint angles are approximately 2/3 of PIP joint angles [18]. Since its relationship can be estimated, when looking for overall hand patterns, DIP joints can be omitted. The CMC joint, although necessary for grasping movements, is a complex multi-dimensional joint that is often simplified in prosthetic applications. We therefore, omitted it from this analysis, focusing on flexion and extension in all fingers. Abduction/adduction angles for application reasons were also not included. The palm arch sensor was omitted as it is not a naturally controlled DoF. Wrist sensor was not included as the current study focused on hand grasping and shaping and not orientation. Thus, the remaining 10 joints capture most variable characteristics of the hand in grasping tasks. We used several objects of different shapes (spheres, circular discs, rectangles, pentagons, nuts, and bolts) and different dimensions (spheres: 1-5 cm in radius; discs: 2-10 cm in radius; rectangles and pentagons: 1-3 cm each side; nuts and bolts: 2-5 cm in length) in the grasping tasks. This allowed us to test the general shapes and sizes of objects encountered is activities of daily living. Real objects, although used in some experimental setups, require increased precision using individual joint movements; this would diminish the generalizability of derived synergies.

Preprocessing
Rapid grasping movements from the first phase were used in deriving synergies. We assume that these synergies are a result of direct impulses from the CNS. By minimizing reaction time and continuous sensory feedback, the resulting task space is driven by synchronous weighted synergies [7]. The natural movements and ASL postural movements were reconstructed using the derived synergies.
First, we calculated angular velocities using the derivative of joint angle profiles collected in the experiment. Flexion is positive and extension is negative; subjects started in a flat hand resting position. We preserved only the relevant projectile movement about 0.45 second or 39 samples under a sampling rate of 86 Hz. This time segment represents movement onset to grasp completion, detected by a constant zero velocity at all digits. Second, we constructed an angular-velocity matrix, V, for each subject. Angular velocity profiles for the ten joints recorded during one grasp trial were concatenated into the same row. Each row, therefore, represented one object grasp, g, in time. The matrix had 100 rows (corresponding to 100 rapid grasping tasks from first phase) and 39 × 10 = 390 columns: represents the angular velocity of joint ( ) 1, ,10 in the g-th grasping task (g = 1, ..., 100).

Dimensionality Reduction
We then performed linear and nonlinear dimensionality reduction on the above angular velocity matrix, V, to derive linear and nonlinear kinematic synergies. For PCA, Statistics Tool Box in MATLAB was used, and for all other global and local nonlinear methods drtoolbox by van der Maaten LJP was used [19]. All the methods used in the paper are briefly described [20,21].
Linear method: Principal Component Analysis: We first began with linear method PCA. We implemented PCA using SVD [20].The angular velocity matrix V was factorized to three matrices U, Σ and S as shown below.
where U is a 100-by-100 matrix, which has orthonormal columns so that U ′U = I 100×100 (100-by-100 identity matrix); S is a 100-by-390 matrix, which has orthonormal rows so that SS′ = I 100×100 ; and Σ is a 100-by-100 diagonal matrix: diag{λ 1 , λ 2 , ..., λ 100 } with  The approximation matrix Ṽ can be written in a more compact form: where U m is a 100-by-m matrix containing the first m columns of U and S m is a m-by-390 matrix containing the first m rows of S. Then each row of S m is called a principal component (PC), and the product U m diag{λ 1 , ... , λ m } is called the weight matrix.
For easy comparison, the elements of S m, in a way similar to (1), were written as: The angular-velocity profiles (obtained by rearranging all joints row-wise for the PCs) can be viewed as synergies.
To decide m, the number of PCs or synergies that we want to use in reconstruction of the testing movements, we consider an index defined as

+ +¼+ + +¼+
This index indicates the fraction of total variance of the data accounted by the PCs. Here 95% variance was achieved with six PCs in all subjects (Table 1).

Global nonlinear methods:
Global dimensionality reduction methods try to retain the global properties of the data. PCA and classical multidimensional scaling (MDS) are such global linear methods. The difference between these methods and nonlinear methods comes from nonlinear transformations of high-dimensional data to low dimensions. a) Sammon mapping: Classical MDS maps high-dimensional space to low-dimensional space by preserving the pairwise inter-point distances of the data [22]. The cost function, also known as stress function, is the Euclidean distance between high-dimensional data points. This projection can be represented as a linear combination of original variables. Sammon Mapping (SaM) is very similar to classical MDS but this method preserves the structure of inter-point distances using a nonlinear stress function [23],ψ, as shown below.
( ) Where x i and x j are the original data points and l i and l j are data points in the low dimensional space.
indicates the Euclidian distance between the data. The minimization of the stress function was performed using an iterative gradient descent method in this paper.

b) Isomap:
Isomap is similar to multidimensional scaling (MDS) [19]. Although Isomap does linear estimations in the data point neighborhoods, the synergies extracted are nonlinear because these small neighborhoods are stitched together without maintaining linearity. The following were the steps involved in estimating nonlinear synergies using Isomap [23] (i) Define neighbors for each data point where N is number of data points and d is Kronecker delta function. c) Stochastic embedding: Stochastic Proximity Embedding (SPE) by Agrafiotis [24] is a self-organizing algorithm that attempts to generate low-dimensional euclideanembeddings, similar to classical MDS and Sammon mapping. The method iteratively refines by selecting pairs of random data points and adjusts their coordinates to optimize the stress function that is similar to in SaM.

d) Stochastic neighbor embedding (SNE)
: SNE is also similar to MDS. The difference is the distance measure and stress function that is optimized [25]. The similarities of nearby points contribute to the stress function, thus the method preserves local properties of the data.

Local nonlinear methods
Local methods preserve the properties of high-dimensional data in small neighborhoods around the data points. Three methods were considered in this category. a) Local linear embedding: Locally Linear Embedding (LLE), as the name suggests, tries to find a nonlinear manifold by stitching together small linear neighborhoods [26]. This is very similar to Isomap. The difference between the two algorithms is in how they interpolate the small linear neighborhoods. Isomap does this by doing a graph traversal by preserving geodesic distances while LLE does it by finding a set of weights that perform local linear interpolations that closely approximate the data. The following were the steps involved in estimating nonlinear synergies using LLE [2]: (i) Define neighbors for each data point (ii) Find weights that allow neighbors to interpolate original data accurately (iii) Given those weights, find new data points that minimize interpolation error in lower dimensional space. b) Laplacian eigenmaps: LaplacianEigenmaps (LaE) is a geometrically motivated algorithm, similar to LLE. This method is widely used in locality preserving and clustering applications. The following cost function of pairwise distances is defined as an eigenproblem using spectral graph theory [19]: wherel i and l j are data points in the low dimensional space. ij corresponds to weights assigned to them. Large weights correspond to nearest original data points, and thus, near-by points in the high dimensional space are placed as close as possible in the low dimensional space. where c jk is the weight of the j-th synergy with a shift of t jk . The matrix B can be viewed as a bank or library of template functions with each row of B as a template. The following optimization problem was used in selection of synergies in reconstruction of a particular movement.
where 1 || · || represents the l 1 norm, 2 || · || represents the l 2 norm or Euclidian norm of a vector and λ is a regulation parameter.

Results
Linear and nonlinear (globally nonlinear and locally nonlinear) dimensionality reduction methods presented in Section II-C were used to derive kinematic synergies from preprocessed hand kinematics measured from Phase 1 of the experiment. Ten synergies were used to observe performance in reconstruction (Figures 1 and 2).
The kinematic synergies derived in this study characterize joint angular velocity patterns over grasp time. An example of the first six synergy profiles for subject 1 are shown in Figures 3 and 4 and the corresponding end postures are shown in Figure 5. The best performing methods, PCA and SaM, shown in rows 1 and 2, respectively, have synergies that cover a range of postures ending in flexion and/or extension across all digits. This performance could be attributed to capturing such physiological and functional movements. SPE, also performed relatively well; however, the method preserved only flexion and it lacked variability across synergies. As the weights of each synergy may be adjusted during reconstruction, capturing inherent physiological patterns of the hand is very important. For example, in the third synergy, LaE was able to capture an index-finger led flexion at all MCP joints. Extreme or nonphysiological movements were captured by synergies from other nonlinear methods. This might have led to their poor performance.
An example reconstruction of a naturally grasped object (wooden triangle) is shown in Figure 6. Each subfigure shows reconstruction c) Local tangent space alignment: Local Tangent Space Alignment (LTSA) describes local properties of data using local tangent space of each data point. The assumption behind choosing local tangent space is when the manifold is unfolded, the tangent hyperplanes become aligned. LTSA assumes a linear mapping between LTS and original high dimensional data and a linear mapping between LTS and low dimensional data. The linear mapping between high dimensional data and LTS, Ө i is calculated by PCA. Linear mapping from LTS to computed low dimensional space is assumed to be K i . The cost function to be minimized is given by the following equation [2]: Where L i represents low dimensional data representation to be computed. J represents the mean centering matrix.

Reconstruction of Natural and ASL Postural Movements
Ten synergies extracted from linear and nonlinear dimensionality reduction methods were used in reconstruction of natural and ASL movements. l 1 -norm minimization was used to optimally and sparsely select the synergies. This was similar to the methods in Dimensionality reduction in control and coordination of human hand [7]. Briefly, these were the steps involved in l 1 -norm minimization algorithm. Let us assume m synergies with a duration of t s samples (t s = 39 in this study) were obtained for a subject. Natural grasping movements, which are to be reconstructed, are longer. Consider a natural grasp's angularvelocity profile ( ) 82in this study T T = represents the movement duration (in samples). This profile can be rewritten as a row vector, denoted v row : Similarly, a synergy ( ) · j s can be written as the following row vector: We add Then we construct a matrix consisting of the row vectors of the synergies and all their possible shifts with 1 .
jk s t T t ≤ ≤ − With the above notation, we are trying to achieve a linear combination of synergies that can reconstruct the velocity profiles as in the following equation.   using synergies derived from a particular method. The normalized reconstruction errors were calculated by the ratio of the difference between the sums of squared errors of recorded and reconstructed joint angular velocities to the sum of squared recorded joint angular velocities at each sampled point. Figure 5 shows the mean reconstruction errors and standard deviations for 100 natural movements for 10 subjects across 10 synergies. Similarly, Figure 6 shows the mean reconstruction   Table 1 shows an example of the mean reconstruction errors (with standard deviations across 10 subjects) during 100 natural grasps and 36 ASL postural movements for all the eight methods using six synergies. PCA has the best overall performance with mean reconstruction errors of 0.0964 and 0.1813 for natural and ASL postural movements, respectively. In the nonlinear methods, SaM and SPE performed nearly close to PCA.

Linear vs. nonlinear synergies
The prime objective of this study was to determine if nonlinear methods should be further explored in order to determine human movement synergies. We applied linear and nonlinear dimensionality reduction methods to derive linear and nonlinear kinematic synergies from human hand grasping movements. We then used these synergies to reconstruct natural hand grasping movements that were similar to activities of daily living. To broaden the applicability of synergies we also tested them on ASL postural movements that were different from natural grasps. The results indicate that the kinematic synergies were efficiently represented by linear PCA when compared to nonlinear methods. The nonlinear methods used in the paper suffered from their own methodological weaknesses as discussed in the following sections. It is important to recognize that even though linear kinematic synergies were efficiently linearly combined in task space, execution of these synergies is most likely neither exclusively linear nor exclusively nonlinear, but a combination of both [27]. Furthermore, the hierarchy of motor control involves not only the motor cortex, sensorimotor cortex, cerebellum and spinal cord, but the end effectors as well including muscles and joints, which are all non-linear. Not only does this add to the dimensionality of the task space, but the relationship between different levels adds complexity. For example, in a study done by de Rugy et al. [27], linear muscle synergies were extracted from a wrist force aiming task using non-negative matrix factorization; muscle synergies mapped linearly to the task space within 20% of maximum force, and thereafter the mapping was not linear.

Linear vs. nonlinear methods
The results (Figures 1 and 2) (Table 1) suggest that nonlinear methods did not perform better than the linear PCA. Similar findings were reported by Clewley [19,28] van der Maaten LJP stating that PCA outperformed nonlinear methods for real world datasets. SaM and SPE performed close to PCA because inherently, these methods implement MDS in nonlinear space while PCA implements MDS in linear space. However, the drawback of SaMand SPE is computation time. Moreover, SaM involves gradient descent method that suffers from local minima. LaE, SNE, LLE, LTSA, and Isomap were next in the order of performance with worst being Isomap. The nonlinear methods suffered, in general, from not being able to capture the global properties of data which were grasped by PCA using variance. Both linear and nonlinear methods suffered during the reconstruction of ASL postural movements. The reconstruction errors increased significantly for PCA when compared to those for natural movements; nevertheless, the reconstruction errors were still to a major extent less than the tested nonlinear methods (with the exception of SaM performing slightly better for 9 synergies for ASL postures in Figure 2) ( Table 2).

Global vs. local methods
Global methods (PCA, SaM, SPE) performed better than local methods (LLE, LaE, LTSA) with the exception of Isomap and SNE. Note that PCA is grouped under global method as it is a global linear dimensionality reduction technique. Both global and local methods suffer from the following weaknesses. Locally nonlinear methods cannot represent the global properties of the data. In Figure 4 we observed that the reconstructions by LaE and LTSA were unable to capture the globally simple nature of T-MCP joint movement. Globally nonlinear methods often ignore the local properties of the data. In Figure 6 we observed that reconstructions by Isomap and SNE were unable to capture the local submovements in M-MCP joint movement. In general, global and local nonlinear methods that are based on neighborhood graphs, such as Isomap, SNE, LaE and LLE suffer from over-fitting the data and misunderstanding the outliers. Overall, the parametric nature of the nonlinear methods makes them susceptible to the curse of dimensionality.
In order to accurately represent data, the number of synergies chosen to include often depends on the amount of variance they described. There reaches a point where any additional synergies only represent unstructured noise, which is characterized by a progressive decrease in the slope. However, not all the methods used in this study are dependent on variance; thus, the reconstruction error seen in Figures 5 and 6 can show the significance of higher and lower order synergies. PCA, SaM, SPE, Isomap, and SNE show a decrease in error as the number of synergies used increases; however, the rate of error change begins to decrease at a certain point. This occurs at synergy 4 for PCA, synergy 6 for SaM and SPE, and synergy 7 for Isomap and SNE. However, LTSA, LaE, and LLE, all locally nonlinear methods, do not show a progressive decrease in the slope of error plots. Instead, lower order synergies cause a greater decrease in error than higher order synergies (e.g. slope between synergy 6 and synergy 7 is greater than the slope between synergy 5 and synergy 6 on the LaE curve in Figure 1).   Table 2: Lambda values, indicating variance accounted for by each of the synergies (S1-S10) derived from PCA for subjects 1-10.
Physiological interpretation of synergies 1) Task specificity: Task specific kinematic synergies have been derived in a number of studies. Synergies were derived using PCA on object manipulation tasks and were used in reconstruction of grasping tasks [29]. While ASL postures are not commonly learned or used postures, many of the ASL postures (e.g. letters A, C, E, F, M, N, O, S, T and numbers 0 and 9) are created using movements similar to those found in grasping. Individual finger movements cause deviation from synergy patterns resulting in increased reconstruction error. Their inclusion in this experiment allows us to test the generalizability of derived synergies to other tasks.

2) Behavioral movement patterns captured in synergies:
The patterns of synergies extracted from both linear and nonlinear methods reflected important characteristics of movement. The joint angular velocity profiles seen in synergies derived by PCA ( Figure 1) were very similar to characteristic velocity profiles observed during reach to grasp (meaning initial acceleration followed by deceleration) [30]. Synergies derived from LLE, LTSA, and, LaE lacked characteristic changes in velocities; instead many synergies showed only acceleration or only deceleration, particularly in MCP joints ( Figure 2). Velocity profiles in synergies derived from SNE, SaM, SPE and Isomap showed broad bell shaped velocity profiles as shown in Figure 1. In each synergy derived from PCA, Isomap, SaM and SNE, velocity patterns are similar across MCP joints and across PIP joints but there were differences in magnitude of velocity. LLE, LTSA, LaE, and SPE synergies were not able to preserve similarities across fingers.
During object grasping, the hand often over-extends during reach [31]. Thus, both extension and flexion need to be represented in derived synergies. All methods, with the exception of SPE, were able to capture both flexion and extension in one or more synergies. The first synergy extracted by PCA shows flexion in MCP joints preceding flexion in PIP joints. The overall movement is similar to a power grasp. This type of movement in kinematic synergies has also been captured by others [11,31,32].

Applications of Derived Kinematic Synergies
Synergies have applications in humanoid robotics and neuroprosthetics. Researchers have begun to translate synergies from physiological data to functional control of anthropomorphic and non-anthropomorphic prosthetics [33,34]. This involves modeling physiological grasping datasets with a dimensionality reduction technique and translating selected synergies onto a mechanical design. More recently, work has been dedicated to grasp structural properties (contact forces, object controllability) which are evaluated based on determined synergies and implementation [33][34][35][36][37][38]. The results of this study help us in selecting fewer optimal synergies to be inserted into such control schemes. For example, SPE, by preserving interpoint distance of high and low dimensional space, captured a cylindrical grasp well (better than PCA), but was not able to escape the cylindrical grasp pattern. Methods that prioritize on globally and commonly found patterns may be implemented in a robotic prosthetic hand; however synergies that still maintain physiological interfinger coupling may be necessary for hand exoskeletons. In the realm of neuroprosthetics, as researchers attempt to find neural correlates of synergies, these couplings are also important.
In summary, kinematic synergies in the grasping hand were extracted from linear and nonlinear dimensionality reduction methods. Movement reconstruction was achieved using the same optimal selection algorithm on synergies obtained from all methods. PCA performed well in movement reconstruction when compared to all nonlinear methods, except the global nonlinear methods SaM and SPE [39]. These methods performed nearly close to PCA. Graphical visualization of postural synergies derived from linear PCA and nonlinear SaM revealed functional and physiological patterns. Synergies extracted from global methods preserved characteristic velocity profiles while local methods failed. Consequently, global methods performed well when compared to local methods. Behavioral movement patterns were captured in synergies extracted from all global methods. Overall, our results show that PCA derived synergies have qualitative (functional and physiological postures) and quantitative (unsupervised and less computation load) advantages over nonlinear methods as a synergies begin to take prime place in prosthetics.

Conflict of Interests
All the authors declare that there is no conflict of interests regarding the publication of this paper.