Acquisition of Contact Force and Slippage Using a Vision-Based Tactile Sensor with a Fluid-Type Touchpad for the Dexterous Handling of Robots

This paper presents a new approach for estimating contact force and slippage by using a vision-based tactile sensor with a fluid-type touchpad for the dexterous handling of robots. The sensor consists of a CCD camera, LED lights, a transparent acrylic plate and a deformable touchpad. The sensor can obtain a variety of tactile information, such as the contact force, shape, contact region, position and orientation of an object in contact with the fluid-type touchpad. The previous method for measuring contact force requires extensive calculation; this paper proposes a new method based on lookup tables for measuring normal force, tangential force and moment, using a fluid-type touchpad. Additionally, we clarify the mechanism of slippage between a fluid-type touchpad and a contacted object. This mechanism is efficient for applying the proposed slippage estimation method to a fluid-type touchpad. The validation of the proposed methods is confirmed in the experimental results.


Introduction
Tactile receptors in the skin allow humans to sense multimodal tactile information such as the contact force, slippage, shape, position and orientation of a contacted object. Humans easily control their muscles by feeding back information from tactile receptors. Therefore, tactile sensing is a key factor in enabling robots to imitate skilled human behaviors. Precise control and dexterity in robots are due to information feedback from tactile sensors.
When considering practical applications, tactile sensors should meet three specific requirements. First, flexible sensor surfaces are optimal, as sensors should fit the object geometrically. Second, a simple structure is required for a compact robot. Third, in order to achieve multifunctional and dexterous robots, we need a sensor that allows the simultaneous acquisition of multiple types of tactile information.
Various tactile sensors have been developed using resistive, capacitive, piezoelectric, ultrasonic or electromagnetic sensing elements [1,2]. A sensor with strain gauges embedded in an elastic body has been proposed [3] for estimating the slippage of a contacted object. Noda et al. have developed a sensor including standing cantilevers and piezo resistors arrayed in orthogonal directions for detecting shear stress [4]. Schmitz et al. have arrayed twelve capacitance-to-digital converter (CDC) chips in an array on the body of each robot finger; these provide twelve 16-bit measurements of capacitance [5]. Hakozaki et al. have arrayed sensing elements on conductive rubber at regular intervals to measure three components of stress based on a capacitive method [6]. However, the key practical issues remain unresolved. Since these sensors require multiple sensing elements and complicated wiring, their structures are complex and cannot satisfy the second important requirement as described in the previous paragraph. A wirefree tactile sensor based on transmitters/receivers [7], and a sensor based on micro coils for changing impedance by contact force [8], were housed in complex structures. Although compact sensors using microelectromechanical system (MEMS) can be manufactured, the surfaces of these sensors are minimally deformable [9][10][11][12] and cannot satisfy the first requirement as described above.
However, vision-based approaches are extremely suitable for tactile sensors [13][14][15]. Typical vision-based sensors include the following two components: a deformable contact surface made of elastic material to fit its shape to contacted objects; and a camera to capture the deformation of the contact surface. Tactile information is acquired by analyzing the deformation of the surface. Compact vision-based sensors can be easily fabricated, as they do not require multiple elements or complex wiring in the contact region. Moreover, using elastic material does not decrease sensitivity in this type of sensor. K. Kamiyama et al. estimated a three-axis contact force by detecting the sensor body's deformation, using a two-layered dotted pattern and a charge-coupled device (CCD) camera [16,17]. The sensor reported in [18,19] consists of rubber sheets with nubs, a transparent acrylic plate, a light source and a CCD camera. Light travels through the transparent plate and is diffusely reflected where the nubs come in contact with the plate. The three-axis contact forces are obtained based on the intensity of the reflected light as captured by the CCD camera. Yamada arranged reflector chips on the surface of an elastic body [20]. In this case, deformation of the contact surface is estimated using the four corner positions of the reflector chips. However, these sensors can only detect one form of tactile reception. In order to achieve the dexterous handling of robots in a dynamically changed environment, various types of tactile information should be acquired simultaneously.
We have proposed a vision-based tactile sensor that can sense multiple types of tactile information simultaneously [21,22].The sensor consists of a CCD camera, LED lights, a transparent acrylic plate and a transparent hemispherical elastic touchpad for contacting the object. Here, the touchpad is deformable and the structure of the sensor is simple. Our proposed sensor satisfies the above requirements: simultaneous acquisition of various kinds of tactile information; simple structure; and a deformable surface for the sensor. This sensor measures normal force, tangential force and rotational moment based on preliminarily-constructed lookup tables, which relate the deformation of the elastic touchpad to force and moment [21]. This sensor can also estimate slippage between the touchpad and a contacted object [22].
Next, a tactile sensor using a fluid-type touchpad has been proposed for estimating the shape, contact region, position, orientation and contact force of an object in contact with the touchpad [23][24][25]. The surface of the fluid-type touchpad is made of a silicon rubber elastic membrane. The inside of the membrane is filled with translucent red-colored water. Implementation of the fluid-type touchpad has extended the sensing ability of our tactile sensor. However, the issue of how to estimate slippage using the fluid-type touchpad has not yet been addressed in these papers [23][24][25]. The difficulty is that the mechanism of slippage in the case of a fluid-type touchpad differs significantly from that of an elastic touchpad. Moreover, the method for measuring contact force by using a fluid-type touchpad requires extensive calculation [25], while the elastic touchpad can measure the contact force rapidly with the use of look-up tables [21]. In consideration of applications requiring rapid sensor response, the methods using lookup tables are much more efficient.
The purpose of this study is to estimate the contact force and slippage of an object by using a fluid-type touchpad. Firstly, we will propose a new method based on lookup tables for measuring normal force, tangential force and moment. Secondly, we will clarify the mechanism of slippage in the case of a fluid-type touchpad, comparing it to an elastic touchpad. Finally, the validation of our proposed methods will be confirmed in the experimental results. Figure 1 shows the configuration of a vision-based tactile sensor consisting of a CCD camera, LED lights, a transparent acrylic plate and a touchpad. The dimensions of the CCD camera and the LED lights are 8 × 8 × 40 mm and 60 × 60 × 60 mm, respectively. The touchpad is hemispherical, with a curvature radius and height of 20 mm and 13 mm. The two types of touchpads are presented in Figure 2: one is a transparent elastic touchpad, and the other is a semitransparent fluid-type touchpad. The elastic touchpad consists of a simple, transparent elastic body. The surface of the fluid-type touchpad is made of an elastic membrane constructed of silicon rubber; the inside of the membrane is filled with translucent, red-colored water. A dotted pattern is printed on the inside of the touchpad surface for observing the touchpad's deformation. Analysis of the deformations formed when the touchpad comes in contact with objects yields multimodal tactile information, using an image of the inside of the deformed touchpad captured by the CCD camera. Figure 3a and 3b show the captured images, sized 640 × 480 effective pixels, in the cases of the elastic touchpad and the fluidtype touchpad, respectively. This sensor can obtain multiple types of tactile information, including the shape, contact region, position and orientation of an object [23][24][25].

Method for measuring normal force
The previous method in [21] regarded the elastic touchpad and a contacted flat object as an elastic spherical object and a rigid flat object, respectively. The contact region in this case is a circle. Xydas et al. and Kao et al. analyzed the stiffness and contact mechanics of soft fingers based on the relationship of the Hertzian contact model [22,23]. Their approach calculates the relationship between the radius of a contact region and normal force, which is given as follows: Here, r c , c and f n are the radius of the contact region, the constant coefficient and normal force, respectively. The constant coefficient c depends on the mechanical properties of the touchpad, such as shape and stiffness. However, the above relation cannot be applied to the fluid-type touchpad because of the difference in structure. Therefore, we newly consider the contact between a fluid-type touchpad and a flat object based on the following approach. When the fluid-type touchpad comes in contact with the flat object in a normal direction, the object is subject only to the inner pressure of the contact region by the touchpad. Therefore, the equilibrium equation of the flat surface in a normal direction can be expressed as follows: ( Here, F z signifies normal force. p in and S c represent the inner pressure of the touchpad, and the contact area between the touchpad surface and the object, respectively. Note that the contact area S c and the inner pressure p in increase along with the depth D of the contact between the touchpad and the object, as shown in Figure 4. There is a non-linear relationship between the contact area S c , the contact depth D and the inner pressure p in . Moreover, the acquisition of inner  pressure p in is not straightforward, while the previous methods can obtain the contact area S c and the contact depth D in [23,24]. Therefore, we express p in as the function of S c or D, and thus normal force F z can also be given by the function of S c or D as follows: ( Here, F S and F D are the functions of S c and D, respectively. We approximate the functions F S and F D as quadratic functions.

Method for measuring tangential force
Tangential force was measured in the previous method by the displacement of the central dot and the radius of the contact region, using the elastic touchpad [21]. Figure 5a shows the displacement of the central dot when an object is in contact with the touchpad.
In the case of the fluid-type touchpad, we measure tangential force by using the displacement of the central dot, the contact area S c and the depth of the contact D with tuning parameters. Adapting the complex characteristics of the fluid-type touchpad is achieved by the tuning parameter which are newly introduced. Here, this relationship depends on normal force, transformed into the contact area S c or the depth of the contact D. When normal force is large, a tangential force is gained with reference to the displacement. In reverse, a smaller tangential force is generated with reference to the displacement when normal force is small. Therefore, we normalize the displacement of the central dot by using the contact area S c or the depth of the contact D with the tuning parameters as follows: Here, d 0 nor , d0, S ref and D ref are the normalized displacement of the central dot, the displacement before the normalization, the reference contact area and the reference depth, respectively. T S and T D are the tuning parameters for adapting the complex characteristics of the fluidtype touchpad, not used by the previous method in [21].

Method for measuring moment around normal direction
In a manner similar to the measurement of tangential force, the previous method using the elastic touchpad also measured moment in a normal direction perpendicular to the contact surface [21]. The rotation angle of the touchpad's surface yields the moment after identifying the relationship between the rotation angle of the contact surface and the moment. Figure 5b shows the rotation angle when an object is in contact with the touchpad.
When we use the fluid-type touchpad, this relationship also depend on normal force, transformed into the contact area S c or the depth of the contact D. Moment increased or decreased with reference to the rotation angle according to S c and D. Therefore, we normalize the rotation angle of the contact surface by using S c and D with the tuning parameters as follows: Here, θ nor and θ are the normalized rotation angle of the contact surface, and the rotation angle before the normalization, respectively. R S and R D are the tuning parameters for adapting the complex characteristics of the fluid-type touchpad, not used by the previous method in [21].

Definition of stick ratio
In order to achieve dexterous handling by robot hands, it is necessary to obtain not only contact force, but also information on slippage between a touchpad and a grasped object. Moreover, it is difficult for a robot to control grip force before macroscopic slippage occurs unless the sensor can predict the slippage. The grasped object may drop due to a delay in controlling the grip force. If macroscopic slippage occurs, a bigger grip force is required to stop slippage because the friction coefficient decreases in a dynamic situation.
The slippage degree between a touchpad and a grasped object has been focused on in [26,27]. Slippage degree refers to the various slip phases, such as perfect sticking, incipient slippage and macroscopic  slippage. The contact state between an elastic sensor and an object is in a state of continuous change from a sticking phase to a slipping phase until macroscopic slippage occurs. When tangential force is small, the elastic sensor surface is partially deformed and incipient slippage occurs in the contact region. Incipient slippage increases along with tangential force until macroscopic slippage finally occurs. If incipient slippage is detected, a robot can keep grasping an unknown object and prevent it from slipping, even if the mass and friction coefficients are unknown. Watanabe et al. proposed a method to calculate and maintain appropriate grip force on an unknown object by feeding back the slippage degree, based on proportional control [28]. When the slippage degree indicates perfect sticking, increasing the grip force should be avoided because it may crush the object. When incipient slippage becomes evident, the grip force should be increased before macroscopic slippage occurs.
In order to evaluate the method for estimating slippage degree, and to apply it to various systems, we first must represent the slippage degree quantitatively. The area ratio of the stick region to the total contact region is called the stick ratio ϕ, which is defined in [21,22] as follows: Here, S s is the area of the stick region, and S c is the area of the contact region. When ϕ = 1, the contact region exists perfectly in the sticking state. As we increase the ratio of tangential force divided by normal force, the stick ratio then decreases. A slippage region appears in the contact region and gradually increases. This state is called incipient slippage. When incipient slippage progresses and the entire contact region becomes the slippage region, the slippage of the object is macroscopically visible and ϕ = 0. This state is called macroscopic slippage. The stick region and the slippage region are graphically represented as shown in Figure 6. In the following section, the estimation of the stick ration is presented.

Method for estimating stick ratio
We obtained the stick ratio from captured images of the surface of the touchpad. We used the relative displacement of each dot to the object between the reference image and the current image. The reference image is a previously captured image, such as the initial image taken before testing, which is appropriately updated [22]. Figure  7 shows the displacement of the dots between the reference and current images. If the relative displacement of a certain dot to the object is zero, the dot remains in its original position relative to the object. Otherwise, the dot slips on the object. Here, the displacement of the object can be approximated as the displacement of the central dot in the dot pattern [21]. Therefore, the dot k satisfies the following inequality as regarded in the stick region: Here, d 0 and d k are the displacement of the central dot and the dot k, respectively, between the reference image and the current image. δ is the minute threshold value. Next, we approximate the two regions Ss and Sc as the number of contacting and sticking dots. Therefore, the stick ratio is obtained in [22] as follows: Here, N c and N s represent the number of the dots in the contact region, and the number of the dots satisfying (7), respectively. When estimating the stick ratio, each dot in the contact region is evaluated based on whether it satisfies (7) between the reference image and the current image.
In the following section, we consider the mechanism of slippage in order to apply this estimation method to the fluid-type touchpad.

Mechanism of slippage degree
In order to estimate slippage degree, tactile sensors must satisfy the structure generating incipient slippage. This means that the stick region and slippage region occur simultaneously in the contact region. For example, if rigid objects are in contact with each other, macroscopic slippage occurs immediately, without incipient slippage, since the objects are barely deformed. When we used the elastic touchpad, we confirmed that the touchpad's structure generated incipient slippage based on Hertzian contact [29]. Here, Hertzian contact addresses the contact between an elastic spherical object and a rigid flat surface. Therefore, Hertzian contact is inefficient when considering fluid-type touchpads, as a fluid-type touchpad consists of a membrane surface and inner liquid.
In order to understand the mechanism of slippage occurring with the fluid-type touchpad, we compartmentalize the surface of the touchpad into many small segments, as shown in Figure 8. We consider the theory of small segments on the surfaces of both the elastic and fluid-type touchpads from a mechanics viewpoint. We also focus on the differences in the small segments between the elastic and fluid-type touchpads.
We formulate the balance of force of a small segment from the viewpoint of the static mechanics, dividing the sensor surface into an I × J array of segments. Figure 9 shows the balance of the segment i, j (i=1,2,…, I, j=1,2,…, J), which is subjected to forces on the four- Here, f ext i,j is the external force applied to the segment i, j. f side i,j ,k (k =1, 2, 3, 4) and fini,j represent the functions of the strains of each crosssection surface ε side i,j ,k (k =1, 2,3,4), and the function of the strain of the inner cross-section surface ε in i,j , respectively. Here, the strain vector ε consists of normal strain and two-directional shear strains. The strains ε side i,j,k (k =1, 2, 3, 4) and ε in i,j are the functions of the gradient of the displacement distribution u dis (p seg i,j ) at the point p seg i,j of the sensor surface, where p seg i,j is the three-dimensional position of each small segment i, j based on the orthogonal coordinate system, as shown in Figure 8. Therefore, the forces on the cross-section surface are given as follows: Substituting (10) and (11) into (9) expresses the external force for balancing the segment statically as follows: Here, we consider the relation between the external force and the position of the segment p seg i,j . We also define the orthogonal coordinate system X(i, j, k)-Y(i, j, k)-Z(i, j, k), such that the X(i, j, k) axis and Z(i, j, k) axis are perpendicular to each cross-section surface and the outer surface of the segment i, j, respectively. When the segment i, j is in contact with the object and does not slip, the X(i, j, k) and Y(i, j, k)directional gradients of the displacement distribution u dis (p seg i,j ) are not changed regardless of u dis (p seg i,j ), as the deformation on the X(i, j, k)-Y(i, j, k) plane is fixed. Therefore, the following equation is satisfied: Here, [U*] XY represents the X(i, j, k) and Y(i, j, k)-directional components of U*. Therefore, the relation between the external force and the segment position p seg i,j satisfies the following equation: Here, the size of the external force increases with the displacement u dis (p seg i,j ) in reference to the initial state, which in most cases is the noncontact state of the touchpad. This result, when taken with the elastic touchpad, is straightforward.
On the other hand, in the case of the segment of the fluid-type touchpad, f in i,j is not the function of the strain of the segment, but also the inner pressure of the membrane p in as follows: From (14) and (16), f ext i,j is also independent of segment position pseg i,j as follows: This characteristic of the fluid-type touchpad is different from that of the elastic touchpad. This indicates that the external force applied to the segments that do not slip does not increase by the displacement of the segment, even if the touchpad's surface is significantly deformed. For example, tangential force applied to the sticking segment does not change when a flat object, in contact with the touchpad, moves in a tangential direction. This characteristic is helpful for understanding the mechanism of incipient slippage when using the fluid-type touchpad. Now, we consider the mechanism by which incipient slippage occurs when using the fluid-type touchpad. When the spherical surface of a fluid-type touchpad is in contact with an object, the pressure is uniformly distributed across the contact surface. This occurs because the applied normal force is balanced by the uniform inner pressure of the fluid-type touchpad, in contrast to the elastic touchpad, as shown in Figure 10. As described in the previous paragraph, tangential force applied to the sticking segment does not increase even when a flat object, in contact with the touchpad, moves in a tangential direction. Tangential force applied in the slippage region remains almost constant, and tangential force increases only at the border between the slippage region and stick region. Therefore, incipient slippage also occurs gradually in the border between the stick region and the slippage region. In our analysis, we demonstrated that the method for estimating slippage degree can be successfully applied to the fluid-type touchpad.

Experimental Results
In this chapter, the proposed methods for using the fluid-type touchpad are confirmed by the experiment's results. The proposed sensor was fixed on a movable stage, in contact with a flat object in a normal direction. When we moved the object and sensor on the movable stage, normal force, tangential force, moment and slippage were simultaneously generated. A laser displacement meter measured the displacement of the object to estimate and evaluate the slippage degree.

Measurement results of normal force
We measured normal force when the fluid-type touchpad was in contact with the object in a normal direction, altering the magnitude of the force. Figure 11 shows the measurement result of normal force. We can thus determine the relationship between normal force and the contact area S c . The relationship between normal force and the contact depth D can also be successfully calculated. These relationships, defined as the functions F S and F D in (3), were identified as follows: Next, the above relationships changed when normal and tangential forces were simultaneously applied. In this case, we compensated the relationships by using the displacement of the central dot d 0 , based on (19) Figure 12 shows the results of this compensation. We can see its success when normal and tangential forces were applied, as the average absolute measurement errors in Figure 12a and 12b were 0.54 (N) and 0.46 (N), respectively.
Finally, we also compensated the relationships change when normal force and moment were simultaneously applied. Regression analysis yielded the following compensation: (20) Figure 13 shows the results of this compensation and its success when normal force and moment were applied. The average absolute measurement errors of the results in Figure 13a and 13b were 0.20 (Nm) and 0.090 (Nm), respectively. In all results, we confirmed that using the contact depth D obtained more accurate results. This is because the contact depth D is calculated more precisely than the contact area S c , when using the previous methods in Ito et al. [23,24].
The average absolute measurement errors in Figure 12a         The average absolute measurement errors in Figure 13a and 13b were 0.20 N and 0.16 N, respectively. The maximum errors in Figure  13a and 13b were 0.49 N and 0.43 N, respectively.

Measurement results of tangential force
We measured tangential force when the object moves in a tangential direction while keeping in contact with the touchpad, as reported in Figure 14. The proposed method using the normalized displacement of the central dot d0nor in (4) can be successfully applied to tangential force as well, since the relationship between the normalized displacement and tangential force is almost linear. The parameters S ref , D ref , TS and T D were identified as 100 mm 2 , 10 mm, 0.65 and 0.75, respectively. In this experiment, using the contact depth D also obtained more accurate results.

Measurement results of rotational moment around normal direction
When we simultaneously applied normal force and moment in a normal direction, moment was measured by the proposed method using a normalized rotation angle of the contact surface θ nor in (5), as reported in Figure 15. Figure 15 shows the linear relationship between the normalized rotation angle and moment. As demonstrated, the fluid-type touchpad can successfully measure moment. The parameters S ref , D ref , R S and R D were identified as 100 mm2, 10 mm, 1.7 and 2.2, respectively. Using the contact depth D also obtained more accurate results in this experiment.

Results of Estimation of Object Slippage:
In this section, we demonstrate the results achieved when the slippage estimation method was applied to the fluid-type touchpad. Tangential force was applied when the touchpad was in contact with a flat object. Figure 16 shows the result of the estimated stick ratio when using the fluid-type touchpad. The upper and lower figures show the stick ratio and the displacements of the central dot and the object, respectively. We can see that the stick ratio was successfully estimated after macroscopic slippage occurred. The rapid increase of the stick ratio after initial macroscopic slippage occurred because a section of the incipient slippage region remained in a deformed state and joined the stick region. This shows that we can estimate the stick ratio in spite of a deformation in the stick region. When macroscopic slippage occurs in the opposite direction, the proposed method can still estimate the stick ratio. Since these results are equivalent to the results achieved when using the elastic touchpad [22], this demonstrates that the slippage estimation method can be successfully applied to the fluid-type touchpad. We confirmed that the fluid-type touchpad can also estimate slippage degree using the proposed method.

Conclusion
We have achieved an estimation of contact force and slippage by using a tactile sensor with a fluid-type touchpad. Consequently, we have proposed a new method based on look-up tables with newer, more efficient compensations for measuring normal force, tangential force and moment. Secondly, we have clarified the mechanism of slippage when using the fluid-type touchpad. We have shown that the fluid-type touchpad is able to generate and estimate slippage degree accurately. Finally, the validation of our proposed methods using the fluid-type touchpad has been confirmed in the experimental results.
Our developed sensor can be fabricated easily and at a low cost, as the sensor has a simple structure and does not require complex sensing elements or wiring. Although the size of the sensor developed and used in this study is relatively large, it can be easily downsized by using a smaller CCD/CMOS camera.
In the process of contributing to this paper, our vision-based sensor with a fluid-type touchpad was developed to a greater level of practicality. Combined with the previous work [23][24][25], our sensor has demonstrated that it can simultaneously obtain multiple types of tactile information, including the contact force, moment, slippage degree, shape, contact region, position and orientation of an object in contact with the fluid-type touchpad. Future work involves the implementation of fluid-type tactile sensors across the industrial and medical fields and in various practical applications, such as robot hands for dexterous handling.