3D Modeling with Support Vector Machines

We developed a new approach based on Support Vector Machines (SVM) to model solids defined by a set of points on their surface. In problems of classification, regression and support of distribution, SVM are focused in the hyper-planes of maximum margin in the feature space. However, the forms that could be described by these surfaces when they return to the input space have not been studied in depth. In this paper, these surfaces are used to model complex objects, connected or non-connected, with a great amount of points, in order of tens of thousands, and with various topologies (hollow, branches, etc.). Two constrains were kept: 1) The use of traditional algorithms of SVM theory; and 2) The design of the appropriate training sets from the object. This combination produced a novel tool and the results obtained illustrated the potential of the proposed method. Therefore, this new application of SVM of Vapnik is capable of creating surfaces of decision and estimation functions, which are well fitted to objects of complex topology.


Introduction
Modeling can be defined as the process of finding a mathematical function that best represents a set of given data from an experiment, with both precision and good generalization. In computer graphics, modeling allows to extend the knowledge of an object represented by a cloud of points. For example, an implicit model can be used to evaluate in a simple way the inside and the outside of an object. An implicit model also can be used to visualize the object by ray casting without an intermediate representation. The implicit model makes easier the detection of proximity and collisions. The implicit model can also compress the information; it provides estimation of parameters, such as curvature; and by using logical operations, the implicit model produce more complex objects.
A great diversity of approaches for modeling of clouds of points exists in computer graphics. One of these advanced methods is the variational approach [1]. In this method, radial basis functions are adjusted to the surface of the object using a system of equations, in the N 2 order, where N is the number of points of the object. The radial basis functions |x| 3 and |x| 2 log|x| are proposed in [1]. This approach solves complicated topologies with great safety. Another approach is to use a learning machine to guide a parametric model. For example a Genetic Algorithm (GA) can be used to fit the parameters of an implicit function based on a quality criteria or a GA adaptation function. This method has been used to model organs of the human anatomy with simple surfaces, for example: the super quadrics [2,3]. The objective of this approach is to describe the organ with very few parameters. The approach used in this research is similar to the variational method for its data-driven characteristic and is similar to the GA approach for using a learning machine as search engine.
The SVMs are frequently used in tasks of pattern recognition, like face recognition [4], characters recognition [5], disease diagnosis [6], and genetic classification [7]. Their application had been limited in Computer Graphics. Faloutsos et al. [8] provided an example for applying the SVM to characters animation. The use of the SVM machines on surface modeling has been proposed by Garcia and Moreno [9]. They used the One Class SVM for the modeling of 3D objects. In this work, we analyzed the usefulness and limitations of this approach.
An important characteristic of the SVM machines is the data mapping to another space, called feature space, where the problem is solved with simple geometries. For example, García and Moreno [9] moved the data points inside of one sphere in the feature space. Schölkopf et al. [10] reformulated the SVM machine in order to project the points of the object between two parallel planes in the feature space, where the training set is the whole object. The proximity of these planes represents the accuracy of the model. In this work, we used a single plane geometry in the feature space. Our proposal is different from Schölkopf's in several aspects: (1) In the Schölkopf's method the Support Vectors (SV) is located on the two planes, and the model oscillates among those two limits. In our method the points of the object move away from the surfaces where the support vectors reside and the model interpolates the object points. (2) In our method, the accuracy is fixed beforehand. (3) Our proposal uses traditional machines.
The objectives of the present work are to: (1) model with the traditional SVM, in order to use the more efficient algorithm available, (2) compare different strategies for building the training sets, and (3) analyze the modeling quality using criteria of accuracy and compression.
tens of thousands of points, and the compression is analyzed. Section 8 presents the conclusions.

One Class Svm
The surface of an object can be represented by a set of connected or disconnected points in the bi-dimensional or three-dimensional space. Two problems can be proposed: (1) to find a surface that contains the object, (2) to find a surface that is contained by the object. Sometimes it is possible to get a surface sufficiently near to the object to consider that it represents the object. These concepts will orient the study of the "One Class SVM" to model a solid surface.

Formulation of the one class svm
The SVM theory proposes to project a set of Points . In the feature space the points can be restricted to simple geometries like the interior of a sphere or behind a plane. When the plane or the sphere returns to the input space they will contain the object in its interior. This proposal is presented in Figure 1 and the following equations describe the optimization problem.
Combining equations (6) and (9) we obtain equation (10), which has a minimum value. ( ) The "dual problem" in the α -space is presented in equations (11)-(13), which we obtain from equations (7), (8) and (10). The inner product in equation (10) To solve the optimization problem it is sufficient to know the inner product ( ) , which is known as the kernel of the transformation between the input space and the feature space. The Mercer's condition establishes that a valid kernel is one whose matrix ( ) There exists a great amount of possible kernels, such as the Gaussian kernel of equation (14) and the polynomial kernel of equation (15).

Comparing two geometrical approaches
As we mentioned early in Formulation of the One Class SVM, the points in the feature space can be restricted to simple geometries like the interior of a sphere or behind a plane. In this section we verify that the hyper-sphere of minimum radius and the plane further away from the origin give similar solutions when a Gaussian kernel is considered.
With a Gaussian kernel the points in the feature space are projected on the surface of a sphere of radius 1, as it is shown in Figure  2. Independently of the kernel, there is a minimum radius hypersphere which contains the points in the feature space. Figure 2 shows that there is a spherical cap containing these points, and this cap is reduced to a minimum size when the condition of minimum hypersphere radius or maximum plane distance is achieved. Therefore the two actions (maximize the distance ρ or minimize the sphere radius) have the same effect. The inner points keep closer to the plane than to the hyper-sphere of minimum radius, which from a modeling point of view signifies that the plane returns to the input space a better model than the hyper-sphere, nearest to the points cloud. In our experiments we have verified the correspondence between the obtained values of maximum distance of the plane and minimum radius of the sphere. In others words, we have verified that it is possible to obtain one value from the other, following two different algorithms.

The solution to the modeling problem
The distance function is represented by the term in equation (2) and therefore the term is the distance function with respect to the plane. By application of equation (6) to the distance function we obtain equation (16), which is the distance function with respect to the plane.
Equation (16) distinguishes between interior and exterior, it is positive behind the plane, negative in front the plane and null over the plane. Furthermore, it represents the planar function in both spaces. In the input space, the surface of zero level of this function is the solution of the modeling problem.

Support vectors for the one class svm
The Karush-Kuhn-Tucker's conditions (KKT) establish that the product between the dual variables ( i α , i β ) and the constraints, see equations (2)-(4), is null when the optimization is achieved. This is expressed in equations (17)-(18), obtained with the help of equation (7). These equations are valid only for the values which solve the optimization problem expressed in equations (11)- (13).  Table 1 presents a summary of this classification.
Tuning the input parameters of the one class svm Equations (1)-(3) contain a single input parameter . In this section we analyze the effect of ν on the number of support vectors and the outliers (allowed error). This analysis is based on equation (12). The sum in this equation contains Sphere of minimum radius which contains the data-An "outlier" is allowed-  The plane comes back to the input space contained by the teapot. We used a polynomial kernel (x i ͦ x j +1) 11 . The solution contains 59 support vector and zero outliers.

Support Vectors
Border Vectors According to equation (25) ν fixes an upper limit to the fraction of error vectors (the outliers) and a lower limit to the fraction of support vectors. A high value of ν produced many support vectors and outliers and a low value of ν had the opposite effect; it is possible to reduce to zero the number of outliers.
These results are used in Figures 3(a) and 3(b). In Figure 3(a) Gaussian kernel is used and ν was adjusted to a low value (ν= 0.001) to obtain zero outliers. In this case the plane comes back to the input space containing the object. If the factor ν is changed to 0.999 the plane comes back to the input space contained by the object. For a polynomial kernel the conditions already discussed are inverted. Figure 3(b) shows the result for a low value of ν. This indicates a partial loss of control, since the points behind the plane in Figure 1, (called interior points), may come back to the interior or to the exterior of the model in the input space. The parameter ν can be used to reverse this condition but at the expense of a large number of support vectors.

Binary Classifiers
The SVM have been constructed by Vapnik [12] as learning machines which minimize the classification error, finding the hyperplane of maximum margin that separates two classes in the feature space. The problem of binary classification is discussed in the following sections.

Single binary classifier (svc)
Given a set of points in input space { } 1, , which assign to the points one of two possible values, Vapnik [11] proposed to map the problem to another space (feature space) using a transformation : n m ℜ → ℜ Ö . In the feature space the classes are linearly separable by a hyper plane of maximum margin. This proposal is presented in Figure 1, and the optimization problem is defined by the following equations: As before, we consider a planar function which represents a "Distance Function" in the feature space. This function is null over the hyper-plane of maximum separation (See Figure 4). Equation (29) indicates that all the points are projected behind the planes with distance function equals 1, except the border vectors that lie over the ±1 planes and outliers with a distance function 1 i ξ − . The variable w (gradient of the distance function) adjusts the smoothness of the function: a minimum value of w gives the maximum smoothness and a maximum separation between the two classes, since the real distance between the two planes of distance function 1 and -1 is 2 / w . Equation (28) presents a minimization problem with several objectives which includes the magnitude w and the sum of the lacking variables.
Equations (31)-(33) provide the dual problem obtained from the Lagrangian. As before it provides the kernel that induces the transformation between the two spaces ( )   , which represents the scalar product in the feature space. Equation (34) represents the decision function of the classifier and the zero level surface of this function will be used to solve the modeling problem.

ν-svc classifier
The ν-SVC classifier differs from the simple binary classifier in that the planes located to a distance 1. In Figure 4 are located to a distance ρ, and ρ are considered as another variable to be maximized. The problem of optimization is defined by equations (35)-(37).

Support vectors for the ν-svc classifier
The Karush-Kuhn-Tucker (KKT)´s conditions establish that the product between the dual variables ( i α , i β , γ ) and the restrictions given in equations (36) and (37) is null when optimization is reached, with the help of equation (40) we obtain equations (47)-(49).
( ) Following these equations the vectors i x can be classified in two groups. A first group of vectors which do not contribute to the decision function, called the interior vectors, which satisfy equation (50).
A second group of vectors, the support vectors, those contribute to the decision function. The support vectors are split in border vectors which satisfy equation (51) and the outliers defined by equation (52). Table 2 presents a classification summary of these vectors.
Looking to equation (56) ν sets an upper limit to the outliers and a lower limit to the support vectors. This behavior is similar to the one described in equations (25)-(27) for the One Class SVM.

Regression machine ε-svr and ν-svr
The regression problem considers a distribution of points in a space of n-dimensions with a real number associated to each point. That is represented in Figure 5 where the horizontal axis represents the feature space, and the vertical axis represents the associated real number. The regression machines ε-SVR and ν-SVR define two very close surfaces which contain the distribution of points in the input space. The SVM determines these surfaces in the feature space ( Figure  5), using simple geometries, represented by two planes with maximum separation (planes ε + and ε − in Figure 5). Under this condition, a planar function estimates the value associated to each point in the feature space; and when this planar function comes back to the input space it solves the non-linear regression problem.
For both machines the problem is formalized with the following statement. Given a set of points in the input space { } The regression machine ν-SVR differs from the machine ε-SVR because it incorporates a priori parameter ν, which affects the band of zero error defined by ε, as can be seen in equation (66).
The estimation function for both machines is given in equation (74). When we compare this equation with equation (34) for the binary classifiers we observe that both machines have the same modeling capacity. On the other hand, when we compare with equation (16) for the One Class SVM we observe that the last one admits only positive coefficients. This restriction limits its modeling capacity. Later on, we will make a more extensive comparison between the binary classifier and the regression machine for modeling geometric solids.
The inner vectors between the two planes for the two regression machines do not contribute to the estimation function. The support vectors are represented by the vectors lying outside the planes or over the planes. As before we can prove that ν controls the proportion of outliers and support vectors, and equations (25)-(27) and (56)-(58) continue to be applicable.

Solid Modeling
In the following sections several approaches are proposed to model connected and disconnected objects using the three SVM algorithms proposed by Vanik. Attention will be focused in the design of the SVM training set.

Modeling with "one class svm"
In this Section the behavior of the "One Class SVM" is analyzed following to the focus of solid modeling. In Figure 6(a) a simple object is modeled using a Gaussian kernel. All the points have been interpolated by the zero level contour of the distance function to the plane, see equation (16). In the feature space shown in Figure 2, the points of the object are projected to the intersection of the hyper-plane, the sphere of radius 1 and the sphere of minimum radius. To be sure that the model touches all the points of the object, these points have to be border vectors. However, there is not an input parameter which controls the modeling error, when the points are not border vectors.
Only the parameter ν affects the error vectors as is shown in equation (25).
When inside points are added as it is show in Figure 6(b) the same model persist with the same support vectors. This follows from the fact that the inner vectors do not produce support vectors. The kernel parameters also affect the quality of the model, for example, the value of γ for a Gaussian kernel determines the smoothness of the interpolated contour.
The "One Class SVM" can model non-connected objects as it is shown in Figure 6(c). This is a very important property that can be used to detect cell populations in a histological image.
The "One Class SVM" also can be used to detect the contours of an object with a hollow topology as it is shown in the Figures 7(a) and 7(b).
In this case the model does not fit exactly to the convexities of the object as it is shown by the arrows in Figure 7(b). In this case it is impossible to reach a result as the one of Figure 7(c), which is obtained with a procedure to be explained in "Modeling with the Binary Classifier".

Modeling with the binary classifier
In this Section we use the decision surface of the binary classifier to model objects. A binary classifier requires two classes and these can only be reproduced in an artificial manner from the surface object points. For this reason, we duplicate each point of the surface ( Figure   8 (a)) and we move them in small quantities ε + and ε − from their initial position, in a sense normal to the surface of the object. In this way, we obtain a dipole with the tags +1 and -1, for each point of the object. The result is two layers of points with tags, which represent two versions of the original object. What our method states is that if 0 ε → and if a binary classifier is able to separate the classes with a zero classification error, independent of the object topology, then this surface models the object with an error less than ε.
In Figure 8(a), this method is apply to an object of size 0.25 using ε= 0.01 (4% the size of the object). We have chosen a high value of ε to obtain a visual perception of this method. In this example we obtain accuracy less than 4% when we used the decision surface to model the given object.
Dipoles of Gaussian functions participate in the method, when we use the Gaussian kernel. In principle, for a symmetric dipole, the zero level surface goes thru its centre and an asymmetry between the two dipoles produces a displacement of the surface. However, most of dipoles disappear as it is observed in Figure 8(d) and only survive few dipoles in regions of higher curvature. The curves of level +1 and -1 (Figure 8(b)) define limit for the model position. An important characteristic of this method is that the points on the surface of the object are not included in the training set and therefore they are not part of the support vectors. In this way the model does not visit the curves of level +1 and -1, and, in consequence, we obtain a better approach to the object.
The SVM input parameter can be adjusted according to two criteria: compression and precision. In Figure 8 the maximum error (separation object-model) was 0.004 (median 0.0019), for a prefixed error value of ε= 0.01. The number of points of the object is 24, the training set is 48, and the support vectors number is 20, therefore we  have a low compression 24/20. When we adjust the parameters C and γ to optimize the SVM machine as function of precision; the maximum error was reduced to 0.00089 (median 0.00045).
The proposed method also models non-connected objects as it is shown in Figure 9. In this case the classifier was adjusted to obtain maximum compression. The object contains 84 points, the training set is composed of 168 points and the support vectors are 45 (see Figure  9(d)). The compression reached is 1.87 with respect to the object and 3.83 with respect to the training set. Figure 7(c) proves that the method is also useful to model objects of hollow topology. It will be interesting to check the behavior of this method for 3D objects.

Modeling with the regression machine
For the regression machine, we designed several training sets and its elements was labeled with one three possible values, namely, 1, 0 and -1. We considered the training set as a sample of a continuous function which covers the data space. The task of the regression machine is to estimate this function with an error previously defined. The surface of zero level of the estimation function is interpolating the object points passing at a distance which depends of the a priori error.  (74)) crossed the plane where the data reside and its intersection with this plane defines a contour of zero level which interpolates the object points. This concept can be easily extended to 3D. Figure 10(g) corresponds to the model analyzed in Modeling with the Binary Classifier, for the binary classifier, which, we include to compare the behavior of two machines, classification and regression, with the same object.
Figures 11(a) and 11(d) depict two novel training set proposals. In Figure 11(a) the external layer goes away from the object and in Figure   11(d) both layers go away from the object. Again we assigned the values 1, 0 and -1 to these three layers. For the five proposals of Figure 10 and 11, the left column represents the model with maximum compression superimposed to the training set, the central column represents the support vectors superimposed to the model of maximum compression and the third column represents the model with optimal precision superimposed to the original points of the object. Tables 3 and 4 show the results of the experiments. Learning is guided by the criteria of optimum compression and optimum precision (minimum error). Columns 3-6 of these tables present the machine type and its parameters. The next three columns quantify the size of the training set, the number of support vectors and the number of outliers. The next column quantifies the execution time. The last two columns present the modeling error, evaluated from a first order approximation given by the equation

Analysis of results
 is the derivative of the prediction function. This equation is valid when the points are really closed to the model. It loses precision as long as the points get farther away. However, the resulting value indicates the behavior of the error.
From tables 3 and 4, we perform the following analysis.

Compression:
The symmetric training sets produce the maximum compression, regardless of the symmetric model of three layers triples the size of the training set. This superiority is due to the "data-driven", (characteristic of the SVM). In consequence, increasing the number of vectors nearby the object expands the universe of solutions. On the other hand, vectors far away from the object in the asymmetric models do not contribute to a better determination of the object.

Precision:
The training sets of three layers produce a better precision. However, other experiments prove that if we reduce the displacement in the training set of the binary classifier we obtain the same precision than the one obtained for training sets of three layers.

Computation time:
The classification machine had more stable computation time when we change the parameters between the conditions of compression and optimum precision. The regression machine with training sets of three layers increase sensitively the computation time as function of precision. This is an important consideration to be taken because the used object is 33 points size, if we intent to model a one hundred bigger object and, in addition, the learning algorithm had a N 2 behavior, therefore the computation time for the classifier goes from 16 ms to 160 seconds (2 minutes and 40 seconds) and for the regression machine using the training set of three asymmetric layers the computation time increases from 1.2 s to 12000 s (3 hours and 20 minutes).

Multikernel-Multiscale Machine
A novel proposal consists in applying a regression method over to the residue. A regression machine of the type ε -SVR or ν -SVR is trained using the minus error produced by the first SVM machine with some of the training sets already studied. As prediction function we use the sum of prediction functions of both machines. Figure 12 shows how this scheme reduces very much the position errors by a factor of 10 or bigger. This is achieved by adding the support vectors of both machines. The main objective of this technique is to supply the prediction function with kernels which had a bigger spatial spectrum. In the example of Figure 12 we increase the value of γ for the Gaussian kernel of the second machine by a factor of 10 with respect to the first machine. In this way the regression machine used the residue to refine the details of the model. The result is a multikernel -Multiscale approach.

3D Modeling
In this section we analyze the modeling of 3D objects. In Figure  3(a), we used the One Class SVM to covering an object. The obtained function can be used to evaluate in a simple style the closeness to the object, when detecting collisions. In this section we used two Vapnik machines for 3D modeling. In all experiments we adjusted the dimension and the position of the object to get the interior of a sphere of radius 1, which its center of mass is in the sphere center. In all experiments we use Gaussian kernel. The experiments were performed with a Pentium IV processor of 1.7 GHz. Table 5 shows results of a series of experiments with a classifier ν -SVC, when modeling the surface of a version of 1488 points of the rabbit of Stanford University (Figure 13(a)). We used a training set of two classes shifted 0.01 along the normal to the surface of the object. Instead of using the criteria of optimization already studied (precision and compression), we used the classification error as criterion of optimization. The forth column of Table 5 represents the number of  ill-classified vectors. A zero classification error guaranties that the decision surface of the classifier is located between the two layers of the training set. Figure 13(b) shows the obtained surface for zero classification error.
Precision and error of classification are deeply related, and the constraint for a priori separation between classes fixes a bound for the precision. But the lower limit of this a priori value depends on the capacity of the SVM machine to separate the two classes. For the Vapnik's classifier the classification error is minimized for some value of parameters of the machine ( γ ν − for the machine ν -SVC, and C γ − for the simple SVC). However, there is no warranty that the classification error reaches zero. If this is the case it is necessary to increase the given a priori separation between classes. This defines the limit to model with a single machine. Beyond that it is necessary to enter into the field of the multikernel-multiscale proposal studied in multikernel-multiscale machine.
As the SVM machine works far away from its limit of capacity to model, therefore it had the capacity to compress. This is verified in the experiment presented in Figure 14. In this figure we model a version of the Stanford University rabbit of 34,835 points. The training set is composed of 69,670 points and the classes are separated by 0.01. The obtained model (Figure 14(b)) contains 3,129 support points. This means a compression of 21.6 with respect to the training set or 10.8 with respect to the object. In Figure 14(a) the surface of the object was colored with the position error affected by a non-linear fitting defined by a gamma correction of 0.1, to identify the low levels values of errors.
Among the training sets analyzed in Modeling with the Regression Machine, to model with the regression machine, we use the asymmetric training set of three layers, (see Figure 15). The zero value layers are the object. The inner layer (-1) is obtained by a displacement of the object points along the normal (0.01 in this experiment). The external layer (+1) is obtained throwing points randomly over the surface of a sphere of radius 1.5; taking care the separation between the points overcomes a minimal value. The object is contained inside of a sphere of radius 1. Figure 16 shows two experiments for a more complex topology. The left object is created by four surfaces, two of them intersecting the teapot, and there is no continuity in the mesh. At left we used a ν -SVC classifier, with a symmetric training set with a displacement of 0.001. This value is selected because the object had a region of small thickness. The greater error is observed in the inferior part of the figure, where we had two discontinue crossing mesh.
At right to the Figure 16, the package of the teapot is modeled, using a ε -SVR regression machine, applied on a symmetrical training set of three layers, with a displacement of 0.001. The machine generates a unique surface that is wrapping in it to adhere to the external and internal surfaces of the object. This is very high constraint because the thickness of the object is zero. A better result can be obtained when we removed this constraint and the object is closed. The high time of polygonization in Figure 16 (b) is due to the size of the cell of polygonization. Before we have been using a cell of size 0.01, and for figure 16(b) it was reduced to 0.005. This supports the convenience of using a non-uniform polygonization model. Another alternative is to perform a ray casting until the first point of crossing by zero value of the function, and applying the Phong model at the point found.

Conclusion
In this paper we demonstrated the performance of the SVM machines to obtain mathematical models of the surface of objects. We verified that is possible to use the traditional SVM algorithms with the appropriate selection of the training set. Therefore, our method guarantees that the most efficient algorithm available can be selected. It was also demonstrated that the Vapnik machines are able to create surfaces from their functions of prediction that adjusted to objects of complex topology. The developed method is robust to variations in parameters of the SVM machines. Important values of compression and precision have been achieved with the proposed methodology. The method had been an acid test for the SVM classifiers and regression machines proposed by Vapnik [10]. It would be interesting to study if other learning machines can reach the same benefits.