The feature selection problem can be viewed as a particular case of a more general subset selection problem in which the goal is to find a subset maximizing some adopted criterion. Feature selection methods search through the subsets of features and try to find the best subset among the competing 2^N-1 candidate subsets according to some evaluation measure, where N denotes the total number of features. Feature selection is used in many application areas as a tool to remove irrelevant and redundant features. The objective of feature selection is to simplify a dataset by reducing its dimensionality and identifying relevant underlying features without sacrificing predictive accuracy (Jensen, 2005). In many applications, the size of a dataset is so large that learning might not work as well before removing these unwanted features. Reducing the number of irrelevant/redundant features drastically reduces the running time of a learning algorithm and yields a more general concept. This helps in getting a better insight into the underlying concept of a real world classification problem (Dash & Liu, 1997).

Protein function prediction is an important problem in functional genomics. Proteins are large molecules that perform nearly all of the functions of a cell in a living organism (Alberts et al., 2002). The primary sequence of a protein consists of a long sequence of amino acids. Proteins are the most essential and versatile macromolecules of life, and the knowledge of their functions is a crucial link in the development of new drugs, better crops, and even the development of synthetic biochemical such as biofuels.

Over the past few decades, major advances in the field of molecular biology, coupled with advances in genomic technologies, have led to an explosive growth in the biological information generated by the scientific community. Although the number of proteins with known sequence has grown exponentially in the last few years, due to rapid advances in genome sequencing technology, the number of proteins with known structure and function has grown at a substantially lower rate (Freitas & de Carvalho, 2007).

Searching for similar sequences in protein databases is a common approach used in the prediction of a protein function. The objective of this search is to find a similar protein whose function is known and assigning its function to the new protein. Despite the simplicity and usefulness this method in a large number of situations, it has also some limitations (Freitas & de Carvalho, 2007). For instance, two proteins might have very similar sequences and perform different functions, or have very different sequences and perform a similar function. Additionally, the proteins being compared may be similar in regions of the sequence that are not determinants of their function.

Another approach that may be used alternatively or in complement to the similarity-based approach is to build a model for predictive classification. The goal of such a model is to classify data instances into one of a predefined set of classes or categories. In this approach a feature vector represents each protein, a learning algorithm captures the most important relationships between the features, and the classes present in the dataset. A major problem in protein datasets is the high dimensionality of the feature space. Most of these dimensions are not relative to protein function; even some noise data hurt the performance of the classifier. Hence, we need to select some representative features from the original feature space to reduce the dimensionality of feature space and improve the efficiency and performance of classifier.

Among too many methods which are proposed for feature selection, population-based optimization algorithms such as particle swarm optimization (PSO)-based method (Wang, 2007) and ant colony optimization (ACO)-based method (Aghdam et al., 2008) have attracted a lot of attention. These methods attempt to achieve better solutions by application of knowledge from previous iterations.

Particle swarm optimization comprises a set of search techniques, inspired by the behavior of natural swarms, for solving optimization problems (Kennedy & Eberhart, 2001). PSO is a global optimization algorithm for dealing with problems in which a point or surface in an n-dimensional space best represents a solution. Potential solutions are plotted in this space and seeded with an initial velocity. Particles move through the solution space and certain fitness criteria evaluate them. After a while particles accelerate toward those with better fitness values.

Meta-heuristic optimization algorithm based on ant's behavior was represented in the early 1990s by M. Dorigo and colleagues (Dorigo & Caro, 1999). Ant colony optimization is a branch of newly developed form of artificial intelligence called swarm intelligence. Swarm intelligence is a field which studies "the emergent collective intelligence of groups of simple agents" (Bonabeau et al., 1999). In groups of insects which live in colonies, such as ants and bees, an individual can only do simple task on its own, while the colony's cooperative work is the main reason determining the intelligent behavior it shows (Liu et al., 2004).

ACO algorithm is inspired by ant's social behavior. Ants have no sight and are capable of finding the shortest route between a food source and their nest by chemical materials called pheromone that they leave when moving (Bonabeau et al., 1999). ACO algorithm was firstly used for solving traveling salesman problem (TSP) (Dorigo et al., 1996) and then has been successfully applied to a large number of difficult problems like the quadratic assignment problem (QAP) (Maniezzo & Colorni, 1999), routing in telecommunication networks, graph coloring problems, scheduling, etc. This method is particularly attractive for feature selection as there seems to be no heuristic that can guide search to the optimal minimal subset every time (Jensen, 2005). On the other hand, if features are represented as a graph, ants will discover best feature combinations as they traverse the graph.

This paper proposed an ACO-based algorithm for the feature selection task in bioinformatics datasets. This paper extended by taking advantage of naive Bayes classifier and enable ACO to select features for a Bayesian classification method, which is more sophisticated than the nearest neighbor classifier. Bayesian classifiers are statistical classifiers that can predict class membership probabilities, such as the probability that a given sample belongs to a particular class. Bayesian classification is based on the Bayes theorem (Feller, 1971).

The naive Bayes (NB) classifier uses a probabilistic approach to assign each record of the dataset to a possible class. A naive Bayes classifier makes significant use of the assumption that all features are conditionally independent of one another given the class. This assumption is called class conditional independence (Mitchell, 1996). It is made to simplify the computations involved and, in this sense, is considered naive. In practice, dependencies can exist between variables; however, when the assumption holds true, then the Naive Bayes classifier is the most accurate in comparison with other classifiers (Han & Kamber, 2001).

For testing the proposed ACO algorithm, it is applied to the problem of predicting whether or not a protein has a post-synaptic activity, based on features of protein's primary sequence and finally, the classifier performance and the length of selected feature subset are considered for performance evaluation.

The rest of this paper is organized as follows. Section 2 presents a brief overview of feature selection methods. Sections 3 and 4 briefly address K-nearest neighbor and naive Bayes classifiers. Ant colony optimization is described in sections 5. Section 6 explains the particle swarm optimization. Section 7 reports computational experiments. It also includes a brief discussion of the results obtained and finally the conclusion is offered in the last section.

Feature selection is a process that selects a subset of original features. The optimality of a feature subset is measured by an evaluation criterion. As the dimensionality of a domain expands, the number of features increases. Finding an optimal feature subset is usually intractable (Kohavi & John, 1997) and many problems related to feature selection have been shown to be NP-hard. A typical feature selection process consists of four basic steps, namely, subset generation, subset evaluation, stopping criterion, and result validation (Dash & Liu, 1997). Subset generation is a search procedure that produces candidate feature subsets for evaluation based on a certain search strategy. Each candidate subset is evaluated and compared with the previous best one according to a certain evaluation criterion. If the new subset turns out to be better, it replaces the previous best subset. The process of subset generation and evaluation is repeated until a given stopping criterion is satisfied. Then the selected best subset usually needs to be validated by prior knowledge or different tests via synthetic and/or real world datasets.

The generation procedure implements a search method (Siedlecki & Sklansky, 1988) that generates subsets of features for evaluation. It may start with no features, all features, a selected feature set or some random feature subset. Those methods that start with an initial subset usually select these features heuristically beforehand. Features are added (forward selection) or removed (backward elimination) iteratively in the first two cases (Dash & Liu, 1997). In the last case, features are either iteratively added or removed or produced randomly thereafter (Jensen, 2005). The disadvantage of forward selection and backward elimination methods is that the features that were once selected/eliminated cannot be later discarded/re-selected. To overcome this problem, Pudil et al. proposed a method to flexibly add and remove features (Pudil et al., 1994). This method has been called floating search method.

According to the literature, the approaches to feature subset selection can be divided into filters and wrappers approaches (Guyon & Elisseeff, 2003). The filter model separates feature selection from classifier learning and selects feature subsets that are independent of any learning algorithm. It relies on various measures of the general characteristics of the training data such as distance, information, dependency, and consistency (Liu & Motoda, 1998). In the wrapper approach feature subset is selected using the evaluation function based on the same learning algorithm that will be used later for learning. In this approach the evaluation function calculates the suitability of a feature subset produced by the generation procedure and it also compares that with the previous best candidate, replacing it if found to be better. A stopping criterion is tested in each of iterations to determine whether or not the feature selection process should continue. Although, wrappers may produce better results, they are expensive to run and can break down with very large numbers of features. This is due to the use of learning algorithms in the evaluation of subsets, some of which can encounter problems while dealing with large datasets (Forman, 2003; Jensen, 2005).

The K-nearest neighbor (KNN) algorithm is amongst the simplest of all machine learning algorithms. An object is classified by a majority vote of its neighbors, with the object being assigned to the class most common amongst its K nearest neighbors. K is a positive integer, typically small. If K = 1, then the object is simply assigned to the class of its nearest neighbor. In binary (two class) classification problems, it is helpful to choose K to be an odd number as this avoids tied votes.

The same method can be used for regression, by simply assigning the property value for the object to be the average of the values of its K nearest neighbors. It can be useful to weight the contributions of the neighbors, so that the nearer neighbors contribute more to the average than the more distant ones.

The neighbors are taken from a set of objects for which the correct classification (or, in the case of regression, the value of the property) is known. This can be thought of as the training set for the algorithm, though no explicit training step is required. In order to identify neighbors, the objects are represented by position vectors in a multidimensional feature space. It is usual to use the Euclidean distance, though other distance measures, such as the Manhattan distance could in principle be used instead. The K-nearest neighbor algorithm is sensitive to the local structure of the data.

The performance of a KNN classifier is primarily determined by the choice of K as well as the distance metric applied (Latourrette, 2000). However, it has been shown in (Domeniconi et al., 2002) that when the points are not uniformly distributed, predetermining the value of K becomes difficult. Generally, larger values of K are more immune to the noise presented, and make boundaries smoother between classes. As a result, choosing the same (optimal) K becomes almost impossible for different applications.

One highly practical Bayesian learning method is the naive Bayes leaner, often called the naive Bayes (NB) classifier. In some domains its performance has been shown to be comparable to that of neural network and decision tree learning. This section introduces the naive Bayes classifier.

The naive Bayes classifier applies to classification tasks where each instance x is described by a conjunction of feature values and where the target function f(x) can take on any value from some finite set V. A set of training examples of the target function is provided, and a new instance is presented, described by the tuple of feature values <a₁,a₂...a_n>. The classifier is asked to predict the target value, or classification, for this new instance (Mitchell, 1996).

The Bayesian approach to classifying the new instance is to assign the most probable target value, v_MAP, given the feature values <a₁,a₂...a_n> that describe in instance.

[1] vMAP=argmaxvj∈VP(vj|a1,a2 ...an)

We can use Bayes theorem to rewrite this expression as

[2] v MAP = argmax v j ∈V P( a 1 , a 2 ... a n | v j )P( v j ) P( a 1 , a 2 ... a n )         = argmax v j ∈V P( a 1 , a 2 ... a n | v j )P( v j )

We could attempt to estimate the two terms in equation (2) based on the training data. It is easy to estimate each of the P(v_i) simply by counting the frequency with which each target value v_i occurs in the training data. However, estimating the different P(a₁,a₂...a_n | v_j) terms in this fashion is not feasible unless we have a very, very large set of training data. The problem is that the number of these terms is equal to the number of possible instances times the number of possible target values. Therefore, we need to see every instance in the instance space many times in order to obtain reliable estimates.

The naive Bayes classifier is based on the simplifying assumption that the feature values are conditionally independent given the target value. In other words, the assumption is that given the target value of the instance, the probability of observing the conjunction a₁,a₂...a_n is just the product of the probabilities for the individual features: P(a₁,a₂...a₁ | v_j) = P(v_j)Π_iP(a_i | v_j). Substituting this into equation (2), we have the approach used by the naive Bayes classifier.

[3] v NB = argmax v j ∈V P( v j ) Π i P( a i | v j )

where v_NB denotes the target value output by the naive Bayes classifier (Mitchell, 1996).

One interesting difference between the naive Bayes classification method and other classification methods we have considered is that there is no explicit search through the space of possible hypotheses (in this case, the space of possible hypotheses is the space of possible values that can be assigned to the various P(v_i) and P(a_i | v_i) terms). Instead, the hypothesis is formed without searching, simply by counting the frequency of various data combinations within the training examples.

To evaluate the performance of the Bayesan classification, we use a 10-fold cross-validation. We divide the data set into 10 equally sized folds. For all class levels each fold maintains roughly the same proportion of classes present in the whole data set before division (called stratified crossvalidation). Eight of the ten folds are used to compute the probabilities for the Bayesian classification. The ninth fold is used as validation set and the tenth fold as test set. During the search for the solution only the validation set is used to compute predictive accuracy. The performance of the candidate solutions is given by the predictive accuracy of the classification in the validation set. The solution that shows the highest predictive accuracy on the validation set is then used to compute the predictive accuracy on the test set. Once the solution is selected, the nine folds are merged and this merged dataset is used to compute the probabilities for the Bayesian classification. The predictive accuracy (reported as the final result) is then computed on the previously untouched test set fold. Every fold will be once used as validation set and once used as test set. A similar process is adopted for the computation of the predictive accuracy using the nearest neighbor classifier, which will be described in more details in subsection 7.2.

Ant colony optimization was introduced by Marco Dorigo (Dorigo, 1992) and his colleagues in the early 1990s. The first computational paradigm appeared under the name ant system (AS). It is another approach to stochastic combinatorial optimization. The search activities are distributed over: "ants" - agents with very simple basic capabilities that mimic the behavior of real ants. The main aim was not to simulate ant colonies, but to use artificial ant colonies as an optimization tool. Therefore the system exhibits several differences in comparison to the real (natural) ant colony: artificial ants have some memory; they are not completely blind; they live in an environment with discrete time. In ACO algorithms, artificial ants construct solutions from scratch by probabilistically making a sequence of local decisions. At each construction step an ant chooses exactly one of possibly several ways of extending the current partial solution. The rules that define the solution construction mechanism in ACO implicitly map the search space of the considered problem (including the partial solutions) onto a search tree.

The paradigm is based on the observation made by ethologists about the medium used by ants to communicate information regarding shortest paths to food by means of pheromone trails. A moving ant lays some pheromone on the ground, thus making a path by a trail of this substance. While an isolated ant moves practically at random (exploration), an ant encountering a previously laid trail can detect it and decide with high probability to follow it and consequently reinforce the trail with its own pheromone (exploitation). What emerges is a form of autocatalytic process through which the more the ants follow a trail, the more attractive that trail becomes to be followed. The process is thus characterized by a positive feedback loop, during which the probability of choosing a path increases with the number of ants that previously chose the same path. The mechanism above is the inspiration for the algorithms of the ACO family (Engelbrecht, 2005).

Feature selection is one of the applications of subset problems. Given an feature set of size n, the feature selection problem is to find a minimal feature subset of size s (s < n) while retaining a suitably high accuracy in representing the original features. Therefore, there is no concept of path. A partial solution does not define any ordering among the components of the solution, and the next component to be selected is not necessarily influenced by the last component added to the partial solution (Aghdam, 2008). Furthermore, solutions to a feature selection problem are not necessarily of the same size. To apply an ACO algorithm to solve a feature selection problem, these aspects need to be addressed. The first problem is addressed by redefining the way that the representation graph is used.

The feature selection task may be reformulated into an ACO-suitable problem. Ant colony optimization requires a problem to be represented as a graph. Here nodes represent features, with the edges between them denoting the choice of the next feature. The search for the optimal feature subset is then an ant traversal through the graph where a minimum number of nodes are visited that satisfies the traversal stopping criterion. Figure 1. illustrates this setup. The ant is currently at node a and has a choice of which feature to add next to its path (dotted lines). It chooses feature b next based on the transition rule, then c and then d. Upon arrival at d, the current subset {a, b, c, d} is determined to satisfy the traversal stopping criterion (e.g. suitably high classification accuracy has been achieved with this subset). The ant terminates its traversal and outputs this feature subset as a candidate for data reduction (Aghdam, 2008).

A suitable heuristic desirability of traversing between features could be any subset evaluation function for example, an entropy-based measure (Jensen, 2005) or rough set dependency measure (Pawlak, 1991). The heuristic desirability of traversal and edge pheromone levels are combined to form the so-called probabilistic transition rule, denoting the probability that ant k will include feature i in its solution at time step t:

[4] P i k (t)={ [ τ i (t)] α . [ η i ] β ∑ l∈ J k [ τ l (t)] α . [ η l ] β        if i∈ J k 0                                    otherwise

where η_j is the heuristic desirability of choosing feature i (η_jis optional but often needed for achieving a high algorithm performance), J^k is the set of feasible features that can be added to the partial solution. α > 0, β > 0 are two parameters that determine the relative importance of the pheromone value and heuristic information (the choice of α, β is determined experimentally) and τ_j(t) is the amount of virtual pheromone on feature i.

Figure 1

The pheromone on each feature is updated according to the following formula:

[5] τ i (t+1)=(1-ρ). τ i (t)+ ∑ k=1 m Δ i k (t)

where

[6] Δ i k (t)={ ψ( S k (t))/| S k (t)|       if  i∈ S k (t) 0                               otherwise

The value 0 = ρ = 1 is decay constant used to simulate the evaporation of the pheromone, S^k(t) is the feature subset found by ant k at iteration t, and |S^k(t)| is its length. The pheromone is updated according to both the measure of the "goodness" of the ant's feature subset (ψ) and the size of the subset itself. By this definition, all ants can update the pheromone.

The overall process of ACO feature selection can be seen in Figure 2. The process begins by generating a number of ants, m, which are then placed randomly on the graph i.e. each ant starts with one random feature. Alternatively, the number of ants to place on the graph may be set equal to the number of features within the data; each ant starts path construction at a different feature. From these initial positions, they traverse edges probabilistically until a traversal stopping criterion is satisfied. The resulting subsets are gathered and then evaluated. If an optimal subset has been found or the algorithm has executed a certain number of times, then the process halts and outputs the best feature subset encountered. If none of these conditions hold, then the pheromone is updated, a new set of ants are created and the process iterates once more.

Figure 2

The main steps of proposed feature selection algorithm are as follows:

The time complexity of proposed algorithm is O(Imn), where I is the number of iterations, m the number of ants, and n the number of original features. This can be seen from Figure 2. In the worst case, each ant selects all the features. As the heuristic is evaluated after each feature is added to the candidate subset, this will result in n evaluations per ant. After the first iteration in this algorithm, mn evaluations will have been performed. After I iterations, the heuristic will be evaluated Imn times.

Particle swarm optimization is an evolutionary computation technique developed by Kennedy and Eberhart (Kennedy & Eberhart, 2001). The original intent was to graphically simulate the graceful but unpredictable movements of a flock of birds. Initial simulations were modified to form the original version of PSO. Later, Shi introduced inertia weight into the particle swarm optimizer to produce the standard PSO (Kennedy & Eberhart, 2001).

PSO is initialized with a population of random solutions, called "particles". Each particle is treated as a point in an n-dimensional space. The ith particle is represented as X_i = (x_i1, x_i2, ... , x_in). The best previous position (pbest, the position giving the best fitness value) of any particle is recorded and represented as P_i = (p_i1, p_i2, ... , p_in). The index of the best particle among all the particles in the population is represented by the symbol "gbest". The rate of the position change (velocity) for particle i is represented as V_i = (v_i1, v_i2, ... , v_in). The particles are manipulated according to the following equation:

[7] V i (t+1)=w. V i (t)+ c 1 . r 1 (t).[ P i (t)- X i (t)]+ c 2 . r 2 (t).[ P g (t)- X i (t)]

where

[8] X i (t+1)= X i (t)+ V i (t+1)

Where w is the inertia weight, it is a positive linear function of time changing according to the generation iteration. Suitable selection of the inertia weight provides a balance between global and local exploration and results in less iteration on average to find a sufficiently optimal solution. The acceleration constants c1 and c2 in equation (7) represent the weighting of the stochastic acceleration terms that pull each particle toward pbest and gbest positions. Low values allow particles to roam far from target regions before being tugged back, while high values result in abrupt movement toward, or past, target regions. r_1d(t) and r_2d(t) ˜ U(0,1) are random values in the range [0,1], sampled from a uniform distribution.

Particles' velocities on each dimension are limited to a maximum velocity, V_max. It determines how large steps through the solution space each particle is allowed to take. If V_max is too small, particles may not explore sufficiently beyond locally good regions. They could become trapped in local optima. On the other hand, if V_max is too high particles might fly past good solutions.

The first part of equation (7) provides the "flying particles" with a degree of memory capability allowing the exploration of new search space areas. The second part is the "cognition" part, which represents the private thinking of the particle itself. The third part is the "social" part, which represents the collaboration among the particles. Equation (7) is used to calculate the particle’s new velocity according to its previous velocity and the distances of its current position from its own best experience (position) and the group's best experience. Then the particle flies toward a new position according to equation (8). The performance of each particle is measured according to a pre-defined fitness function.

In this section, we report and discuss computational experiments and compare ACO feature selection algorithm with PSO-based approach. The quality of a candidate solution is computed by the naive Bayes classifier which described in section 2. Finally the classifier performance and the length of selected feature subset are considered for evaluating the proposed algorithm.

A series of experiments was conducted to show the utility of proposed feature selection algorithm. All experiments have been run on a machine with 3.0GHz CPU and 1024 MB of RAM. We implement ACO algorithm and PSObased algorithm in Java and Weka 3.5.5. The operating system was Windows XP Professional. For experimental studies we have considered Postsynaptic dataset. The following sections describe Postsynaptic dataset and implementation results.

Figure 3

Experimental results show that the use of unnecessary features decrease classifiers' performance and hurt classification accuracy and feature selection is used to reduces redundancy in the information provided by the selected features. Using only a small subset of selected features, the binary PSO and the ACO algorithms obtained better predictive accuracy than the Baseline algorithm using all features. ACO has the ability to converge quickly; it has a strong search capability in the problem space and can efficiently find minimal feature subset. Experimental results demonstrate competitive performance.

The ACO clearly enhances computational efficiency of the classifier by selecting fewer features than the standard binary PSO algorithm. Therefore, when the difference in predictive accuracy is insignificant, ACO is still preferable. In the proposed ACO algorithm, the classifier performance and the length of selected feature subset are adopted as heuristic information. So, we can select the optimal feature subset without the prior knowledge of features.

Also as we expected, computational results show the clear difference in performance between nearest neighbor and naive Bayes classifiers. To summarize, the naive Bayes classification method involves a classification step in which the various P(v_i) and P(a_i | v_i) terms are estimated, based on their frequencies over the training data. The set of these estimates corresponds to the learned hypothesis. This hypothesis is then used to classify each new instance by applying the rule in equation (3). Whenever the naive Bayes assumption of conditional independence is satisfied, this naive Bayes classification v_NB is identical to the maximum a posterior classification. The naive Bayes approach outperformed the nearest neighbor approach in all experiments.