Strengths and weaknesses of Discriminant Analysis

The simplest (and usually very poor) parametric classification technique consists in assigning an observation to the class whose barycenter is closest to the observation, in the Euclidian sense of the term. This is a "first order" technique, in that in considers only the barycenters of the classes, whose coordinates are linear (first degree) functions of the observations, and does not take their spreads in the various directions of space into consideration.

The next step is to take the "second order information" about classes into account, that is, to consider their covariance matrices in addition to their barycenters : the coefficients of a covariance matrix are quadratic (second degree) functions or the observations . This is exactly what Discriminant Analysis does. In fact, rather than emphasizing the "class normality" assumption, it is equally justified to say that Discriminant Analysis is the simplest "second order classification technique", with then no reference to a particular functional form of the class densities. But because a multinormal density is fully determined by its mean and its covariance matrix, Discriminant Analysis is the ideal classification technique for multinormal classes.

The beauty of it is that it will turn out that Discriminant Analysis may still be interpreted as a "nearest barycenter" technique (first order), provided that the usual Euclidian distance be replaced by another "distance", the Mahalanobis distance, which is defined from second order information about the classes.

So the strength of Discriminant Analysis is that it is an optimal yet simple, complete and accurate classification technique whenever the basic "second order" assumptions accurately describe the setting (multinormal classes).

Of course, this is also where its weakness is. Classes are never perfectly multinormal : to what extent do the performances of Discriminant Analysis degrade when the class densities depart form normality ? Theory has no definite answer to this question, but one of the reasons for the popularity of Discriminant Analysis is that decades of intensive use show that DA is reasonably robust with respect to departure from the standard assumptions.

Another weakness of the full Discriminant Analysis is that as many covariance matrices have to be estimated as there are classes. This easily leads to models containing tens or even hundreds of parameters, a rather large number in view of the usually limited amount of design data. Consequently, the full DA tends to be unstable (strong dependence of design data), whereas the "restricted" versions, with fewer parameters, tend, to the contrary, to be more stable but at the expense of a strong bias. Modern versions of Discriminant Analysis incorporate regularization mechanisms, whose principles are similar to that of Ridge Regression, and which allow fine tuning of the balance between bias and variance of a DA model.


Ankit Agarwal, 12070, Marketing