PASCAL - Pattern Analysis, Statistical Modelling and Computational Learning

Interpretation of Hybrid Generative/Discriminative Algorithms
Jinghao Xue and Mike Titterington
Neurocomputing 2007.

Abstract

In discriminant analysis, probabilistic generative and discriminative approaches represent two paradigms of statistical modelling and learning. In order to exploit the best of both worlds, hybrid modelling and learning techniques have attracted much research interest recently, one example being the so-called hybrid generative/discriminative algorithm proposed in~\citet{Raina:2003} and its multi-class extension~\citep{Fujino:2007IPM}. In this paper, we interpret this hybrid algorithm from three perspectives, namely class-conditional probabilities, class-posterior probabilities and loss functions underlying the model. We suggest that the hybrid algorithm is by nature a generative model with its parameters learnt through both generative and discriminative approaches, in the sense that in fact it assumes a scaled data-generation process and uses scaled class-posterior probabilities to perform discrimination. Our suggestion can also be applied to the multi-class extension. In addition, using simulated data, we compare the performance of the normalised hybrid algorithm as a classifier with those of the na\"{i}ve Bayes classifier and linear logistic regression. In general, our simulation studies suggest the following: if the covariance matrices are diagonal matrices, the na\"{i}ve Bayes classifier performs the best; if the covariance matrices are non-diagonal matrices, linear logistic regression performs the best. In other words, our studies cannot support that the hybrid algorithm offers better performance than either the na\"{i}ve Bayes classifier or linear logistic regression alone, a phenomenon observed from the empirical studies in~\citet{Raina:2003}

PDF - Requires Adobe Acrobat Reader or other PDF viewer.
EPrint Type:Article
Project Keyword:Project Keyword UNSPECIFIED
Subjects:Learning/Statistics & Optimisation
ID Code:3383
Deposited By:Mike Titterington
Deposited On:09 February 2008