Semi-supervised Learning by Entropy Minimization
Yves Grandvalet and Yoshua Bengio
In: CAp 2005, 31 May - 03 Jun 2005, Nice, France.
We consider the semi-supervised learning problem, where a decision
rule is to be learned from labeled and unlabeled data. In this
framework, we motivate minimum entropy regularization, which
enables to incorporate unlabeled data in the standard supervised
learning. This regularizer can be applied to any model of posterior
probabilities. Our approach provides a new motivation for some
existing semi-supervised learning algorithms which are particular
or limiting instances of minimum entropy regularization.
of experiments illustrates that the proposed solution benefits from
unlabeled data. The method challenges mixture models when the data
are sampled from the distribution class spanned by the generative
model. The performances are definitely in favor of minimum entropy
regularization when generative models are misspecified, and the
weighting of unlabeled data provides robustness to the violation of
the ``cluster assumption''. Finally, we also illustrate that the
method can be far superior to manifold learning in high
dimension spaces, and also when the manifolds are generated by moving examples along the discriminating directions.