An Efficient Method for Gradient-Based Adaptation of Hyperparameters in SVM Models
Sathiya Keerthi, Vikas Sindhwani and Olivier Chapelle
We consider the task of tuning hyperparameters in SVM models based on minimizing a smooth performance validation function, e.g., smoothed k-fold cross-validation error, using non-linear optimization techniques. The key computation
in this approach is that of the gradient of the validation function with respect to
hyperparameters. We show that for large-scale problems involving a wide choice
of kernel-based models and validation functions, this computation can be very efficiently done; often within just a fraction of the training time. Empirical results
show that a near-optimal set of hyperparameters can be identified by our approach
with very few training rounds and gradient computations.