On the Effectiveness of Margin-Maximization for Learning Binary Functions
Joel Ratsaby
The 15th International Conference on Algorithmic Learning Theory 2004.

## Abstract

Let $\mF$ be a finite VC-dimension ($d$) class of binary-valued functions on $[n]=\{1, \ldots, n\}$. Let $\mH\subseteq \mF$ consist of functions $h$ which have a margin no larger than $N$ on every element in $[n]$, where the margin $\mu_h(x)$ of $h\in \mH$ on a point $x\in [n]$ is defined as the largest non-negative integer $a$ such that $h$ is constant on the interval $I_a(x) =[x-a, x+a] \subseteq [n]$. An estimate on the cardinality of $\mH$ with a dependence on $N$, $n$ and $d$, is obtained. There exists a critical threshold $N^* = O((n\ln n)/d)$ such that for $N > N^*$ or $N\leq N^*$ the cardinality of $\mH$ increases or decreases sharply toward the cardinality of $\mF$ or zero, respectively. This result is used to obtain an upper bound on the cardinality of a class $\mH(S)$ which consists of all functions in $\mF$ that have a margin greater than $N$ on all elements of a sample $S\subset[n]$ of size $|S|=l$. The cardinality of $\mH(S)$ decreases at an exponential rate with respect to the margin parameter $N$ for all $N>N'$ where $N'=O\left((n-l)(\ln d)/d\right)$.