Multi-objective Evolutionary Ensemble Pruning Guided by Margin Distribution

Ensemble learning trains and combines multiple base learners for a single learning task, and has been among the state-of-the-art learning techniques. Ensemble pruning tries to select a subset of base learners instead of combining them all, with the aim of achieving a better generalization performance as well as a smaller ensemble size. Previous methods often use the validation error to estimate the generalization performance during optimization, while recent theoretical studies have disclosed that margin distribution is also crucial for better generalization. Inspired by this finding, we propose to formulate ensemble pruning as a three-objective optimization problem that optimizes the validation error, margin distribution, and ensemble size simultaneously, and then employ multi-objective evolutionary algorithms to solve it. Experimental results on 20 binary classification data sets show that our proposed method outperforms the state-of-the-art ensemble pruning methods significantly in both generalization performance and ensemble size.