Bias-variance tradeoff is a well-known problem in machine learning and a motivating principle behind many regularization techniques. We can define them as:

- Bias measures the average difference between predicted values and true values. As bias increases, a model predicts less accurately on a training dataset. High bias refers to high error in training. Optimization signifies attempts to reduce bias.

- Variance measures the difference between predictions across various realizations of a given model. As variance increases, a model predicts less accurately on unseen data. High variance refers to high error during testing and validation. Generalization refers to attempts to reduce variance.

Bias and variance thus inversely represent model accuracy on training and test data respectively.⁵ They are two of three terms that comprise a model’s total error rate, the third being irreducible error. This third term denotes error resulting from inherent randomness in a dataset. Total model error can be defined by the formula:⁶

Any one model training algorithm consists of numerous variables—e.g. training data, hyperparameters, and so forth—that affect the consequent model’s total error. Thus, even a single training algorithm can produce different models, each with their own bias, variance, and irreducible error rates. By combining several diverse models, ensemble algorithms can yield a lower overall error rate while retaining each individual model’s own complexities and advantages, such as notably low bias for a specific data subset.⁷

Research suggests that, in general, the greater diversity among combined models, the more accurate the resulting ensemble model. Ensemble learning can thus address regression problems such as overfitting without trading away model bias. Indeed, research suggests that ensembles comprised of diverse under-regularized models (i.e. models that overfit to their training data) outperform single regularized models.⁸ Moreover, ensemble learning techniques can help resolve issues stemming from high-dimensional data, and so effectively serve as an alternative to dimensionality reduction.

Literature widely categorizes ensemble learning methods in machine learning into two groups: parallel and sequential.

- Parallel methods train each base learner apart from the others of the others. Per its name, then, parallel ensembles train base learners in parallel and independent of one another.

- Sequential methods train a new base learner so that it minimizes errors made by the previous model trained in the preceding step. In other words, sequential methods construct base models sequentially in stages.⁹

Parallel methods are further divided into homogenous and heterogenous methods. Homogenous parallel ensembles use the same base learning algorithm to produce all of the component base learners. Heterogenous parallel ensembles use different algorithms to produce base learners.¹⁰

How do ensemble methods combine base learners into a final learner? Some techniques—e.g. stacking—use separate machine learning algorithms to train an ensemble learner from the base learners. But one common method for consolidating base learner predictions is voting—and more precisely, majority voting.

Majority voting considers each base learner’s prediction for a given data instance and outputs a final prediction determined by whatever the majority of learners predict. For instance, in a binary classification problem, majority voting takes predictions from each base classifier for a given data instance and uses the majority prediction as the end prediction. Weighted majority voting is an extension of this technique that gives greater weight to certain learner’s predictions over others.¹¹

Perhaps three of the most popular ensemble learning techniques are bagging, boosting, and stacking. In fact, these together exemplify distinctions between sequential, parallel, homogenous, and heterogenous types of ensemble methods.

Note that this overview is not exhaustive; there are several additional ensemble methods, such as blending and weighted average ensembles. This is merely meant to survey some of the more prominent methods in literature.

Bagging is a homogenous parallel method sometimes called bootstrap aggregating. It uses modified replicates of a given training data set to train multiple base learners with the same training algorithm.¹² Scikit-learn’s ensemble module in Python contains functions for implementing bagging, such as BaggingClassifier.

More specifically, bagging uses a technique called bootstrap resampling to derive multiple new datasets from one initial training dataset in order to train multiple base learners. How does this work? Say a training dataset contains n training examples. Bootstrap resampling copies n data instances from that set into a new subsample dataset, with some initial instances appearing more than once and others excluded entirely. These are bootstrap samples. Repeating this process x times produces x iterations of the original dataset, each containing n samples from the initial set. Each iteration of the initial set is then used to train a separate base learner with the same learning algorithm.¹³

Random forest is an extension of bagging that specifically denotes the use of bagging to construct ensembles of randomized decision trees. This differs from standard decision trees in that the latter samples every feature to identify the best for splitting. By contrast, random forests iteratively sample random subsets of features to create a decision node.¹⁴

Stacking, or stacked generalization,¹⁵ is a heterogenous parallel method that exemplifies what is known as meta-learning. Meta-learning consists of training a meta-learner from the output of multiple base learners. Stacking specifically trains several base learners from the same dataset using a different training algorithm for each learner. Each base learner makes predictions on an unseen dataset. These first model predictions are then compiled and used to train a final model, being the meta-model.¹⁶

Note the importance of using a different dataset from that used to train the base learners in order to train the meta-learner. Using the same dataset to train the base learners and the meta-learner can result in overfitting. This can require excluding data instances from the base learner training data to serve as its test set data, which in turn becomes training data for the meta-learner. Literature often recommends techniques such as cross-validation to ensure these datasets do not overlap.¹⁷

Much as bagging, the sklearn.ensemble module in Python provides various functions for implementing stacking techniques.

Boosting algorithms are a sequential ensemble method. Boosting has many variations, but they all follow the same general procedure. Boosting trains a learner on some initial dataset, d. The resultant learner is typically weak, misclassifying many samples in the dataset. Much like bagging, boosting then samples instances from the initial dataset to create a new dataset (d₂). Unlike bagging, however, boosting prioritizes misclassified data instances from the first model or learner. A new learner is trained on this new dataset d₂. Then a third dataset (d₃) is then compiled from d₁ and d₂, prioritizes the second learner’s misclassified samples and instances in which d₁ and d₂ disagree. The process repeats n times to produce n learners. Boosting then combines and weights the all the learners together to produce final predictions.¹⁸

Boosting algorithms largely differ in how they prioritize erroneously predicted data instances when creating a new dataset. Two of the most prominent boosting methods may illustrate this:

- Adaptive boosting (AdaBoost) weights model errors. That is, when creating a new iteration of a dataset for training the next learner, AdaBoost adds weights to the previous learner’s misclassified samples, causing the next learner to prioritize those misclassified samples.

- Gradient boosting uses residual errors when training new learners. Rather than weight misclassified samples, gradient boosting uses residual errors from a previous model to set target predictions for the next model. In this way, it attempts to close the gap of error left by one model.¹⁹

Unfortunately, sklearn contains no pre-defined functions for implementing boosting. The Extreme Gradient Boosting (XGBoost) open-source library, however, provides code for implementing gradient boosting in Python.

Given difficulties in acquiring large, fair-use, labeled datasets for training learners, ensemble learning has seen many applications in an attempt to improve learner performance with less data. For instance, several recent studies show promising results with improving model generalizability using ensemble methods for computer vision task, such as training several models with different representations of a dataset²⁰ or combining several biased models.²¹

Despite ensemble methods’ ability to improve generalizability, they nevertheless can suffer unfairness. In machine learning, fairness denotes attempts to alleviate algorithmic bias (often against minority groups) in automated systems, usually resulting from learners trained on sensitive data. A handful of studies propose metrics, preprocessing, and postprocessing techniques for improving fairness in ensemble models.²² Continued efforts to improve fairness and ethical practices in AI remain a much needed area for future research.