What is upsampling?

Upsampling is an effective way to address imbalance within a dataset. An imbalanced dataset is defined as a dataset in which one class is greatly underrepresented in the dataset relative to the true population, creating unintended bias. For instance, imagine a model is trained to classify images as showing a cat or a dog. The dataset used is composed of 90% cats and 10% dogs. Cats in this scenario are overrepresented, and if we have a classifier predicting cats every time, it will yield a 90% accuracy for classifying cats, but 0% accuracy for classifying dogs. The imbalanced dataset in this case will cause classifiers to favor accuracy for the majority class at the expense of the minority class. The same issue can arise with multi-class datasets.¹

The process of upsampling counteracts the imbalanced dataset issue. It populates the dataset with points synthesized from characteristics of the original dataset’s minority class. This balances the dataset by effectively increasing the number of samples for an underrepresented minority class until the dataset contains an equal ratio of points across all classes.

While imbalances can be seen by simply plotting the counts of data points in each class, it doesn’t tell us whether it will greatly affect the model. Fortunately, we can use performance metrics to gauge how well an upsampling technique corrects for class imbalance. Most of these metrics will be for binary classification, where there are only two classes: a positive and a negative. Usually, the positive class is the minority class while the negative class is the majority class. Two popular metrics are Receiver Operating Characteristic (ROC) curves and precision-recall curves.¹

• No Information Loss: Unlike downsampling, which removes data points from the majority class, upsampling generates new data points, avoiding any information loss.
• Increase Data at Low Costs: Upsampling is especially effective, and is often the only way, to increase dataset size on demand in cases where data can only be acquired through observation. For instance, certain medical conditions are simply too rare to allow for more data to be collected.

• Overfitting: Because upsampling creates new data based on the existing minority class data, the classifier can be overfitted to the data. Upsampling assumes that the existing data adequately captures reality; if that is not the case, the classifier may not be able to generalize very well.
• Data Noise: Upsampling can increase the amount of noise in the data, reducing the classifier’s reliability and performance.²
  
• Computational Complexity: By increasing the amount of data, training the classifier will be more computational expensive, which can be an issue when using cloud computing.²

Random oversampling is the process of duplicating random data points in the minority class until the size of the minority class is equal to the majority class.

Though they are similar in nature, random oversampling is distinct from bootstrapping. Bootstrapping is an ensemble learning technique that resamples from all classes. By contrast, random oversampling resamples from only the minority class. Random oversampling can thus be understood as a more specialized form of bootstrapping.

Despite its simplicity, random oversampling has limitations, however. Because random oversampling solely adds duplicate datapoints, it can lead to overfitting.³ But it still has many advantages over other methods: its ease of implementation, lack of stretching assumptions about the data, and low time complexity due to a simple algorithm.²

The Synthetic Minority Oversampling Technique, or SMOTE, is an upsampling technique first proposed in 2002 that synthesizes new data points from the existing points in the minority class.⁴ It consists of the following process:²

1. Find the K nearest neighbors for all minority class data points. K is usually 5.
2. Repeat steps 3-5 for each minority class data point:
3. Pick one of the data point’s K nearest neighbors.
4. Pick a random point on the line segment connecting these two points in the feature space to generate a new output sample. This process is known as interpolation.
5. Depending on how much upsampling is desired, repeat steps 3 and 4 using a different nearest neighbor.

SMOTE counters the problem of overfitting in random oversampling by adding previously unseen new data to the dataset rather than simply duplicating pre-existing data. For this reason, some researchers consider SMOTE a better upsampling technique than random oversampling.

On the other hand, SMOTE’s artificial data point generation adds extra noise to the dataset, potentially making the classifier more unstable.¹ The synthetic points and noise from SMOTE can also inadvertently lead to overlaps between the minority and majority classes that don’t reflect reality, leading to what is called over-generalization.⁵

One popular extension, Borderline SMOTE, is used to combat the issue of artificial dataset noise and to create ‘harder’ data points. ‘Harder’ data points are data points close to the decision boundary, and therefore harder to classify. These harder points are more useful for the model to learn.²

Borderline SMOTE identifies the minority class points that are close to many majority class points and puts them into a DANGER set. DANGER points are the ‘hard’ data points to learn, which again is because they’re harder to classify compared to points that are surrounded by minority class points. This selection process excludes points whose nearest neighbors are only majority class points, which are counted as noise. From there, the SMOTE algorithm continues as normal using this DANGER set.³

Adaptive Synthetic Sampling Approach (ADASYN) is similar to Borderline SMOTE in that it generates more difficult data for the model to learn. But it also aims to preserve the distribution of the minority class data.⁶ It does this by first creating a weighted distribution of all the minority points based on the number of majority class examples in its neighborhood. From there, it uses minority class points closer to the majority class more often in generating new data.

The process goes as follows:²

1. Create a KNN model on the entire dataset.
2. Each minority class point is given a “hardness factor”, denoted as r, which is ratio of the number of majority class points over the total number of neighbors in KNN.
3. Like SMOTE, the synthetically generated points are a linear interpolation between the minority data and its neighbors, but the number of points generated scales with a point’s hardness factor. What this does is generate more points in areas with less minority data and less points in areas with more.

Data augmentation creates new data by creating variations of the data. Data augmentation applies across a variety of machine learning fields.

The most basic form of data augmentation deals with transforming the raw inputs of the dataset. For example, in computer vision, image augmentations (cropping, blurring, mirroring and so on) can be used to create more images for the model to classify. Similarly, data augmentation can also be used in natural language processing tasks, like replacing words with their synonyms or creating semantically equivalent sentences.

Researchers have found that data augmentation effectively increases model accuracy for computer vision and NLP tasks because it adds similar data at a low cost. However, it is important to note some cautions before executing these techniques. For traditional geometric augmentations, “safety” of transformations should be looked at before performing them. For example, rotating an image of a “9” would make it look a “6,” changing its semantic meaning.⁷

SMOTE extensions and deep learning have been the focus of upsampling techniques in recent years. These methods aim to improve model performance and address some of the shortcomings of upsampling, like introduced bias in the distribution of the minority class.

Some developments in SMOTE include a minority-predictive-probability SMOTE (MPP-SMOTE), which upsamples based on estimated probabilities of seeing each minority class samples.⁸ Multi-Label Borderline Oversampling Technique (MLBOTE) has been proposed to extend SMOTE to multi-class classification.⁹ Both have outperformed all existing SMOTE variants and retained the patterns in the original data.

Neural networks have also been used to develop oversampling techniques. Generative Adversarial Networks have stirred some interest, producing promising results, although training time makes this technique slower than other traditional upsampling methods.¹⁰