While few-shot learning can also be applied to regression tasks (or even reinforcement learning), most FSL literature focuses on classification use cases. Some FSL methods can used alongside other solutions that address scarcity of labeled data: for example, in semi-supervised learning methods that incorporate information from large amounts of unlabeled data alongside information from few-shot learning on the limited labeled samples available.¹

Transfer learning-based methods focus on adapting a pre-trained model to learn new tasks or previously unseen classes of data.

When few labeled samples are available, using supervised learning to train a model from scratch—especially one with a large number of parameters, like the convolutional neural networks (CNNs) typically used in computer vision or the transformer-based networks used in natural language processing (NLP)—often leads to overfitting: the model might perform well on test data, but poorly on real-world data. But gathering a sufficiently large amount of data to avoid overfitting is often a bottleneck in model training.

Transfer learning offers a practical solution: leverage useful features and representations that a trained model has already learned. One simple approach is to fine-tune a classification model to perform the same task for a new class through supervised learning on a small number of labeled examples. More intricate approaches teach new skills through the design of relevant downstream tasks–often meta learning tasks—to a model that been pre-trained via self-supervised pretext tasks: this is increasingly common in NLP, particularly in the context of foundation models.

More complex transfer learning approaches adapt a trained neural network via changes to the network architecture: for example, replacing or re-training the outer layers of a neural network, where final classification occurs, while maintaining the internal layers where feature extraction occurs. Freezing (or otherwise regularizing changes to) model weights for all but the outermost layers can ensure that subsequent updates don’t result in “catastrophic forgetting” of already-learned knowledge. This allows for greatly expedited learning in a few-shot context.

Transfer learning is most successful when the model’s initial training is relevant to the new task. For example, a model trained on certain species of birds will generalize well to unseen species of birds after fine-tuning with only a few labeled samples, because the learned weights of the filters the CNN uses for convolutions are already optimized to capture features relevant to bird classification (like plumage, beaks, wing size, etc.)—but using few-shot learning to teach the same model to recognize vehicles will yield less satisfactory performance.

An alternative solution to the problem of limited labeled data samples is to generate additional training samples. This is particularly useful when real-world examples of a given class of data are exceedingly scarce, as may be the case when dealing with rare diseases or exotic species.

Data generation, via generative models like Generative Adversarial Networks (GANs) or variational autoencoders (VAEs), can potentially yield enough samples resembling the original labeled samples to perform conventional supervised learning, provided the original samples had sufficient diversity to avoid overfitting.

Data augmentation, the process of creating new samples by applying different transformations to original samples, can be combined with other methods: for example, it can be used to create matching samples for use in metric meta learning in a process similar to contrastive self-supervised learning.

Unlike supervised learning or fine-tuning, in which a classifier is trained on the exact tasks it will be used for and the training set contains the same classes the model will be tested on, meta learning takes a broader, more indirect approach. Whereas approaches built upon transfer learning adapt pre-trained models, meta learning methods often train systems end-to-end from scratch. 

According to Santoro, et al, “meta learning” refers to scenarios in which multiple tasks are used to train a model at both a short-term and long-term level. Within each task, the model learns rapidly to make predictions relevant to the limited domain of that specific task; across tasks, the model gradually accrues knowledge by capturing the way patterns and task structure vary across target domains. This two-tiered process is often described as the model “learning to learn.” ²

For example, the goal of many prominent meta learning methods is to train a model function, across multiple training episodes, to output a prediction for the degree of similarity between data points from any classes—including classes the model has not yet seen—to then use learnings from that process to solve downstream tasks (like specifically defined classification problems).

Some meta learning approaches work on a more abstract level, by training models to be easy to train. In traditional supervised learning, a model’s parameters (like weights and biases) are what’s “learned,” while the model’s hyperparameters—like the learning rate, or how parameters are initialized—are configured prior to training and not part of the learning process. Meta learning can approximate the benefits of transfer learning by learning ideal starting points: parameter initializations or other hyperparameter choices that will generalize well to different datasets in a minimal amount of training steps.

Though a wide variety of machine learning model architectures can be used for few-shot learning, the structure of FSL training and evaluation generally follows an N-way-K-shot framework, in which N represents the number of classes and K represents the number of examples (or “shots”) provided for each class.

 In N-way-K-shot classification, the model undergoes multiple episodes of training. Each training episode consists of one or more training tasks. Models are evaluated via test tasks, whose structure mirrors that of the training tasks. Each training task (and test task) comprises two datasets:

• The support set contains K labeled training samples for each of the N classes. The model uses these support samples to learn generalized representations for each class. For example, the dataset for a 3-way-2-shot classification task contains 3 classes of images and provides 2 examples of each. When K=1, the task is one-shot learning. When K=0, the problem is zero-shot learning—which typically requires unique solutions.
• The query set contains one or more new examples for each of the N classes. Using representations learned from the support set, the model predicts classification for each example in the query set. A loss function measures the divergence (“loss”) between the model’s predictions and the “correct” predictions; after each training episode, model parameters are adjusted—optimized—to minimize loss. 

Because the goal of meta-learning is to train models to generalize well to unseen data, rather than to recognize any specific classes of data, each training task typically includes different data classes than those used in any preceding training tasks.

To test the model’s ability to make accurate similarity predictions for heretofore unseen classes, the support set and query set used for testing must contain entirely new classes of data that the model has not yet been exposed to in training tasks.