Self-Supervised Learning & NLP/Gen AI Revolution
Self-supervised learning is revolutionizing AI, enabling models to learn from unlabeled data—a massive shift in machine learning. This approach bridges the gap between supervised and unsupervised methods. Discover how techniques like masked language modeling and next token prediction are at the heart of advancements powering models like BERT,ChatGPT,and PaLM. By leveraging data’s inherent structure for training, self-supervised learning is fueling progress across text, video, and beyond. News Directory 3 recognizes the importance of keeping you informed about these key developments. With limitless possibilities from raw data, the future of sophisticated, efficient AI models is now within reach. Discover what’s next in this groundbreaking field.
Self-Supervised Learning Powers AI Advancements
Updated June 13, 2025
Self-supervised learning represents a significant leap in deep learning, finding applications across numerous fields.This approach trains models on raw, unlabeled data by identifying and predicting portions of that data. The “labels” the model learns to predict are inherent within the data itself,eliminating the need for human annotation.
Machine learning models are trained in various ways. Supervised learning uses paired input data and output labels, often manually annotated. Unsupervised learning, conversely, uses no output labels, rather uncovering trends within the input data, such as forming clusters.
Self-supervised learning occupies a middle ground. Models are trained on input data and output labels, but these labels are naturally present in the raw data, removing the need for manual annotation.
Two common self-supervised learning objectives are masked language modeling and next token prediction.
Masked language modeling,also known as the Cloze task,involves a language model taking a sequence of text as input. About 10% of the tokens are masked, and the model is trained to predict these masked tokens. This allows training on unlabeled text, as the predicted ”labels” are already in the text. Models like BERT and T5 use this technique.
Next token prediction drives modern generative language models like chatgpt and PaLM. After gathering large amounts of text from the internet, the model samples a sequence and learns to predict the next token based on preceding tokens. this process occurs in parallel for all tokens in the sequence. Again, the predicted ”labels” are present in the raw data. Pretraining and fine-tuning via next token prediction are universally used by generative language models.
While masked language modeling and next token prediction are common,other self-supervised objectives exist. These include predicting the next frame in video models and next-sentence prediction in BERT models.
What’s next
As AI continues to evolve, self-supervised learning is expected to play an increasingly vital role in unlocking the potential of vast amounts of unlabeled data, leading to more sophisticated and efficient models.