Training & Optimization

LongLoRA

LongLoRA is a parameter-efficient fine-tuning technique that extends the context window of pre-trained large language models with substantially lower...

Large Language Models

Loss

See also: Machine learning, Loss function, Gradient descent In machine learning, loss (sometimes called error) is a single non-negative number that measures...

Machine Learning

Loss Curve

See also: Machine learning terms, Loss function A loss curve is a plot that shows the value of a loss function over the course of training a machine learning...

Deep LearningMachine Learning

Loss Function

See also: Machine learning terms A loss function is a mathematical formula that measures how far a machine learning model's prediction is from the correct...

Deep LearningMachine Learning

Loss Surface

The loss surface (also called the loss landscape, the error surface, or the objective function surface) is the geometric representation of a loss function as a...

Deep LearningMachine Learning

Masked autoencoder (MAE)

Masked autoencoder (MAE) is a self-supervised learning method for vision transformers that masks roughly 75% of an input image's patches and trains a network...

Computer VisionMachine Learning

Megatron-LM

Megatron-LM is NVIDIA's open-source framework for training very large transformer language models across GPU clusters, and the name of the tensor-parallelism...

NVIDIAOpen Source AI

Mini-batch stochastic gradient descent

Mini-batch stochastic gradient descent (often shortened to mini-batch SGD or MB-SGD) is the optimization algorithm used to train almost every modern neural...

Minimax Loss

Minimax loss is a loss function rooted in game theory and decision theory that measures the worst-case performance of a strategy, algorithm, or model. In...

Generative AIMachine Learning

Minitron

Minitron is a family of compact language models from NVIDIA, together with the model-compression method used to build them: take one large, already pretrained...

NVIDIASmall Language Models

Model Predictive Control

Model Predictive Control (MPC), also called receding horizon control (RHC), is a feedback control method that, at every sampling instant, solves a finite...

Robotics

Momentum

See also: Machine learning terms, Optimizer, Gradient descent Momentum is an optimization technique that accelerates gradient descent by accumulating an...

Multi-token prediction

Multi-token prediction (often abbreviated MTP) is a language modeling training objective in which the model is trained to predict several future tokens at each...

Large Language ModelsMachine Learning

Muon (optimizer)

Muon (short for MomentUm Orthogonalized by Newton-Schulz) is a neural-network optimizer that updates the two-dimensional weight matrices of hidden layers by...

Deep Learning

NCCL (NVIDIA Collective Communications Library)

NCCL (the NVIDIA Collective Communications Library, pronounced "Nickel") is an open-source library from Nvidia that provides topology-aware, multi-GPU and...

AI HardwareAI Infrastructure

NaN Trap

See also: gradient descent, backpropagation, loss function, mixed precision training, numerical stability A NaN trap (short for "Not a Number" trap) is a...

Deep LearningMachine Learning

NormalFloat 4-bit (NF4)

NormalFloat 4-bit (NF4) is a 4-bit numerical data type for storing the weights of deep neural networks, introduced in the 2023 QLoRA paper by Tim Dettmers,...

AI Inference

ORPO

ORPO (Odds Ratio Preference Optimization) is a preference alignment algorithm for large language models that merges supervised fine-tuning and preference...

Large Language ModelsMachine Learning

Objective

See also: Machine learning terms In machine learning, an objective (or objective function) is the scalar function that a learning algorithm optimizes during...

Machine Learning

Objective function

See also: Machine learning terms An objective function is the single scalar-valued quantity that an optimization algorithm tries to minimize or maximize during...

Optimizer

An optimizer in machine learning is an algorithm that iteratively adjusts a model's learnable parameters to minimize (or maximize) an objective function,...

Deep LearningMachine Learning

PEFT

Parameter-efficient fine-tuning (PEFT) is a family of methods that adapt a pre-trained large language model to a downstream task by updating only a small...

Parameter Server (PS)

See also: Distributed training, Machine learning systems The Parameter Server (PS) is a distributed system architecture for training large machine learning...

MLOps

Parameter update

See also: Machine learning terms A parameter update is the step in neural-network training where a model's trainable weights are adjusted using the gradient of...

Partitioning strategy

See also: Data parallelism, Model parallelism, Pipeline parallelism, Tensor parallelism, Distributed training A partitioning strategy in distributed deep...

MLOps

PiSSA

PiSSA (Principal Singular values and Singular vectors Adaptation) is a parameter-efficient fine-tuning method for large language models that initializes...

Large Language Models

Pipeline Parallelism

Pipeline parallelism (often abbreviated PP) is a distributed training strategy that splits the layers of a deep neural network across multiple accelerator...

AI InfrastructureLarge Language Models

Pipelining

See also: Machine learning terms Pipelining is a term used in two distinct senses within machine learning and artificial intelligence. The first refers to the...

MLOpsMachine Learning

Policy gradient methods

Policy gradient methods are a family of reinforcement learning algorithms that directly parameterise the agent's policy and optimise it by stochastic gradient...

Machine LearningReinforcement Learning

Pre-Trained Model

A pre-trained model is a machine learning model that has already been trained on a large, general-purpose dataset and can then be reused, either as a fixed...

Deep LearningMachine Learning

Pre-training

Pre-training is the first and most compute-intensive stage of building a modern AI model: a neural network is trained on a massive, mostly unlabeled dataset...

Artificial IntelligenceComputer Vision

Proximal Policy Optimization (PPO)

Proximal Policy Optimization (PPO) is an on-policy policy gradient reinforcement learning algorithm that stabilizes training by clipping the policy update so...

Machine LearningReinforcement Learning

Pruning

Pruning is a family of techniques used in machine learning and artificial intelligence to remove parts of a model or search space that are estimated to be...

AI InferenceMachine Learning

QLoRA

QLoRA (Quantized Low-Rank Adaptation) is a parameter-efficient fine-tuning method that finetunes a 65-billion-parameter large language model on a single 48 GB...

AI InferenceDeep Learning

RLOO (REINFORCE Leave-One-Out)

RLOO (REINFORCE Leave-One-Out) is an online reinforcement learning algorithm for aligning large language models with reward signals such as those derived from...

AI AlignmentReinforcement Learning

RLVR

Reinforcement Learning with Verifiable Rewards (RLVR) is a post-training paradigm for large language models in which the reward signal comes from a...

AI InferenceReasoning Models

RMSProp

RMSProp (Root Mean Square Propagation) is an adaptive learning-rate optimizer that divides each parameter's gradient by a running root-mean-square of that...

Deep Learning

Regularization

Regularization is a set of techniques used in machine learning to prevent overfitting by adding constraints or penalties during training that discourage a...

Deep LearningMachine Learning

Regularization Rate

The regularization rate (commonly denoted as or alpha) is a hyperparameter that controls the strength of the penalty applied to a model's parameters during...

Machine Learning

Ridge Regression

Ridge regression is a method of estimating the coefficients of a linear regression model by adding a penalty proportional to the sum of squared coefficients...

Machine Learning

Ridge Regularization

Ridge regularization, also known as L2 regularization or Tikhonov regularization, is a technique in statistics and machine learning that adds a squared L2-norm...

Machine LearningStatistics

Ring Attention

Ring Attention, formally Ring Attention with Blockwise Transformers, is a distributed algorithm for computing the self-attention operation of transformer...

Transformer Models

SOAP (optimizer)

SOAP (ShampoO with Adam in the Preconditioner's eigenbasis) is a second-order optimization algorithm for training deep neural networks, introduced by Nikhil...

Deep Learning

SPIN (Self-Play Fine-Tuning)

SPIN (Self-Play fIne-tuNing) is a post-training method for large language models introduced by researchers at the University of California, Los Angeles (UCLA)...

AI AlignmentLarge Language Models

Schedule-Free optimizer

Schedule-Free is a family of optimization algorithms for deep learning and convex stochastic optimization that matches or exceeds the performance of tuned...

Deep Learning

Self-Instruct

Self-Instruct is a semi-automated framework for aligning a pretrained large language model with instruction-following behavior by bootstrapping its training...

Data & DatasetsLarge Language Models

Self-Rewarding Language Models

Self-Rewarding Language Models (SRLM) is an iterative alignment method in which a single large language model alternately plays the role of policy (generating...

AI AlignmentLarge Language Models

Self-Taught Evaluator

Self-Taught Evaluator is a method for training a strong LLM-as-a-judge without any human preference annotations, using synthetic training data and an iterative...

Meta AIModel Evaluation

Sequence Parallelism

Sequence parallelism (SP) is a family of distributed training techniques for transformer-based neural networks that partitions activations along the sequence...

AI InfrastructureDeep Learning

Shampoo (optimizer)

Shampoo is a second-order stochastic optimization algorithm for training neural networks that maintains, for each parameter tensor, a set of small full-matrix...

Deep Learning

Shrinkage

See also: Machine learning terms Shrinkage in machine learning and statistics is a regularization technique that deliberately pulls model coefficient estimates...

Machine Learning

SimPO

SimPO (Simple Preference Optimization) is a reference-free offline preference learning algorithm for aligning large language models with human preferences. It...

AI AlignmentLarge Language Models

Sophia (optimizer)

Sophia (a backronym for Second-order Clipped Stochastic Optimization with Adaptive estimator) is a stochastic second-order optimization algorithm introduced in...

Deep Learning

Squared Hinge Loss

Squared hinge loss (also called L2 hinge loss or L2-loss) is a loss function used in machine learning for classification tasks, most commonly in support vector...

Machine Learning

Squared Loss

Squared loss, also called quadratic loss, L2 loss, or squared error loss, is a loss function that penalizes a prediction by the square of its error: for a true...

Machine LearningStatistics

Staged training

Staged training (also called multi-stage training or phased training) is a strategy in machine learning where a model is trained in two or more distinct...

Deep Learning

Step

See also: Machine learning terms In machine learning, a step is one discrete update in an iterative process, most often a single update of a model's parameters...

Step size

See also: learning rate, parameter update, Adam, SGD In machine learning, the step size (also called the learning rate, usually written as the Greek letter ...

Stochastic Gradient Descent (SGD)

See also: Machine learning terms Stochastic gradient descent (SGD) is an iterative optimization algorithm that updates model parameters using the gradient of...

Deep LearningMachine Learning

Supervised fine-tuning

Supervised fine-tuning (SFT) is the process of further training a pre-trained large language model (LLM) on a smaller, labeled dataset of input-output pairs so...

Artificial IntelligenceDeep Learning