Showing 241-300 of 314 articles
Rotary Position Embedding
Rotary Position Embedding (RoPE) is a positional encoding method for transformer models that encodes a token's absolute position by rotating its query and key...
Large Language ModelsModel Architecture
SDEdit
SDEdit (Stochastic Differential Editing) is a method for guided image synthesis and editing that turns a rough user guide, such as a stroke painting, a coarse...
Generative AI
SOAP (optimizer)
SOAP (ShampoO with Adam in the Preconditioner's eigenbasis) is a second-order optimization algorithm for training deep neural networks, introduced by Nikhil...
Training & Optimization
SSD (Single Shot MultiBox Detector)
SSD (Single Shot MultiBox Detector) is a single stage object detection model that predicts bounding boxes and per class confidence scores in one forward pass...
Computer Vision
Saliency map
A saliency map is an explainable AI visualization that highlights which parts of an input, most often the individual pixels of an image, most influenced a deep...
Computer VisionInterpretability
SambaNova Systems
SambaNova Systems is an American artificial intelligence hardware and software company headquartered in Palo Alto, California. Founded in November 2017 by...
AI CompaniesAI Hardware
Scaling Laws
Neural scaling laws are empirical, power-law relationships that describe how the performance of neural networks in artificial intelligence improves predictably...
Large Language ModelsMachine Learning
Schedule-Free optimizer
Schedule-Free is a family of optimization algorithms for deep learning and convex stochastic optimization that matches or exceeds the performance of tuned...
Training & Optimization
Self-Extend
Self-Extend (written SelfExtend in the original paper) is a training-free technique that lets a pretrained large language model process inputs much longer than...
Neural Networks
Self-Supervised Learning
Self-supervised learning (SSL) is a machine learning approach in which a model learns representations from unlabeled data by generating its own supervisory...
Machine Learning
Self-attention
Self-attention is a mechanism that lets a neural network weigh how much every element of a single input sequence should influence every other element,...
Machine LearningModel Architecture
Self-training
See also: Machine learning terms Self-training is a semi-supervised learning procedure in which a model trained on a small labeled set is used to generate...
Machine Learning
Semantic Segmentation
Semantic segmentation is a computer vision task that assigns a category label to every single pixel in an image, producing a dense map in which each pixel...
Computer Vision
Sequence Model
A sequence model is a class of machine learning models that processes, generates, or predicts ordered data where the position and surrounding context of each...
Machine LearningNatural Language Processing
Sequence Parallelism
Sequence parallelism (SP) is a family of distributed training techniques for transformer-based neural networks that partitions activations along the sequence...
AI InfrastructureTraining & Optimization
Sequence-to-Sequence Task
See also: Machine learning terms, Natural language processing, Recurrent neural network A sequence-to-sequence (seq2seq) task is any machine learning problem...
Machine LearningNatural Language Processing
Shampoo (optimizer)
Shampoo is a second-order stochastic optimization algorithm for training neural networks that maintains, for each parameter tensor, a set of small full-matrix...
Training & Optimization
Show-o
Show-o is a unified multimodal model, introduced in 2024, that handles both multimodal understanding and visual generation inside a single Transformer. Its...
Generative AI
Sigmoid Function
See also: Machine learning terms The sigmoid function is a mathematical function that maps any real number to a value between 0 and 1 using the formula ,...
Machine LearningMathematics
SimCLR
SimCLR (Simple Framework for Contrastive Learning of Visual Representations) is a self-supervised learning method for computer vision in which a network is...
Computer VisionMachine Learning
SmoothGrad
SmoothGrad is a saliency map technique that reduces visual noise in gradient-based explanations of neural network predictions by averaging gradients over many...
Interpretability
Soft Actor-Critic
Soft Actor-Critic (SAC) is an off-policy, maximum-entropy deep reinforcement learning algorithm that trains a stochastic actor-critic to maximize expected...
AlgorithmsReinforcement Learning
Soft MoE
Soft MoE (Soft Mixture of Experts) is a fully differentiable variant of the sparse mixture of experts (MoE) layer. Instead of routing each token to a small set...
Neural Networks
Softmax
See also: Machine learning terms The softmax function is a mathematical function that converts a vector of real numbers, often called logits, into a...
Machine LearningNeural Networks
Sophia (optimizer)
Sophia (a backronym for Second-order Clipped Stochastic Optimization with Adaptive estimator) is a stochastic second-order optimization algorithm introduced in...
Training & Optimization
Sparse attention
Sparse attention is a family of techniques that cut the computational and memory cost of the attention mechanism in transformer models by letting each token...
Machine LearningModel Architecture
Sparse autoencoder
A sparse autoencoder (SAE) is a neural network that adds a sparsity penalty to an autoencoder's training loss so that only a small number of hidden units...
InterpretabilityMachine Learning
Sparsity
See also: Machine learning terms Sparsity is the property of a data structure, model, or representation in which most values are zero or near-zero, so that...
Machine Learning
Speculative Decoding
See also: transformer, large language model, inference, KV cache, autoregressive model, knowledge distillation Speculative decoding is a lossless inference...
AI InferenceLarge Language Models
Speech recognition
Speech recognition, also known as automatic speech recognition (ASR), is the technology that converts spoken language into written text. It sits at the...
Machine LearningNatural Language Processing
Staged training
Staged training (also called multi-stage training or phased training) is a strategy in machine learning where a model is trained in two or more distinct...
Training & Optimization
State space model (deep learning)
A state space model (SSM) in deep learning is a class of sequence model that maps an input sequence to an output sequence through a fixed-size latent state,...
Machine Learning
Stochastic Gradient Descent (SGD)
See also: Machine learning terms Stochastic gradient descent (SGD) is an iterative optimization algorithm that updates model parameters using the gradient of...
Machine LearningTraining & Optimization
Stride
Stride is the step size by which a filter (or pooling window) moves across the input in a convolutional neural network (CNN): a stride of 1 shifts the filter...
Computer VisionMachine Learning
Supervised fine-tuning
Supervised fine-tuning (SFT) is the process of further training a pre-trained large language model (LLM) on a smaller, labeled dataset of input-output pairs so...
Artificial IntelligenceMachine Learning
SwiGLU
SwiGLU (Swish-Gated Linear Unit) is the activation function used inside the feed-forward sublayer of most modern transformer large language models, including...
Model ArchitectureNeural Networks
Swin Transformer
The Swin Transformer (Shifted Window Transformer) is a hierarchical vision transformer architecture that computes self-attention within local, non-overlapping...
Computer VisionNeural Networks
Tanh (hyperbolic tangent)
The hyperbolic tangent, written tanh, is a smooth, S-shaped activation function that maps any real number into the open interval , passing through the origin...
Neural Networks
Target Network
A target network is a separate, slowly updated copy of a neural network used in deep reinforcement learning to compute stable learning targets, decoupling the...
Machine LearningReinforcement Learning
Tensor
In machine learning, a tensor is a multi-dimensional array of numbers that serves as the fundamental data structure for representing and manipulating data....
Machine LearningMathematics
Tensor Processing Unit (TPU)
A Tensor Processing Unit (TPU) is a custom application-specific integrated circuit (ASIC) built by Google to accelerate machine learning workloads, and it is...
AI HardwareAI Infrastructure
Tensor Shape
A tensor shape is a tuple of integers that describes the number of elements along each dimension (or axis) of a tensor. For a tensor of rank r, the shape is...
Machine LearningMathematics
TensorFlow
TensorFlow is a free, open-source software library for machine learning and numerical computation, developed by the Google Brain team and first released on...
AI Tools & ProductsGoogle
Text summarization
Text summarization is the natural language processing (NLP) task of automatically producing a shorter version of one or more documents that preserves the most...
Machine LearningNatural Language Processing
Textual Inversion
Textual Inversion is a technique for personalizing text-to-image diffusion models that teaches a frozen model a new visual concept from only three to five...
Generative AI
Token
A token is the basic unit of text that a language model reads and writes: a word, a subword fragment, a single character, or a byte, produced by splitting text...
Machine LearningNatural Language Processing
Training
Training in machine learning is the process of fitting a model's parameters to data so that the model can make accurate predictions or generate useful outputs....
Machine Learning
Transfer Learning
Transfer learning is a machine learning technique that reuses knowledge a model has gained on one task or domain to improve performance on a different but...
Machine Learning
Transformers
> Note: This article is about the neural network architecture introduced in 2017. For the open-source Python library by Hugging Face, see Hugging Face...
Model ArchitectureNeural Networks
Transfusion
Transfusion is a recipe, introduced by Meta in 2024, for training a single Transformer over a mixture of discrete text and continuous image data using two...
Generative AI
Translational invariance
Translational invariance (also called translation invariance or shift invariance) is the property of a function, system, or machine learning model whose output...
Computer VisionNeural Networks
Twin Delayed DDPG
Twin Delayed Deep Deterministic Policy Gradient (TD3) is an off-policy actor-critic reinforcement learning algorithm for continuous action spaces, introduced...
AlgorithmsReinforcement Learning
U-Net
U-Net is a convolutional neural network architecture designed for biomedical image segmentation. It was introduced by Olaf Ronneberger, Philipp Fischer, and...
Computer VisionMachine Learning
VGG
VGG (also called VGGNet) is a deep convolutional neural network architecture, introduced in 2014 by Karen Simonyan and Andrew Zisserman of the Visual Geometry...
Computer VisionNeural Networks
VQ-VAE (Vector Quantized Variational Autoencoder)
VQ-VAE (Vector Quantized Variational Autoencoder) is a generative neural network that compresses data into a grid or sequence of discrete tokens drawn from a...
Generative AI
VQGAN (Taming Transformers)
VQGAN (Vector Quantized Generative Adversarial Network) is a two-stage image-synthesis method that first compresses an image into a small grid of discrete...
Generative AI
Vanishing Gradient Problem
See also: Machine learning terms The vanishing gradient problem is a difficulty in training deep neural networks where the gradients used to update the network...
Machine LearningNeural Networks
Variational Autoencoder
A variational autoencoder (VAE) is a generative model that learns a probabilistic latent space representation of data using an encoder-decoder architecture...
Generative AI
Veo
Veo is a family of text-to-video generative AI models developed by Google DeepMind, and is best known as the first video model from a leading AI lab to...
Generative AIGoogle DeepMind
Visual Autoregressive modeling (VAR)
Visual Autoregressive modeling (VAR) is an image generation paradigm, introduced in 2024, that reframes autoregressive image synthesis as coarse-to-fine...
Generative AI