PyTorch for Deep Learning
The 2026 Skills Guide

The torch.Tensor is the fundamental data structure in PyTorch — an n-dimensional array that can live on CPU or GPU and optionally participates in automatic differentiation. Three concepts you must understand deeply:

• Device management — .to(device) moves a tensor to a given device ('cpu', 'cuda', 'cuda:1', 'mps' for Apple Silicon). Writing device-agnostic code — device = torch.device('cuda' if torch.cuda.is_available() else 'cpu') — is standard practice. Forgetting to move a tensor to the right device is one of the most common runtime errors.
• Dtypes — float32 is the default. float16 and bfloat16 (brain float) are used for mixed-precision training. bfloat16 has the same exponent range as float32 (avoiding overflow/underflow) but half the precision — it is the preferred mixed-precision dtype on recent NVIDIA and Google TPU hardware. int8 is used for quantisation.
• Autograd — Setting requires_grad=True on a tensor tells PyTorch to track all operations involving it in a computation graph. Calling .backward() on a scalar loss traverses this graph via reverse-mode automatic differentiation, accumulating gradients in each leaf tensor's .grad. The gradient of the loss with respect to parameter θ is: ∂L/∂θ — exactly what gradient descent needs. Wrap inference in torch.no_grad() to skip graph construction and save memory.

torch.nn.Module is the base class for all neural network components in PyTorch. Every model — from a single linear layer to a billion-parameter transformer — is a subclass of nn.Module. Understanding how it works internally is essential for debugging and implementing custom architectures.

• __init__ and forward() — Define layers (as nn.Parameter or sub-modules) in __init__; define the forward computation in forward(). Parameters registered as attributes or via nn.ModuleList/nn.ModuleDict are automatically tracked by .parameters() and included in state_dict().
• .parameters() and .named_parameters() — Return iterators over all learnable parameters. Used to construct the optimiser: optim.Adam(model.parameters(), lr=1e-4). .named_parameters() gives (name, param) pairs, useful for weight decay exclusion (biases and layer norms typically excluded).
• .train() and .eval() — Switches the model between training and evaluation modes, affecting Dropout (disabled in eval) and BatchNorm (uses running statistics in eval rather than batch statistics). Forgetting to call model.eval() during inference is a common bug that causes inconsistent results.
• Key layers — nn.Linear, nn.Conv2d (with kernel_size, stride, padding parameters), nn.MultiheadAttention (the building block of transformers), nn.Embedding (for token embeddings), nn.LayerNorm (now preferred over BatchNorm in transformer architectures), nn.Dropout.

The canonical PyTorch training loop follows a specific pattern. Understanding why each step happens in this order — and what goes wrong if you change it — is a common interview topic:

1. optimizer.zero_grad() — Clear gradients from the previous step. PyTorch accumulates gradients by default (useful for gradient accumulation); call this at the start of each step unless you are deliberately accumulating.
2. Forward pass — outputs = model(inputs). This builds the computation graph.
3. Compute loss — loss = criterion(outputs, targets). The loss must be a scalar (or you must specify a gradient for vector losses).
4. loss.backward() — Reverse traversal of the computation graph, computing and accumulating gradients.
5. Gradient clipping (optional) — torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0) before the optimiser step. Essential for transformer training; prevents gradient explosions that destabilise training.
6. optimizer.step() — Update parameters using the computed gradients according to the optimisation algorithm (Adam, AdamW, SGD).
7. LR scheduler step — scheduler.step() after optimizer.step(). Common schedulers: CosineAnnealingLR, OneCycleLR, LinearWarmupCosineDecay (for transformers).

Gradient accumulation: When GPU memory prevents using the desired batch size, accumulate gradients over N mini-batches before calling optimizer.step(). Effective batch size = mini-batch size × N. Divide the loss by N to keep gradient scale consistent.

PyTorch's data loading system is designed for performance at scale. Understanding Dataset and DataLoader is essential for building efficient training pipelines.

• Map-style Dataset — Implement __len__() and __getitem__(idx). DataLoader calls these to fetch samples. Suitable when all data fits in a structure accessible by index.
• Iterable-style IterableDataset — Implement __iter__(). Used for streaming datasets that cannot be indexed — e.g., data read from a database or a stream. Requires careful handling of worker processes to avoid duplicate data.
• DataLoader key parameters — num_workers: number of subprocesses for data loading (typically 4–8; 0 runs in the main process). pin_memory=True: allocates data in pinned (page-locked) host memory for faster CPU→GPU transfers. prefetch_factor: number of batches loaded in advance per worker. persistent_workers=True: keep worker processes alive between epochs.
• Custom collate_fn — Controls how a list of samples is assembled into a batch. Essential for variable-length sequences (e.g., padding text to the longest sequence in the batch), multi-modal data, or nested data structures.
• Samplers — WeightedRandomSampler for imbalanced datasets. DistributedSampler for DDP training (ensures each process gets a non-overlapping subset). Sampler is passed to DataLoader; using DDP without DistributedSampler causes each process to train on the full dataset.