Tensors
Lesson 1 of 8
1Tensors
The big pictureA tensor is a NumPy array that knows how to differentiate itself and how to move to a GPU. Three properties govern every tensor: shape, dtype and device, and most of your PyTorch errors will be a dispute over one of them.
What the picture shows: Every number in the shape has a meaning and a dimension index under it. When you write dim=1 you pick the channels dimension, and that is the one that disappears from the shape. Reading shapes this way catches half your PyTorch bugs before you run anything.
Map of this part
The concepts in order, click any one to jump to it:
Two ways into the same idea
A tensor is a box of numbers with any number of dimensions: one number, a row, a table, or a stack of tables like a colour image. The key question you always ask is: what is its shape? How many rows, columns and layers. Most deep learning bugs are one shape not matching another.
A torch.Tensor has a shape, a dtype (like float32) and a device (cpu or cuda). Operations need compatible shapes and the same device, and view, reshape and permute change shape, often without copying. Printing x.shape after every step is the fastest debugging there is.
Before You Start
🧮Arrays with NumPy
You know shapes, indexing and broadcasting from the arrays lesson, and a tensor works almost the same way.
📉Derivatives and gradients
You know what a gradient is and why training needs it, since a tensor can track operations to compute one later.
The Core Idea
A tensor is the array you know from NumPy, with two extras. In this lesson you read its three properties, then learn which dimension disappears, then change the shape, then see how the type sets the memory.
1Shape, dtype and device
Every tensor has three properties: the shape says how many elements each dimension holds, the dtype says how each number is stored, and the device says where it lives. Most of your PyTorch errors are a dispute over one of them.
The first dimension is almost always the batch size, because a model processes several examples at once. The dtype is float32 for weights and inputs, and torch.long for class indices.
Look at the picture: the tensor x in the next example has shape (2, 3), two rows of three numbers each. Print these three properties before anything else.
2The dimension that disappears: dim
Operations like sum and mean fold a whole dimension into one number. You choose that dimension with dim. Look at the picture: the window moves across the columns, and each column of two numbers becomes one number.
This code needs PyTorch on your machine, so the output shown is recorded from a real run. Install it with pip install torch and run it yourself.
import torch x = torch.tensor([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]]) print(x.shape, x.dtype, x.device) print(x.sum(dim=0)) print(x.sum(dim=1)) print(x.mean(dim=1, keepdim=True).shape)
dim=0 sums across rows, leaving one value per column (3 values), and dim=1 sums across columns, leaving one per row (2 values). The rule: the dimension you name is the one that disappears. keepdim=True keeps it with length 1, which is what you usually need to subtract the mean from each row by broadcasting.
3Changing the shape: view and permute
view and reshape change the shape without changing the data, permute reorders the dimensions, and unsqueeze adds a dimension of length 1. Look at the picture: a batch of images becomes a batch of vectors.
Before any linear layer you need to turn each image into a vector. -1 means "work this dimension out yourself".
import torch
batch = torch.randn(32, 3, 28, 28)
flat = batch.view(32, -1)
print("images:", tuple(batch.shape))
print("flat: ", tuple(flat.shape))
print("one channel:", tuple(batch[:, 0].shape))
print("channels last:", tuple(batch.permute(0, 2, 3, 1).shape))
try:
batch.view(32, 100)
except RuntimeError as e:
print("error:", str(e).split(".")[0])
3 × 28 × 28 = 2352, so every image became a 2352-long vector. permute(0, 2, 3, 1) moves channels to the end, the order image libraries like Pillow and matplotlib use. The last line shows the helpful error: 75264 elements cannot become 32 × 100.
4The dtype decides the memory
Each float32 number takes 4 bytes, and each float16 number takes 2. Look at the picture: the same million numbers, in half the memory.
import torch
big = torch.ones(1000, 1000)
print(big.dtype, big.element_size() * big.nelement() / 1e6, "MB")
half = big.to(torch.float16)
print(half.dtype, half.element_size() * half.nelement() / 1e6, "MB")
device = "cuda" if torch.cuda.is_available() else "cpu"
print("device:", device)
A million float32 numbers take 4 megabytes, half that in float16. A seven-billion-parameter model takes 28 GB in float32 and 14 in float16, exactly the difference between a GPU that fits it and one that does not. We return to this in the speed and quantisation lessons.
Mixing types in one operation either raises an error or converts silently. Integers are for indices and labels, floats for everything else.
How It Works, Step by Step: how you read and change a tensor's shape
Print x.shape, x.dtype and x.device before anything else, since these three properties govern every operation.
Read the shape aloud: (32, 3, 28, 28) means 32 images, each with 3 channels, each channel 28 by 28.
In operations like sum and mean, the dimension you name is the one that disappears, unless keepdim=True keeps it with length 1.
To flatten images use view(32, -1), and to reorder dimensions use permute, not view.
Worked example
If you understood the arrays lesson in the data track, you already know 80% of tensors. The difference is that a tensor carries the device it lives on, and can remember the operations that made it so it can compute gradients later.
2. Read the shape as a sentence
(32, 3, 28, 28) reads: "32 images, each with 3 colour channels, each channel 28 by 28 pixels". Get used to saying shapes aloud like that and you will catch half your bugs before running the code.
view with permute. x.view(32, 28, 28, 3) on a (32, 3, 28, 28) tensor raises no error, since the element count matches, but it scrambles pixels across channels and the images become noise. To reorder dimensions always use permute.
forward, like # (B, 3, 28, 28) -> (B, 2352). This small habit saves hours when you return to the code a month later.
Try it without code
Myth vs Reality
| Myth | Reality |
|---|---|
| A tensor is something completely different from a NumPy array. | It is the same idea with mostly the same operations. It adds only two things: the device it lives on, and tracking operations to compute gradients. |
| If no error appears, the reshape is correct. | view(32, 28, 28, 3) on a (32, 3, 28, 28) tensor raises no error because the element count matches, but it scrambles pixels across channels and the images become noise. |
| The number type is a detail that changes nothing. | A million float32 numbers take 4 megabytes and float16 half that. A seven-billion-parameter model takes 28 GB or 14, which can be the difference between a GPU that fits it and one that does not. |
Try it yourself
An image tensor of shape (8, 3, 64, 64). What is the shape of x.mean(dim=1)?
Dimension 1 is channels, so it is folded away, leaving for each image the mean of its three channels per pixel, that is a greyscale image. The last option is what you get with keepdim=True.
You get: Expected all tensors to be on the same device, but found cuda:0 and cpu. Most likely cause?
The operation needs every tensor on the same device. The right pattern: model.to(device) once, and xb, yb = xb.to(device), yb.to(device) inside the loop for each batch.
Quick review cards
dim means in an operation like sumkeepdim=True, which keeps it with length 1.float32 is the training default, float16 or bfloat16 take half the memory, and integers are for indices and labels. Mixing types in one operation either errors or converts silently..to(device), or you get "Expected all tensors to be on the same device".Check Your Understanding
You want to subtract each row's mean from a (16, 10) tensor. Why do you need keepdim=True?
Without it the mean has shape (16,), and broadcasting compares dimensions from the right: 10 against 16, which do not match, so it errors (or, if the two happened to be equal, subtracts the wrong thing silently). With keepdim=True the shape is (16, 1), which broadcasts across the ten columns and subtracts each row's mean from its own row.
What shape do you get from batch.permute(0, 2, 3, 1) on a (32, 3, 28, 28) tensor?
(32, 28, 28, 3): the channels moved to the end, the order libraries like Pillow and matplotlib use.
Resources for this lesson
Grouped by type. Start with the official docs for precision, or an article or video for the explanation.
Official docs3
Done with this lesson?
Sign in to save your progress.