Week 0 · Lesson 4
Tensors are a buffer plus a way to read it
A tensor looks like a grid, but memory is a single row of numbers. Shape and strides are the rule that turns a grid position into a spot in that row. Most of an ML compiler is moving that rule around without moving any data.
One row of memory
A tensor is an array with any number of dimensions, also called axes. A single number has 0 axes, a vector has 1, a matrix has 2, a stack of matrices has 3.
Lesson 1 showed that memory is a long array of bytes. A 3 by 4 matrix of floats is 12 floats stored back to back. Nothing in memory says "3 rows". The grid is a story we tell about the buffer.
The story has three parts:
- shape: how many positions each axis has, like
(3, 4). "3 rows, 4 columns." - strides: how far to move in the buffer when one index goes up by 1. For
(3, 4)stored row by row, they're(4, 1). Moving down a row skips 4 cells, and moving right a column skips 1. - offset: where element
[0, 0]lives. Usually 0.
To find a position, multiply each index by its stride and add them up:
flat = offset + i * stride[0] + j * stride[1]That one formula is the whole lesson. Every view below just changes the shape, strides or offset. The buffer never changes.
Strides here count elements, not bytes, so lesson 2's "C * 4 bytes" is a stride of C here. (NumPy counts strides in bytes, but PyTorch and this course count elements.)
Notation: (4,) is a shape with one axis of size 4 (the comma makes it a tuple of one number). "Contiguous" means stored back to back with no gaps. You'll also see ndim, the number of axes, and numel, the total number of elements.
Row-major
"Row-major" (C order) means the last axis is contiguous: its stride is 1. Each earlier stride is the product of all the sizes after it. For shape (2, 3, 4) the strides are (12, 4, 1).
Row-major strides
Write row_major_stride(ndim, shape, d): the stride of axis d in a contiguous row-major buffer of that shape. It's the product of every size after axis d, and the last axis is always 1.
Explore: one buffer, six views
Pick a view, then hover a cell on the left, or step through the cells in order. The highlighted cell on the right is the one actually read. Watch which views read the buffer in order, and which ones jump around.
Reshape: new shape, same buffer
Reshaping (3, 4) to (2, 6) or (4, 3) just recomputes row-major strides for the new shape. The read order is still 0, 1, 2, ... 11, so no data moves. This works because the buffer is contiguous. Reshaping a buffer that has already been permuted isn't always possible without a copy, and week 5 deals with exactly that.
Permute: swap the strides
A transpose of (3, 4) is shape (4, 3) with strides (1, 4). Same buffer, swapped strides. It's free to create, but reading it row by row now jumps 4 cells at a time. Lesson 2 showed why that's slow: every jump can land on a different cache line.
Broadcast: stride 0
Adding a 4-element vector to every row of a 3 by 4 matrix needs the vector to "look like" 3 by 4. Give the row axis stride 0. Moving down a row then moves 0 cells, so every row reads the same 4 numbers. No copy happens.
Shrink and pad: index ranges and a mask
Shrink takes a window. The middle 2 by 2 of our matrix, written [1:3, 1:3] (rows 1 to 2 and columns 1 to 2; the end of a range is excluded), starts at offset 5, because element [1, 1] is at 1·4 + 1 = 5. It keeps the same strides (4, 1). Only the offset and shape change.
Pad goes the other way and adds positions that have no memory behind them. A pad of 1 on each side of the columns turns (3, 4) into (3, 6). A logical column j is valid only when 1 ≤ j < 5. Valid positions read buffer column j - 1. Invalid ones don't touch memory at all and produce 0. That "is this index valid" check is a mask. It's also why the course bounds-checks every load: a padded index that skipped its mask would read someone else's memory.
Read through a pad
A row of n real values is padded with left zeros in front and any number of zeros after. Write padded_get(row, n, left, j), which returns logical element j of the padded view: row[j - left] when that is a real element, and 0 for every other j, including negative j and j past the end.
The starter forgets the mask. The tests put other numbers right next to the row, so reading outside it shows up as a wrong answer, which is exactly the bug a missing mask causes in a real kernel.
A view is not data. Writing to a broadcast view would mean three logical cells sharing one memory cell. Compilers either forbid it or copy first (materialize). Weeks 4 and 5 are mostly about deciding when to copy.
In weeks 4 and 5 your compiler represents RESHAPE, PERMUTE, PAD and SHRINK exactly like this. They stay index math until a loop reads through them. A later step (called rangeify, in week 4) turns the shape into loops and the strides into the index math inside each memory read.
Code it
Where does logical element k live?
A view is shape, strides and offset over a buffer. Number its logical positions 0 .. numel-1 in row-major order. Write view_at(ndim, shape, strides, offset, k), which returns the buffer index that position k reads.
Turn k back into one index per axis (the last axis changes fastest), then apply flat = offset + Σ index[d] * strides[d]. For shape (3, 4) and k = 5: the last index is 5 % 4 = 1, then 5 / 4 = 1 is the row index, so the position is [1, 1]. It has to work for every view on this page: reshape, permute, broadcast (stride 0) and shrink (an offset).
With ndim = 0 there is one element, and it lives at offset.
Check yourself
A row-major tensor of shape (3, 4). What are its strides? Answer like (a, b).
The last axis is contiguous, so its stride is 1.
One step down a row skips a whole row. How long is a row?
The last axis is contiguous (stride 1). The row stride is the row length, 4.
Same tensor. Which flat buffer index holds element [2, 1]?
Multiply each index by its stride and add: i * stride0 + j * stride1.
2 * 4 + 1 * 1 = 9.
Row-major strides for shape (2, 3, 4)?
Each stride is the product of the sizes of every axis after it.
The last stride is 1; the middle one is the length of the last axis.
Each stride is the product of the sizes after it: 3*4 = 12, 4, then 1.
You permute (3, 4) with strides (4, 1) to shape (4, 3). What happens to the buffer?
Permute only relabels the axes. Does that need to move any bytes?
Whatever happens to the shape happens to the strides, in the same order.
Permute reorders the axes, so it reorders the strides with them. The buffer is untouched.
Reshape the contiguous (3, 4) buffer to (2, 6). Which flat index holds [1, 0]?
Reshape of a contiguous buffer keeps the bytes and recomputes row-major strides for the new shape.
New strides (6, 1), so 1 * 6 + 0 = 6.
A 4-element vector is broadcast to shape (3, 4). What are the strides?
Every row of the result reads the same 4 values. How far does moving down one row need to move in the buffer?
Rows repeat, so the row axis moves 0 cells. Columns walk the vector with stride 1.
A (4,) buffer holding [5, 6, 7, 8] is padded by 1 on each side to (6,). What does logical index 0 read?
Padding by 1 on the left shifts the real data: logical index i reads buffer index i - 1.
Is buffer index -1 a real element, or does the mask stop the read?
Index 0 is outside the valid range 1 ≤ i < 5, so the mask is false. Nothing is loaded and the value is 0. Index 1 would read buffer index 0, which is 5.