ML compilers, from zero Course GitHub

Build a machine learning compiler from scratch.

A free, open-source course that starts with interpreting add 2 3 and by week 5 has your compiler turning reshapes, matmuls, and convolutions into C. Read the lessons here, then write the compiler yourself, in any language, and grade it with each week's tests.

Start the courseGet the code

tests/week1/run/i32_div_trunc.uir
# expect: pass
# div truncates toward zero, like C (not floor like Python)
%a = div -7 2
%t0 = assert %a -3
%b = div 7 -2
%t1 = assert %b -3
%c = div -7 -2
%t2 = assert %c 3
python tools/test.py --compiler ./mine --week 1
FAIL tests/week1/run/i32_div_intmin.uir
    expected exit 0 (pass), got 1
FAIL tests/week1/run/i32_div_trunc.uir
    expected exit 0 (pass), got 1
FAIL tests/week1/run/lazy_unused_runtime_error.uir
    expected exit 0 (pass), got 1

139 / 142 passed   (-v for details)

Each week has a written spec and a test suite that pins it down. This compiler gets division wrong, and the grader names the three tests it breaks.

How it works

  1. 1

    Read a lesson

    Every idea comes with a diagram you can step through, a quiz, and a small exercise that runs on your machine.

  2. 2

    Write your compiler

    In OCaml, C, Rust, or anything that builds a program. The course gives you the spec, not the code.

  3. 3

    Run the tests

    One command grades your compiler against the week's suite and tells you which rule each failure breaks.

What you build

Six weeks so far. Week 0 teaches the C, memory, and OCaml the rest assumes, so you can start without a systems background.

  1. 0PrerequisitesC and pointers, caches, IRs, tensors, matmul, and an OCaml crash course
  2. 1UOps and rewritesAn interpreter for a tiny graph language, then constant folding and symbolic simplification
  3. 2Memory and rendering CBuffers, loads and stores, proving every access in bounds, and generating C with no undefined behavior
  4. 3LoopsRanges, accumulators, and a matmul and a convolution written by hand
  5. 4RangeifyWhole-vector math lowered to loops, reduce, and when to fuse or keep a buffer
  6. 5Movement opsReshape, permute, pad, and expand at any rank, compiled to index arithmetic

36 lessons, 221 quizzes, 84 exercises, and 501 tests. The tests catch every one of 158 bugs planted in the reference compiler, so a passing run means something.

Get started

The lessons read fine right here. To run the exercises and grade your compiler, get the course on your machine. You need Python 3.10 or newer and a C compiler (clang or gcc). OCaml is optional.

git clone https://github.com/wilsonmike/ml-compilers-course
cd ml-compilers-course
python tools/serve.py
# then open http://127.0.0.1:4111

The runner compiles your exercise code on your own computer and only listens on 127.0.0.1.

Get an email when new weeks ship

Weeks 6 and on, and the courses after this one. One email per release, and every email has an unsubscribe link.