Level 2
Weigh more than one design against each other, then build the core of your choice in the practical exam.
Who can apply
Open one year after obtaining the Level 3 certificate.
Required programming language
Required at every level, from Level 5 to Level 1.
Shell Scripting
The single most important language for working with AI.
The is useful for running , managing files, and automating development. can also run through , apps, notebooks, and . A is one useful interface, not a requirement for every AI user.
Learn shell programmingThe one question, at every level
You want to build an artificial intelligence completely from scratch. Design how you would build it.
Every level from 5 to 1 answers this same question. What changes is the depth expected of the answer, and the constraints attached to it.
What to learn
- Produce two or more designs and compare them.
- Weigh speed, cost and accuracy against each other in numbers.
- Build the core of your design yourself in the practical exam.
How you are marked
- Is the reason for your choice backed by numbers?
- Did you say what you gave up in exchange?
- Does what you built match what you designed?
Exam format
Stage 1 is a written paper at the venue; Stage 2 is an interview followed by a practical build. Both stages use the same question.
Held at a designated exam venue. You write your answers on the exam paper by hand. This stage tests the design you can put into words.
An examiner interviews you about your design, then you build the for real. This stage tests whether the plan on paper actually works in your hands.
Practice questions
The official exam gives you just the one common question above. The 20 questions below are practice questions that check the words, names and facts you need to know before you write that design. Every one is a short-answer question with a single correct answer: read the description, write your answer, then open the answer to check. They are not official questions.
Evaluation metrics
-
1
Of all the items the model labeled "positive", what do you call the share that are really positive? (For example, the share of mails flagged as spam that really are spam.)
Show answer
Answer Precision
With low precision, many good mails end up in the spam folder.
-
2
Of all the items that are really positive, what do you call the share the model found? (For example, the share of real spam that got caught.)
Show answer
Answer Recall
With low recall, lots of spam reaches the inbox. Raising precision usually lowers recall, and the other way round.
-
3
What is the metric that combines precision and recall into one number using their harmonic mean?
Show answer
Answer F1 score
If one of the two is high and the other is low, the F1 score is low too.
-
4
What do you call the table that counts the true answers against the model's predictions in the cells true positive, false positive, false negative and true negative?
Show answer
Answer Confusion matrix
Accuracy, precision and recall are all calculated by counting the cells of this table.
-
5
What is the language model metric that shows how unsure the model is about the next token, where lower is better?
Show answer
Answer Perplexity
A perplexity of 10 roughly means the model is hesitating between about 10 candidates for the next token.
Speed and cost
-
6
What do you call the time from sending a question until the response comes back?
Show answer
Answer Latency
For chat services, the time until the first word appears matters most.
-
7
What do you call the amount of work done in a set time, such as tokens processed per second?
Show answer
Answer Throughput
Batching requests raises throughput but can raise the latency of a single request.
-
8
What is the method of training a small model (the student) to copy the outputs of a large model (the teacher), keeping similar performance at a smaller size?
Show answer
Answer Knowledge distillation
The small model runs faster and cheaper. It is a classic design trade between speed or cost and accuracy.
-
9
While generating text, what do you call the memory where a language model keeps the attention keys and values it already computed, so it can reuse them and run faster?
Show answer
Answer KV cache
It saves recomputing earlier tokens every time, but it uses more memory as the text grows longer.
-
10
What is the method of making a trained model lighter by cutting out weights or connections that matter little?
Show answer
Answer Pruning
The name comes from pruning the useless branches of a tree.
Design options
-
11
Write the abbreviation for the architecture that, for each input, computes only a few of many "expert" sub-networks, so it has many parameters but needs less computation.
Show answer
Answer MoE (Mixture of Experts)
The total parameter count is large, but only a few experts work on any one token.
-
12
Write the name of the low-cost fine-tuning method that freezes the original weights and trains only small low-rank matrices added to them.
Show answer
Answer LoRA (Low-Rank Adaptation)
It trains very few parameters, so even a large model can be fine-tuned with little memory.
-
13
Write the abbreviation for the method that builds a reward model from people choosing the better of two answers, then trains the language model with reinforcement learning on that reward.
Show answer
Answer RLHF (Reinforcement Learning from Human Feedback)
It is used to shape chat models so they answer the way people want.
-
14
What do you call the database that stores embedding vectors and quickly finds the vectors closest to a given one?
Show answer
Answer Vector database
In a RAG design, it finds the documents closest in meaning to the question.
-
15
What is the similarity, common in embedding search, that measures how alike two vectors' directions are by the angle between them?
Show answer
Answer Cosine similarity
It is 1 for the same direction, 0 at a right angle and -1 for opposite directions.
Implementation tools
-
16
Write the name of the Python library that provides multi-dimensional arrays and fast numerical computing, which most machine learning libraries are built on.
Show answer
Answer NumPy
It computes whole arrays at once instead of using loops, doing the same work far faster than plain Python.
-
17
Write the name of the popular Python deep learning framework with automatic differentiation that Meta first built and released.
Show answer
Answer PyTorch
It is now run by the PyTorch Foundation under the Linux Foundation.
-
18
What is the feature that works out a function's gradient (derivative) automatically by following the computation, without anyone solving the formula by hand?
Show answer
Answer Automatic differentiation (autograd)
It is the core feature of deep learning frameworks that saves you from writing backpropagation yourself.
-
19
Write the name of the most widely used activation function, applied to a layer's output, that turns values below 0 into 0 and leaves larger values as they are.
Show answer
Answer ReLU (Rectified Linear Unit)
As a formula it is max(0, x). It is cheap to compute, which speeds up training deep networks.
-
20
Write the name of the container tool that packs a program together with its environment (libraries and settings) so it runs the same on any computer.
Show answer
Answer Docker
It prevents "but it works on my machine". Use it to move what you built in the practical exam and run it as is.