AI Literacy Is the New Digital Literacy

Every few years a skill moves from specialist to baseline. Spreadsheet fluency did it in the 1990s, search literacy in the 2000s, and version control somewhere around 2012. Artificial intelligence is doing it now — but the skill that is becoming baseline is not the one most curricula are teaching.

Why prompting is not the skill

The first wave of AI education taught prompting: phrasing, role instructions, few-shot examples, chain-of-thought. It was useful and it aged badly. Model interfaces improved, defaults got better, and the specific incantations that worked in one release stopped mattering in the next. A curriculum built on phrasing tricks has a shelf life measured in months.

What has not aged is the harder skill underneath: deciding whether the output in front of you is good enough to act on. That is an evaluation problem, and evaluation is a discipline with forty years of literature behind it. It does not go out of date when a model version increments.

In our AI track we spend roughly one session on prompting and eleven on evaluation, measurement and failure analysis. Mentees find this frustrating for about three weeks, and then they start catching things their more experienced colleagues miss.

The four layers of AI literacy

We teach AI literacy as four stacked layers. Each one is useful alone; each one is dangerous without the layer beneath it.

1. Mechanical. What kind of system is this? A retrieval system, a classifier, a generative model, a hybrid? What was it trained on, when, and by whom? You cannot reason about failure modes without knowing the shape of the thing.
2. Statistical. What does the output distribution look like? Where is the model confident, where is it confidently wrong, and what does its confidence score actually represent? Calibration is the single most misunderstood concept in applied machine learning.
3. Evaluative. Against what standard are you judging this? A golden dataset, human review, downstream task performance, or a vibe? If the answer is the last one, you do not have an evaluation — you have an opinion with a graph.
4. Consequential. Who is affected when this is wrong, how would they find out, and what recourse do they have? A misclassified product recommendation and a misclassified loan application are the same technical error and completely different events.

Most commercial AI failures we have examined with mentees are not layer-one failures. Nobody deployed a classifier thinking it was a database. They are layer-three and layer-four failures: a system evaluated against the wrong standard, shipped to a population that had no way to contest its output.

Building an evaluation habit

Evaluation is a habit before it is a technique. The habit is: never look at a single output in isolation. One good answer tells you almost nothing; twenty answers across a deliberately varied set tells you a great deal.

We ask mentees to build what we call a personal eval set in week two of the track. It is deliberately small — thirty to fifty items — and deliberately adversarial. The rules are simple:

• Include at least five items where you know the correct answer with certainty, because you produced it yourself.
• Include at least five where the correct answer is genuinely contested, and write down what a defensible range looks like.
• Include at least five that are out of scope, where the right behaviour is to refuse or express uncertainty.
• Include at least five that are near-misses of your in-scope items — the cases that look answerable and are not.
• Re-run the whole set whenever anything changes: the model, the prompt, the retrieval corpus, the temperature, the wrapper library.

Thirty items sounds trivially small next to a benchmark of ten thousand. It is, and that is fine. The purpose is not to publish a leaderboard result; it is to make regression visible to one person working on one system. A mentee who runs a thirty-item set weekly will catch more real problems than a team that runs a ten-thousand-item benchmark once a quarter.

Drift, and the maintenance problem nobody budgets for

A model is not a feature you ship. It is a feature that starts decaying the moment it meets production, because the world it was trained on keeps moving and the training data does not.

Three kinds of drift matter in practice, and mentees should be able to name and detect all three:

• Data drift. The inputs change shape. A fraud model trained before a new payment method launches sees traffic it has never encountered.
• Concept drift. The inputs look the same but the correct answer changes. Spam does not look different because your filter improved — spammers change what they send because your filter improved.
• Feedback drift. The model's own outputs change the world it measures. A recommendation system that surfaces popular items makes those items more popular, then reports that it predicted popularity well.

Feedback drift is the one that catches experienced engineers, because the metrics look excellent right up until the system has quietly narrowed the world it operates in. The detection technique is straightforward and rarely done: hold out a small random slice of traffic from the model entirely, permanently, and compare.

A model with no maintenance budget is a liability with a launch date.

Marcus Bell, Director of Curriculum, BTMP

Knowing when not to use a model at all

The most valuable output of AI literacy is the ability to say, credibly and with reasons, that a model is the wrong tool here. This is career-defining in a way that prompt fluency never will be, because it is the judgement that senior people are actually paid for.

Our working heuristic, which mentees learn to apply and then to argue with:

Notice that only one of these rows is technical. The rest are questions about the problem, the consequences and the organisation. That is the point. AI literacy is mostly not about AI.

A twelve-week self-study path

If you cannot join a programme right now, here is the sequence we would use anyway. It assumes six to eight hours a week and no prior machine learning background — only comfort with a programming language and a willingness to be wrong in writing.

1. Weeks 1–2: probability you can defend. Conditional probability, base rates, Bayes. Work through twenty worked problems by hand before touching a library.
2. Weeks 3–4: build a classifier from scratch. Logistic regression, no framework. Implement the loss, the gradient and the training loop yourself.
3. Weeks 5–6: evaluation. Precision, recall, ROC, calibration curves, and why accuracy on an imbalanced set is a lie. Build your thirty-item personal set.
4. Weeks 7–8: a real dataset with real problems. Missing values, leakage, temporal splits. Find the leakage deliberately; it is always there the first time.
5. Weeks 9–10: modern architectures at working depth. Enough transformer understanding to reason about context, attention cost and why long inputs behave oddly.
6. Weeks 11–12: deploy something small. An endpoint, a monitor, a rollback plan and a written evaluation. The deployment is the assignment; the model can be mediocre.

At the end of twelve weeks you will not be a research scientist. You will be something the market is shorter of: a person who can be handed a model and asked, credibly, whether it should be trusted.

Frequently asked questions