Math as cultural institution.

The same week that Claude finished formalizing the proof of Fermat’s Last Theorem in Lean, a paper landed in my inbox titled, The Spherical Hadwiger Theorem. The Spherical Hadwiger Conjecture1, which has been open since about 1974, describes a niche-but-important piece of integral-geometric machinery. I’m not going to get into the details of the conjecture here; if you are interested you can see a discussion in my previous post where the theorem (then still a conjecture2) greatly simplifies the proof of a little lemma of mine from grad school.

But to the point: this new preprint by Wang & Wu of Hunan University apparently proves the conjecture using AI assistance. The final section contains the disclaimer:

During the preparation of this manuscript, OpenAI Codex was used to assist with developing proof details, identifying gaps and points requiring clarification, organizing and typesetting the manuscript, and editing the English. The authors reviewed and verified all AI-assisted mathematical content and suggested changes, made all final mathematical and editorial decisions, and take full responsibility for the manuscript.

This disclaimer leaves open the possibility that Codex did a substantial portion of the work that, until very recently, required a research-level mathematician: developing proof details3, finding and fixing gaps, and apparently writing the paper. Moreover, the work is very polished and readable (if you are a research mathematician in this field).

To be clear, I haven’t fully verified the proof; I worked through it with Claude Fable and it passes the sniff test, but fully digesting it will take a bit more energy than I have right now. None of this is a knock on Wang & Wu—this seems to be a great paper, and is worth digesting. They’ve even followed all the principles for AI use laid out in the Leiden Declaration.

A milestone, close to home

For me, the proof of the Spherical Hadwiger Theorem hits home. I tried to prove it in grad school, and made a half-hearted attempt again with AI assistance earlier this year. It’s not a headline-grabbing theorem. That didn’t save it.

I shouldn’t have been surprised. When GPT-4 launched, OpenAI released a report on the potential labor impact of LLMs. The exposure of the work of mathematicians to disruptions from AI was the highest of any category they modeled; the whitepaper estimated that 100% of a mathematician’s job was exposed to LLMs, across three distinct labor models. Higher than writers, translators, artists, and graphic designers. The only difference is that it took a bit longer for mathematicians to begin to feel the pain.

It’s tempting, if somewhat arrogant, to claim that this delay in LLM dominance in mathematics arose because research-level mathematics is among the most challenging human endeavors. I suspect the delay owes as much to research mathematics having less economic value—and less training data—than these other creative domains.

Off to the conservatory?

Consider classical music. Our society does not support classical musicians in the same way that we support ‘popular’ musicians. Classical music has been institutionalized, sent to the conservatory as a relic. A small segment of society has decided the ability to perform it is worth preserving, and devotes a sliver of capital to that end: training young people, and paying a few of the best players in the biggest cities to do it professionally.

Could this model work for mathematicians? It’s easy to imagine: in Euclid’s time, mathematics largely existed as an intellectual pursuit worthy of a few inclined people. A mathematical conservatory could help math flourish even when it is no longer hard to create new results, just hard to understand and communicate their import.

But pure mathematics is already in a conservatory, better known as the academy. Outside of the university, the jobs for pure mathematicians remain slim. The results are only understood by a select few. Does it even matter that the hard results are all going to be proved by computers soon?

The bottom line

It’s worth supporting people to continue the cultural endeavor that we currently call “research mathematics.” Mathematics, especially pure mathematics, has always been about communicating stories that help us understand reality more deeply. By simplifying and abstracting, we begin to see the hidden structure: parallel lines never cross, the sphere looks the same in every direction, the primes never end. But the current support systems for mathematicians, like so many other human creative fields, need to change drastically to handle the new realities of AI.

But the incentive structures that support this work, like those in so many other creative fields, are ill-suited to what AI is bringing. These tools make it harder to tell whether a complex argument is even correct, much less who deserves funding, tenure, and fame. The choice before us is how do we continue to make humans matter when their intellectual labors simply don’t compare to computers. When intelligence is limited only by silicon and electricity, will we be willing to continue to support a culture that has mathematicians? I sure hope so.

1. See Problem 3 in Glasauer’s thesis. ↩︎
2. In fact, it was only after I wrote the initial blog post that Prof. Rolf Schneider reminded me that the conjecture was still open at the time. ↩︎
3. Which proof details, I wonder? I’d love to see the full chat transcript. While the proof follows the rough approach in Klain and Rota for the Euclidean theorem, the details require overcoming several major obstructions, and I’m curious how many of them were identified and resolved by GPT 5.6. ↩︎

Do we need simplicity?

Recently, I gave a talk titled "Precise, consistent, and reliable code coverage". It's about a truly gnarly bug that took my company 9 months to debug. At the end, my friend Predrag asks:

How would you recommend that we think about building tools such that these epic debugging stories aren't as necessary?

and I answer him:

We need to prioritize simplicity. If you go back to my coverage pipeline, there are a lot of nodes in this diagram. [...] The tooling's complicated. We need to rethink how our computing works.

I'm not satisfied with that answer.

---

Unix pipelines are not simple

Consider two programs to calculate the frequency of words of a file. First, a small unix pipeline:

cat README.md \
  | tr --complement --squeeze-repeats '[:alpha:]' '\n' \
  | tr A-Z a-z \
  | sort \
  | uniq --count \
  | sort --reverse --numeric-sort

This says "read README.md, translate each word boundary into a newline, collapsing multiple newlines, convert uppercase to lowercase, count the number of occurrences of each word, then show them in frequency order".

I think this is what most people think of when they think of "simple": each program is small, they're designed to be joined together ad-hoc in this way, it's concise and somewhat easy to read.

Next, consider a Clojure program:

(->> (slurp "README.md")
     (re-seq #"[a-zA-Z]+")
     (map str/lower-case)
     frequencies
     (sort-by val >)
     ; for every (word, count) pair in the sequence, call an anonymous function that prints it.
     (run! (fn [[word count]] (println count word))))

This does the same thing, with a few more names and higher-order functions thrown in.

Now, let's say we want to make a small change: show the output in the original file order. In Clojure, this is fairly straightforward: store an ordered sequence of the words in word_seq, store a map from each word to its frequency in freq_map, iterate over the sequence, and look up each word in the map:

(let [word_seq (->> (slurp "README.md")
                 (re-seq #"[a-zA-Z]+")
                 (map str/lower-case))
      freq_map (frequencies word_seq)]
  (->> (distinct word_seq)
       ; for every distinct word, in original order, print its frequency (from our `freq` map) and the word itself
       (run! (fn [w] (println (freq_map w) w)))))

In Bash you need a bunch of temp files and ugly opaque regexes, sorts, and joins:

tr < README.md --complement --squeeze-repeats '[:alpha:]' '\n' \
  | grep . > words
sort words \
  | uniq --count \
  | sed --regexp-extended 's/^ *([0-9]+) (.*)/\2 \1/' \
  | sort > counts
nl --body-numbering=a words \
  | sort --key=2,2 --key=1,1n \
  | uniq --skip-fields=1 \
  | sort --key=2,2 > firstseen
join -1 2 -2 1 -o 1.1,2.2,1.2 firstseen counts \
  | sort --numeric-sort \
  | cut --delimiter=' ' --field=2,3

That's because our original program was small but not simple.

---

What is simplicity?

In Simple Made Easy (transcript), Rich Hickey defines "simple" from its root, "sim-plex": having only one braid. He contrasts this to "com-plex": braiding multiple things together. In this post I'll use "coupled" as a synonym for "complex" to avoid ambiguity.

And that gives us a language to talk about what's going on with our first Unix pipeline: it's small but it's coupled. Let's look at exactly what makes it that way.

tr < README.md --complement --squeeze-repeats '[:alpha:]' '\n' \
  | tr A-Z a-z \
  | sort \
  | uniq --count \
  | sort --reverse --numeric-sort

There are a bunch of little things here I could nitpick, but the main thing that's coupled (braided together) is the sort | uniq --count. If we look at uniq's man page, it says this:

Repeated lines in the input will not be detected if they are not adjacent, so it may be necessary to sort the files first.

There's no native Unix equivalent to frequencies, this sort | uniq -c is the closest we can get. Not only is it less performant (it has to collect the full input into memory before continuing), but it ties aggregation to ordering. This is exactly the thing that makes "separate ordering from aggregation" so hard; we end up having to do this weird dance with table-joins-through-text-files.

You might have heard the phrase "Write programs that do one thing and do it well" in reference to Unix systems. Maybe you heard it called the Unix Philosophy. I think "do one thing" is commonly understood to be about simplicity, but in practice it's actually about size. Unix tools are small but they are not simple.

Large is not the same as coupled

Now, let's consider the opposite end. Say you have Google Drive for Desktop running on your computer. This is a massively large program: it depends on platform-specific file watchers, "all of Google3", a streaming and syncing network client, and conflict resolution logic. But to the user it feels quite simple: Install the program, tell it which folder you want it to watch, tell it whether to keep the files locally or primarily on Google's infra. It does all the rest.

Decoupling

When I think about complex programs, I think about coupling. Programs are complex when different features are coupled to each other, even when they don't have to be.

Let's take one small example. In Rust, you can associate names to values with a map, or with a struct:

struct HttpResponse {
  status: u16,
}
let strukt = HttpResponse { status: 200 };

let mut map = HashMap::new();
map.insert("status", 200);

println!("map: {}", map.get("status").unwrap());
println!("struct: {}", strukt.status);

It's very clear from this that a struct gets you known present fields. For the map, we have to call unwrap(), because the type checker doesn't know what keys are in a map. For the struct it does, so we can just directly access the value.

What might not be clear about this is that a struct loses runtime information. If you want to iterate a map, that's easy: call for (key, val) in map { .... If you want to iterate a struct ... get fucked? write a proc-macro?

The reason for this is that in Rust, a struct couples type-checking to a fixed data representation. You can't get one without the other.

Contrast this to Clojure, where you can. In Clojure, structs are maps: rather than defining a type, you annotate which fields a map is allowed to have. If we wanted to translate our struct HttpResponse, we could write this:

; bind the name `http-response` to a list of keywords (interned strings).
; this is a normal list that is created and manipulated at runtime, it is not special in any way.
(def http-response [:map [:status :int]])
; bind the name `print-resp` to a function.
; `^{}` is a "metadata" map that will be associated with that name.
; metadata on bindings can be retrieved at runtime.
(defn ^{:malli/schema [:=> [:cat http-response] :nil]}
  print-resp [map]
  (println "status:" (:status map)))

Here, we've created a type annotation that's checked at runtime with the function (malli/instrument!). Notably, this is checked with a library (Malli), not by a compiler; and the annotation is inspectable. You can, for example, write a schema->md function that acts as your own little mini Rustdoc, without needing to integrate with compiler APIs. And all of this works without giving up type safety, reflection, or iteration over the values of the map.

This works because Clojure decouples data representations from type checking. Typed Racket does a similar trick, but using macros so that the type checking happens at compile time instead of runtime.

When is it useful to be small?

Being small makes sense when you as the maintainer don't have a lot of resources to dedicate to your program. Maybe you're Brian Kernighan and your program is running on a literal PDP-11. Maybe you're an open source maintainer with only a couple hours a month to dedicate to your project. Maybe you work in an environment where doing anything is a victory and you can only get support for a small subset of the features you actually want to build. All of these are good reasons to keep your program small.

But small is not the same as simple. The answer to "when should your program be simple?" is: always. There is very little advantage to introducing coupling to parts of your program; it makes it harder for you as a developer to maintain the program, and is less flexible for your users.

How do we make simple programs?

Ah, now this is the hard part. To write simple programs, you need to have a good mental model of your program. You also need to have good taste, which is something I don't yet know how to teach.

Sometimes, you also need to Suck It Up And Write The Hard Thing. CSS and SQL are highly decoupled (mostly): you write a declarative specification of what you want the program to do, and the browser engine or database runtime figure out how to do it. This is really really hard! SQLite alone has had centuries of person-years put into making it work reliably. Blink (Chrome's renderer) has probably had tens of thousands of person-years put into it. In some domains, that's what it takes to let you write programs that are decoupled.

It doesn't always make sense to spend that much time on a program. Crunchy technical work can be a mothlamp problem: it attracts a certain kind of person who loves dreaming about how code might, should, could work. Sometimes it's better to put down your tools and take a nap in the sun instead. But when it does work—

If we go back to the start of the post, the coverage pipeline I describe actually got larger after I fixed it, not smaller. But at the same time it got simpler, because there were fewer hidden dependencies between parts of the dataflow graph.

What next?

I hope this post encourages you to write programs that are simple, not small, and to look for tools that you use that are unnecessarily coupled.

In a future post, I hope to extend these ideas: how to develop your sense of taste; how programs can be vertically integrated while still being decoupled; and how to build large systems without making them complex.

TL;DR: Speculative decoding allows vLLM to verify multiple drafted tokens in a single target-model pass. In our experiments, its effect on output-token throughput varied across drafting methods and proposal lengths, and also depended on the model family, draft checkpoint, workload, and acceptance behavior.

---

Introduction

Large language models support a wide range of applications, but serving them at scale requires careful optimization. Standard autoregressive decoding is the baseline used by most LLM serving systems: the model generates one token, appends it to the sequence, and then uses the updated sequence to generate the next token. This process is simple and reliable, but the serving loop still advances one committed token at a time because output tokens must be produced in strict left-to-right order.

Speculative decoding [1] builds on this baseline through a draft-and-verify mechanism. A lightweight draft component proposes candidate future tokens, and the target model verifies those candidates before they are committed. When several draft tokens are accepted, the system can commit multiple output tokens from a single target-model verification step while preserving the target model's output behavior.

This post explores how speculative decoding works in vLLM and shares measurements from our test environment. We first review the autoregressive decoding baseline and the draft-and-verify process. We then examine five speculative-drafting approaches: native MTP, Gemma 4 MTP, EAGLE-3, DFlash, and DSpark. These methods differ in how the draft component receives information from the target model and whether candidate tokens are generated sequentially, autoregressively, in parallel, or through a hybrid approach. Finally, we show how to enable the methods tested in our environment, report measurements from our experiments on AMD Instinct™ MI300X and MI355X GPUs using the ROCm™ open software platform, and discuss practical tuning and observability considerations.

---

The autoregressive decoding baseline

In standard autoregressive decoding, each decode step produces and commits one new token. For example, generating four output tokens requires four sequential decode steps:

Step 1: context → model → T1

Step 2: context + T1 → model → T2

Step 3: context + T1 T2 → model → T3

Step 4: context + T1 T2 T3 → model → T4

After each step, the generated token is appended to the sequence and becomes part of the input for the next step. This makes the decoding loop straightforward, but it also requires one model decode step for every output token. During long generations, this token-by-token loop can dominate latency and limit serving throughput.

The key question behind speculative decoding is therefore:

Can we preserve the output behavior of the original model while reducing how often generation advances by only one token at a time?

Speculative decoding addresses this by separating proposal from verification. A draft component first proposes several candidate future tokens. The original model, acting as the target model, then verifies those candidates before they are committed.

---

Core idea of speculative decoding

Speculative decoding does not replace the original model. Instead, it keeps the original model as the target model, which remains responsible for the final output, and adds a faster proposal stage in front of it.

The process has two parts:

• Draft: propose several candidate future tokens.
• Verify: use the target model to check those candidates.

During each speculative decoding round, as illustrated in Figure 1, a lightweight draft component proposes one or more future tokens. These tokens are only candidates and are not committed immediately. The target model then evaluates the candidate token sequence in one verification pass.

Verification proceeds from left to right. Each draft token is checked using the target model's result at the corresponding position. Accepted tokens are committed to the output sequence. When a draft token is rejected, later candidates from the same proposal are no longer accepted.

If a draft token is rejected, the target model provides the next token. The remaining draft tokens are discarded, and generation continues from the updated sequence.

Conceptually, standard autoregressive decoding advances one token at a time. Speculative decoding instead allows several candidate positions to be evaluated together. This can reduce the number of target-model decoding rounds when multiple candidates are accepted. When the draft component produces tokens that the target model accepts, several output tokens can be committed from one target-model verification step. When a proposal is rejected, the target-side result determines how generation continues.

A simple accept/reject example

Figure 2 gives an example of one speculative decoding round. Green boxes are draft tokens that survive verification, the red box marks the first rejected draft token, and the gray box is a later draft token that is discarded. The blue token in the output comes from the target model, not from the draft proposal.

Suppose the current prompt is:

The weather today is

The draft component proposes several future tokens:

sunny and warm outside

The target model verifies the draft tokens from left to right:

The first two draft tokens, sunny and and, are accepted. At the third position, the draft proposes warm, but the target model selects clear. The remaining candidate, outside, is discarded because it follows the first rejected position.

The next decoding round therefore continues from:

The weather today is sunny and clear

---

How the drafting methods work

Although all speculative decoding methods follow the same overall draft-and-verify process, they differ in how the draft component is designed and how it works with the target model.

The main differences are:

• The type of information received from the target model.
• How this information is incorporated into the drafting process.
• Whether candidate tokens are generated sequentially or in parallel.

Based on these differences, the drafting methods discussed in this post can be grouped into three broad categories: native MTP modules, separate MTP drafters, and dedicated target-conditioned draft networks.

• Native MTP modules: built directly into the target-model architecture; use a model-native auxiliary prediction path; generate candidate tokens sequentially.
• Separate MTP drafters: use a separate checkpoint paired with a specific target model; use target-model activations and shared KV-cache information during inference; generate candidate tokens sequentially.
• Dedicated target-conditioned draft networks: use separate speculator models trained for a specific target model, including EAGLE-3, DFlash, and DSpark. EAGLE-3 drafts autoregressively from target-model hidden states, DFlash drafts parallel blocks from target-model hidden states, and DSpark adds lightweight causal correction and confidence-based prefix selection.

These categories describe the draft component architecture, not the target-model family. A target model may support native MTP while also having separately trained EAGLE-3, DFlash, or DSpark draft models.

The draft component does not operate entirely on its own. Depending on the method, the draft component may receive:

• A hidden representation from the target model.
• Hidden states from several selected target layers.
• The target model's KV cache.
• Features produced by combining multiple target-model representations.

The following sections explain how each method uses this information and how it generates candidate tokens.

Native MTP

Multi-Token Prediction, or MTP, refers to a family of model-native mechanisms for predicting tokens beyond the immediate next token. In vLLM, native MTP is available when the target model includes a compatible auxiliary prediction component [2]. The exact MTP architecture varies across model families, but each implementation provides an auxiliary path for proposing future tokens.

At the first speculative step, the MTP component combines a hidden representation from the target model with information from the current token to predict the first draft token. At subsequent steps, the newly drafted token and the hidden state produced by the previous MTP step are used to predict the next candidate. After the configured number of candidates has been proposed, the target model evaluates them together in one verification pass.

Many native MTP implementations follow a similar pattern. A hidden representation from the target model or from the previous MTP prediction is combined with the embedding of a shifted input token or the latest drafted token.

The two inputs serve different purposes: (1) the hidden representation carries information about the preceding sequence; and (2) the token embedding identifies the latest token from which drafting continues. In common implementations, they are combined along the hidden dimension and transformed before entering the auxiliary prediction layer.

The number of physical MTP layers and the configured speculative length are separate concepts. When num_speculative_tokens exceeds the prediction depth directly provided by the checkpoint, vLLM can reuse the MTP path through additional forward passes. A larger value therefore proposes more candidates before verification, but also introduces more sequential drafting work.

Native MTP is closely tied to the target-model architecture. In many implementations, parts of the MTP path share components with the target model, which can keep the additional memory overhead relatively modest. However, generating multiple speculative tokens still requires sequential drafting before verification.

Gemma 4 MTP

Gemma 4 uses a separately packaged MTP draft component paired with a specific target model [3]. Although the draft component has its own checkpoint, it remains closely connected to the target model during inference.

The draft component uses activations produced by the target model and shares the target model's KV cache. This allows it to reuse contextual information that the target has already computed instead of processing the accepted prefix independently.

As with native MTP, the number of layers in the draft component is separate from the configured speculative length. When several candidate tokens are requested, the draft component generates them sequentially.

EAGLE-3

EAGLE-3 uses a dedicated draft network trained for a specific target model. The draft component has its own execution path, but it remains closely conditioned on information produced by the target model [4].

During the target-model forward pass, EAGLE-3 records hidden states from three stages of the target Transformer: near the beginning, around the middle, and near the end. These are contextual representations of the same accepted sequence at different stages of target-model processing.

The three hidden states are concatenated and projected into a single fused target feature. This fused representation is then combined with the embedding of the sampled token before entering the EAGLE-3 draft decoder.

The two inputs serve different purposes:

• The fused target feature summarizes the accepted sequence using information from several stages of the target-model forward pass.
• The sampled-token embedding identifies the token from which drafting continues.

EAGLE-3 generates draft tokens autoregressively. For the first draft token, it uses the fused target feature computed from the accepted sequence together with the sampled-token embedding. After a draft token is produced, its embedding is fed into the next drafting stage.

Because the target model has not yet processed the later speculative positions, target-model hidden states for those positions are not available. EAGLE-3 therefore uses the previous draft-component output when continuing the draft sequence.

This sequential feedback gives later draft tokens direct dependence on earlier drafted tokens along the proposed sequence. However, generating more speculative tokens also requires more sequential drafting work before verification.

DFlash

DFlash uses a dedicated draft network trained for a specific target model. Unlike MTP and EAGLE-3, which generate candidate tokens sequentially, DFlash predicts a whole block of future positions in parallel [5].

DFlash begins each draft block with an anchor token. The anchor is a known token produced or confirmed by the target model, so DFlash does not need to predict it. Instead, it provides a known starting point for the masked positions that follow. In later decoding rounds, this is typically the additional target token returned by the previous verification pass.

The anchor occupies the first position of the block, while the remaining positions are masked and predicted in parallel.

Like EAGLE-3, DFlash first combines hidden states from several target-model layers into a fused representation.

The main difference is how this fused representation is used. EAGLE-3 combines it with the sampled-token embedding at the input of its autoregressive draft network. DFlash instead converts the fused target context into additional Key and Value representations that are available in every layer of the draft network.

Queries from the masked draft positions can therefore attend to both:

• Key and Value representations derived from the target model.
• Key and Value representations produced from the draft block itself.

The target-model context therefore remains available throughout the draft network, rather than being supplied only once at its input.

After the draft block has been generated, the target model evaluates all proposed tokens in one verification pass. The acceptance decision is then applied from left to right: accepted tokens are committed until the first rejection, and the remaining candidates are discarded.

A defining characteristic of DFlash is that all masked positions are predicted together in one draft-network forward pass.

Because all masked positions are predicted together, a later position is not conditioned on the sampled output of an earlier position during the same pass. This removes the token-by-token feedback used by autoregressive drafting. The effectiveness of later positions therefore depends on the trained checkpoint and workload, particularly when longer draft blocks are used.

DSpark

DSpark extends parallel drafting with two additional mechanisms:

• A lightweight sequential head that introduces dependence between tokens within the draft block.
• Confidence-based selection of the prefix submitted for target-model verification.

DSpark uses a modified DFlash model as its parallel backbone [6]. The backbone performs the main draft computation for all positions in one forward pass, producing a hidden state and a set of base logits for each draft position. It therefore inherits the target-context conditioning described in the DFlash section.

A fully parallel draft component predicts every position without first seeing the tokens selected at earlier positions in the same block. When several continuations are plausible, this can produce inconsistent combinations. For example, both "of course" and "no problem" may be reasonable continuations, but independent position-wise predictions could produce "of problem."

DSpark addresses this behavior by applying a lightweight sequential head after the parallel backbone. The backbone still computes the base logits for every position together. The sequential head then selects tokens from left to right, adjusting each position using information from the previously selected draft tokens.

DSpark applies a lightweight Markov head that introduces dependence between the selected draft tokens. For each position, the Markov head uses the immediately preceding selected token to produce a small bias. This bias adjusts the base logits produced by the parallel backbone.

The main draft network processes all candidate positions together in one forward pass. After that, only the lightweight Markov head runs from left to right to adjust each position using the previously selected draft token.

This allows later draft tokens to depend on tokens already selected within the same block without running the full draft network again for every position.

The DSpark design also includes a confidence head that can select a shorter draft prefix for target-model verification. This feature was not active in the vLLM path used for our experiments, so the benchmark results reflect only the parallel draft network and lightweight Markov correction.

The target model evaluates the proposed sequence in one verification pass, and draft tokens are committed from left to right until the first rejection.

Summary of the drafting methods

Figure 3 gives a visual side-by-side view of the five drafting methods: what the draft component looks like, which target-model information it uses, and whether candidate tokens are generated sequentially or in parallel. The table below the figure restates the same comparison in a compact form. In all five methods, the target model still evaluates the proposed sequence in one verification pass, and the acceptance decision is applied from left to right until the first rejected draft token.

Method	Draft component	Target-model information used	How draft tokens are generated
Native MTP	Model-native auxiliary MTP path	A target-model or previous MTP hidden representation combined with current draft-token information	Sequentially through repeated use of the MTP path
Gemma 4 MTP	Separate MTP draft component paired with the target model	Target-model activations and the shared target KV cache	Sequentially through the paired MTP component
EAGLE-3	Dedicated autoregressive draft network	Hidden states captured near the beginning, around the middle, and near the end of the target-model forward pass, fused into one representation	Sequentially, with each drafted token influencing the next
DFlash	Dedicated parallel draft network	Fused target-model hidden states provided as additional Key and Value information in every draft layer	All candidate positions are predicted together in one parallel forward pass
DSpark	DFlash-style parallel draft network with a lightweight Markov head	The same target-conditioned information used by the parallel draft network	One parallel forward pass followed by lightweight sequential adjustment of token selection

---

How to enable speculative decoding in vLLM

In vLLM, speculative decoding is configured through --speculative-config. The main differences are the method name, whether a separate draft checkpoint is required, and the number of candidate tokens requested. Current vLLM supports mtp, eagle3, dflash, and dspark as method values.

Method	Separate draft checkpoint	Typical configuration
Native MTP	No	"method": "mtp" "num_speculative_tokens": <N>
Gemma 4 MTP	Yes	"method": "mtp" "model": "<matching-assistant>" "num_speculative_tokens": <N>
EAGLE-3	Yes	"method": "eagle3" "model": "<matching-speculator>" "num_speculative_tokens": <N>
DFlash	Yes	"method": "dflash" "model": "<matching-speculator>" "num_speculative_tokens": <N>
DSpark	Yes	"method": "dspark" "model": "<matching-speculator>" "num_speculative_tokens": <N>

For native MTP, the draft component is included with the target model, so the model field is omitted:

vllm serve <target-model> \
  --speculative-config '{
    "method": "mtp",
    "num_speculative_tokens": <N>
  }'

For Gemma 4 MTP, EAGLE-3, DFlash, and DSpark, the model field normally points to a checkpoint trained for the target model:

vllm serve <target-model> \
  --speculative-config '{
    "method": "<method>",
    "model": "<matching-draft-checkpoint>",
    "num_speculative_tokens": <N>
  }'

Gemma 4 assistant checkpoints use the MTP path even though they are supplied through the model field. vLLM connects the assistant component to the target model and allows it to share the target KV cache.

Before enabling a method, check that:

• The installed vLLM version supports the method and model architecture.
• The draft checkpoint is compatible with the target model and method.
• num_speculative_tokens is compatible with the checkpoint.
• The model card supports the intended hardware and inference backend.

Memory considerations

Native MTP does not load a separate draft checkpoint and may share components such as the embedding table or output head with the target model. Gemma 4 MTP, EAGLE-3, DFlash, and DSpark load additional draft weights, so sufficient GPU memory headroom should be reserved. The actual overhead depends on the draft-component size, numerical precision, tensor-parallel configuration, and runtime buffers.

---

Where to find the pretrained draft models

Several organizations now publish pretrained draft models on Hugging Face. Google provides MTP assistants for Gemma 4, while Z-Lab maintains a collection of DFlash checkpoints. Red Hat AI offers draft models across EAGLE-3, DFlash, and DSpark, and DeepSeek's DeepSpec collection provides matched checkpoints for all three methods. LightSeek focuses on EAGLE-based draft models for Kimi, while Inferact publishes draft models for MiniMax and Kimi.

Draft-model publisher	Methods	Representative models and targets
Google	Gemma 4 MTP	Assistant checkpoints for Gemma 4 E2B, E4B, 12B, 26B-A4B, and 31B target models. [7]
LightSeek Foundation	EAGLE-3 and EAGLE-3.1	EAGLE-based draft models for Kimi-K2.5, Kimi-K2.6, and Kimi-K2.7-Coder, including standard and MLA variants. [8]
Red Hat AI	EAGLE-3, DFlash, and DSpark	A collection covering target families such as Llama, Qwen, Gemma, GPT-OSS, GLM, Nemotron, and Mistral. Common suffixes include -speculator.eagle3, -speculator.dflash, and -speculator.dspark. [9]
Z-Lab	DFlash	DFlash checkpoints for targets including Qwen3, Qwen3.5, Qwen3.6, Gemma 4, Kimi, MiniMax, GPT-OSS, and Llama. Checkpoint names generally follow the <target>-DFlash pattern. [10]
DeepSeek AI	EAGLE-3, DFlash, and DSpark	The DeepSpec collection provides versions of all three methods for Qwen3-4B, Qwen3-8B, and Qwen3-14B, as well as Gemma 4 12B. Examples include eagle3_qwen3_8b_ttt7, dflash_qwen3_8b_block7, and dspark_qwen3_8b_block7. [11]
Inferact	EAGLE-3 and DSpark	Draft models including Inferact/MiniMax-M3-EAGLE3, its GQA variants, and Inferact/Kimi-K3-DSpark. [12]

---

Experimental setup and measurements

After enabling speculative decoding, the practical question is whether the additional drafting work improves end-to-end serving performance. Candidate tokens do not need to be correct at every position because the target model evaluates them before they are committed. Performance therefore depends on how many proposed tokens are accepted and whether the saved target-model decoding work outweighs the cost of drafting and verification.

We evaluate model quality and serving performance using task-grounded benchmarks rather than random token sequences. Acceptance behavior depends on the structure and predictability of actual model outputs, so task-based prompts provide a more representative view of practical performance.

The main performance indicators are:

• Output-token throughput and speedup over the non-speculative baseline.
• Mean accepted length and draft-token acceptance rates, where available.
• Model quality relative to the non-speculative baseline.

Models and experiment coverage

The experiments cover five speculative-drafting approaches across several target-model families. A check mark indicates that benchmark results are available for that target-method combination; a dash indicates that the combination was not included in the current experiments.

Target model	Native MTP	Gemma 4 MTP	EAGLE-3	DFlash	DSpark
google/gemma-4-26B-A4B-it	-	✓ Google	✓ Red Hat AI	✓ Z-Lab	-
google/gemma-4-31B-it	-	✓ Google	✓ Red Hat AI	✓ Z-Lab	✓ Red Hat AI
Qwen/Qwen3-8B	-	-	✓ Red Hat AI	✓ Z-Lab	✓ DeepSeek
Qwen/Qwen3.5-27B	✓ Built-in	-	-	✓ Z-Lab	-
Qwen/Qwen3.5-122B-A10B	✓ Built-in	-	-	✓ Z-Lab	-
Qwen/Qwen3.6-27B	✓ Built-in	-	-	✓ Z-Lab	-
Qwen/Qwen3.6-35B-A3B	✓ Built-in	-	-	✓ Z-Lab	-
moonshotai/Kimi-K2.5	-	-	✓ LightSeek	✓ Z-Lab	-
MiniMaxAI/MiniMax-M3-MXFP8	-	-	✓ Inferact	-	-

The table summarizes the target-method combinations included in the experiments and shows how speculative decoding behaves across different models, workloads, and proposal lengths. Each result should be interpreted within its test configuration, since model architecture, active parameter count, draft-component size, workload, and serving conditions can all affect performance.

Throughput measurements

For throughput, we measure generated tokens per second against a standard autoregressive baseline and sweep the number of speculative tokens to study how speculation depth affects end-to-end serving throughput.

Main observations

The measurements varied by target model, drafting method, workload, and proposal length.

For gemma-4-26B-A4B-it, the largest measured throughput ratios within the tested sweep were 2.74× and 2.62× for Gemma 4 MTP on GSM8K and MBPP, respectively, and 2.87× and 2.79× for DFlash on MATH500 and HumanEval. The EAGLE-3 measurements ranged from 2.11× to 2.27× across the four datasets.

For gemma-4-31B-it, Gemma 4 MTP measurements reached 2.00× on GSM8K and 1.99× on MBPP, while DFlash reached 2.34× on MATH500 and 2.05× on HumanEval. The EAGLE-3 and DSpark measurements were also above baseline across the four evaluated datasets. The proposal length associated with the largest measured throughput varied by workload.

For Qwen3-8B, the DSpark measurements ranged from 1.15× on MATH500 to 1.63× on GSM8K. DFlash measurements ranged from 1.08× to 1.27×. EAGLE-3 was above baseline on GSM8K, HumanEval, and MBPP, while its largest measured MATH500 value remained below the baseline.

For Qwen3.5-27B, Qwen3.5-122B-A10B, and Qwen3.6-27B, the maximum measured native-MTP values within the tested sweeps were higher than the corresponding maximum DFlash values. The largest ratio in this group was 2.20× for Qwen3.5-122B-A10B on MATH500. The native-MTP proposal length associated with the largest measured throughput ranged from N=4 to N=7, depending on the model and dataset.

For Qwen3.6-35B-A3B, the DFlash measurements ranged from 1.77× to 2.06×, with the largest value occurring at N=7 for each of the four datasets. Native-MTP measurements ranged from 1.28× to 1.49×, with the largest values occurring at N=6. The difference from the Qwen3.6-27B measurements shows that results can vary between models in the same family.

For MiniMax-M3-MXFP8, the EAGLE-3 measurements reached 2.09× on HumanEval at N=4. For Kimi-K2.5, EAGLE-3 measurements reached up to 2.33× and DFlash measurements reached up to 2.68×. Within the tested sweeps, the largest EAGLE-3 values generally occurred at N=4, while the largest DFlash values occurred at N=7.

Across the experiments, the proposal length associated with the largest measured throughput was not constant. For the sequential methods, throughput often increased over the first few values of N before reaching a plateau. For DFlash and DSpark, N=7 was frequently among the higher-throughput settings, while larger values did not consistently increase throughput.

These observations reflect the hardware, software, target model, draft checkpoint, workload, and sweep settings used in this study.

---

Tuning considerations

Speculative decoding should be treated as a runtime optimization rather than a fixed setting that works equally well for every workload. The value of num_speculative_tokens associated with the highest throughput depends on how many proposed tokens are accepted and whether the avoided target-model decode work outweighs the cost of drafting and verification.

Observability is therefore important. A model-card recommendation or example configuration provides a useful starting point, but the final setting should be selected using representative workloads and end-to-end measurements. Useful signals include throughput, mean accepted length, overall acceptance rate, and per-position acceptance rate.

A larger proposal window gives the system more opportunities to commit several tokens in one verification pass. However, acceptance may decrease at later draft positions. When this happens, the additional candidates contribute little while still adding drafting and verification work, causing throughput to flatten or regress.

Start from a supported configuration

For native MTP, N=1 is a conservative starting point because it introduces the least additional sequential drafting work:

{"method": "mtp", "num_speculative_tokens": 1}

After confirming correctness and stability, sweep larger values such as 2, 3, 4, 5, 6, and 7.

In our measurements, the native-MTP setting associated with the largest measured throughput varied by target model and workload. For Qwen3.5-27B, the largest measured throughput occurred at N=5 for GSM8K and MATH500, N=4 for HumanEval and MBPP, and N=3 for MT-Bench. For Qwen3.5-122B-A10B, the largest measured throughput across the four listed reasoning and code datasets occurred at N=7.

The Qwen3.6 measurements also show that this setting can change between models in the same family. For Qwen3.6-27B, the largest measured values occurred at N=4 or N=5, while throughput for the tested Qwen3.6-35B-A3B configurations increased through N=6.

For Gemma 4 MTP and EAGLE-3, increasing N also adds sequential drafting work. A short sweep is therefore useful even when the checkpoint provides a recommended configuration. In our Gemma 4 and EAGLE-3 experiments, measured throughput generally increased over the first few values of N before reaching a plateau.

For DFlash, begin with the proposal lengths recommended or supported by the draft checkpoint. Many DFlash checkpoints are trained with a fixed block size. For example, when:

block_size = 16

the maximum proposal length is normally:

num_speculative_tokens = 15

because the first position is the confirmed anchor token and the remaining 15 positions are draft candidates.

This is the maximum supported proposal length, not necessarily the highest-throughput setting. In practice, it is useful to test smaller values such as:

N = 3, 7, 11, 15

Across our DFlash experiments, N=7 was frequently among the higher-throughput settings. For some workloads, the largest measured throughput occurred at N=11.

For DSpark, num_speculative_tokens sets the number of candidate tokens generated in each speculative round. In our vLLM experiments, the full configured proposal was submitted for target-model verification, so values such as N=3 and N=7 should be compared using end-to-end throughput.

Monitor acceptance behavior

Relevant signals to monitor include:

Signal	What it shows
Throughput	How end-to-end serving performance changes relative to the non-speculative baseline
Mean accepted length	How many draft tokens are committed per speculative round on average
Overall acceptance rate	What proportion of proposed draft tokens are accepted
Per-position acceptance rate	Whether later positions in the proposal remain useful

Per-position acceptance is particularly helpful when tuning proposal length. If the first few positions are accepted frequently but later positions contribute very little, reducing num_speculative_tokens may improve throughput by avoiding unnecessary draft work.

Acceptance metrics should be interpreted together with throughput. A method may show higher throughput relative to baseline even with a lower acceptance rate when draft generation is inexpensive. Conversely, a high acceptance rate does not necessarily correspond to higher throughput when the draft component adds additional overhead.

Match the sweep to the workload

Different workloads can produce different acceptance patterns.

In our GSM8K and MATH500 measurements, medium or deeper proposal lengths were often associated with higher measured throughput within the tested sweeps. For native MTP on Qwen3.5-122B-A10B, measured throughput increased through N=7. For DFlash, higher measured values frequently occurred at N=7 or N=11.

For HumanEval and MBPP, moderate proposal lengths were often among the higher-throughput settings. Code contains predictable local structure, but formatting, identifiers, and implementation choices can cause an otherwise plausible continuation to diverge.

Example tuning workflow

1. Begin with a configuration supported or recommended for the checkpoint.
2. Benchmark using representative prompts and generation settings.
3. Record throughput, mean accepted length, and acceptance rates.
4. Sweep several smaller and larger proposal lengths.
5. Select a setting based on the metric most relevant to the intended workload. In these experiments, end-to-end serving throughput was the primary selection metric.

The selected configuration does not necessarily have the longest proposal, the highest acceptance rate, or the largest mean accepted length. Selection should consider the trade-off among drafting cost, verification cost, accepted tokens, and the metric most relevant to the intended workload.

---

Training a speculator for a new target model

This guide does not cover speculator training in depth. The following workflow summarizes practical considerations from the referenced vLLM Speculators and DeepSpec resources [13], [14], and [15].

A typical workflow is:

1. Prepare representative prompts.
2. Generate responses with the target model.
3. Choose a hidden-state generation mode.
4. Collect the required target-model hidden states.
5. Train the speculator.
6. Test acceptance and serving throughput.

Prepare representative prompts

Start with prompts that reflect the expected workload, such as chat, mathematics, code generation, tool use, or multilingual tasks. Keep a separate set of prompts for evaluation.

The responses used for training should be generated by the exact target model that the speculator will support. The tokenizer, chat template, thinking mode, and generation configuration should also match the intended deployment. The vLLM documentation emphasizes that applying the target model's tokenizer or chat template to existing responses does not make the data target-specific; the responses themselves must come from the target model.

Choose how to obtain hidden states

The speculator receives internal hidden states from the target model during training. The vLLM Speculators workflow supports three ways to provide them:

Training mode	How it works	Main consideration
Online	Hidden states are generated by a running vLLM server when needed and discarded afterward	Avoids a large disk cache but requires resources for target inference and training at the same time
Offline	Hidden states are generated and stored before training begins	Frees all GPUs for training afterward but requires substantial storage
Hybrid	Hidden states are generated and cached during the first epoch, then reused	Pays the generation cost once without requiring a separate preprocessing stage

The selected mode changes where the hidden states come from; the remaining training workflow is largely the same.

Collect target-model information

A vLLM server can run the target model and expose hidden states from the layers required by the selected drafting method. When custom target layers are chosen, the same layer selections must also be used in the speculator-training configuration.

The information collected depends on the method:

• EAGLE-3 uses hidden states from selected target-model layers for autoregressive drafting. [4]
• DFlash uses target-model features to train a network that predicts a block of future positions in parallel. [16]
• DSpark adds lightweight sequential and confidence heads to a DFlash-style draft network. [6]
• MTP training fine-tunes the target model's own MTP component and therefore requires a target model that already contains compatible MTP layers. [13]

Train and test the speculator

The speculator configuration must match the target model's hidden size, vocabulary, tokenizer, and selected target layers. Method-specific settings such as draft-network depth, block size, sequence length, and learning rate must also be selected.

After training, inspect the checkpoint and serve it together with the target model in vLLM. Training loss alone is not enough to judge the result; the important measurements are accepted length, acceptance rate, draft latency, GPU memory use, and end-to-end serving throughput. The vLLM Speculators tutorial covers the complete path from data preparation and hidden-state extraction to checkpoint testing and serving.

When acceptance is weak for a particular workload, the prompt mixture or training configuration can be adjusted and the process repeated. The main principle is to use the same target model, generation mode, and representative workload that the speculator is expected to support.

---

Summary

This blog explored speculative decoding in vLLM as a draft-and-verify approach for LLM serving. A draft component proposes candidate future tokens, and the target model evaluates the proposal before any tokens are committed.

We examined five drafting approaches: native MTP, Gemma 4 MTP, EAGLE-3, DFlash, and DSpark. They differ mainly in how they use information from the target model and whether candidate tokens are generated sequentially, in parallel, or through a combination of parallel prediction and lightweight sequential correction.

The experiments covered selected Gemma, Qwen, MiniMax, and Kimi models on AMD Instinct™ MI300X and MI355X GPUs using the ROCm™ software platform. Measured throughput varied across target models, draft checkpoints, workloads, proposal lengths, and serving configurations.

Across the tested configurations, some settings produced smaller changes or throughput below the non-speculative baseline, while several model-workload combinations produced throughput ratios above 2×. Examples at the upper end of the observed range included 2.87× for DFlash on gemma-4-26B-A4B-it, 2.83× for Gemma 4 MTP on the same target, and 2.68× for DFlash on Kimi-K2.5.

Proposal length was also an important experimental variable. Increasing num_speculative_tokens sometimes increased throughput over the first few settings, while larger values could lead to a plateau or lower throughput. Checkpoint recommendations can provide starting points, but representative workload measurements and acceptance metrics are needed when selecting a deployment configuration.

Future work

Future benchmarking could include non-learned approaches such as n-gram speculation and suffix decoding, particularly for workloads with repeated token patterns such as code editing and agentic loops.

Broader evaluation across concurrency levels, prompt and output lengths, batch sizes, and sampling settings would also help show how speculative decoding behaves under different serving conditions.

Another useful direction is to study how speculator training data affects acceptance across code, mathematics, chat, multilingual prompts, tool use, and structured output. This could provide clearer guidance when choosing or training a draft checkpoint for a specific workload.

Finally, deeper profiling of draft generation, target verification, KV-cache behavior, graph execution, and scheduling would help explain the performance differences observed across target models and workloads.

---

References

1. vLLM documentation, "Speculative Decoding" https://docs.vllm.ai/en/latest/features/speculative_decoding/
2. vLLM documentation, "MTP Speculative Decoding" https://docs.vllm.ai/en/latest/features/speculative_decoding/mtp/
3. Google Developers Blog, "Multi-token prediction in Gemma 4" https://blog.google/innovation-and-ai/technology/developers-tools/multi-token-prediction-gemma-4/
4. EAGLE-3 paper, "Scaling up Inference Acceleration of Large Language Models via Training-Time Test" https://arxiv.org/pdf/2503.01840
5. Z-Lab, "DFlash" GitHub repository https://github.com/z-lab/dflash
6. DSpark paper, arXiv preprint https://arxiv.org/pdf/2607.05147
7. Google, "Gemma 4" Hugging Face collection https://huggingface.co/collections/google/gemma-4
8. LightSeek Foundation model collection on Hugging Face https://huggingface.co/lightseekorg/models
9. Red Hat AI, "Speculator Models" Hugging Face collection https://huggingface.co/collections/RedHatAI/speculator-models
10. Z-Lab, "DFlash" Hugging Face collection https://huggingface.co/collections/z-lab/dflash
11. DeepSeek-AI, "DeepSpec" Hugging Face collection https://huggingface.co/collections/deepseek-ai/deepspec
12. Inferact model collection on Hugging Face https://huggingface.co/Inferact/models
13. vLLM Speculators documentation, "Training a Speculator" https://docs.vllm.ai/projects/speculators/en/latest/user_guide/tutorials/train/
14. vLLM Project, "Speculators" GitHub repository https://github.com/vllm-project/speculators
15. DeepSeek-AI, "DeepSpec" GitHub repository https://github.com/deepseek-ai/DeepSpec
16. DFlash paper, arXiv preprint https://arxiv.org/pdf/2602.06036

– Some cows in the fields.

Brain Fog

My brain is not the same as it was a few years ago. When I started working as a Software Engineer, everything was novel and mentally demanding. I notably remember how during my first professional months I’d come back home and just go straight to bed due to the mental exhaustion of learning so much stuff and trying to solve software engineering problems that were mostly alien to me at that point in my career.

But now, I’m on the opposite end. Nothing is that novel anymore, and solving day to day problems, more than requiring mental exhaustion, mostly require spending some time at them. Add to that the usage of AI. While I know I’m way more productive than during those first months, my thoughts are getting slower, less profound, lazier.

Plus, dopamine is available at just a tap on my phone. There’s a ton of content in shorts that interest me and require little to no investment on my end, both time and effort-wise. Add to that videogames, comics/manga, TV shows, etc, and I literally just can’t get bored anymore.

Reading

One key metric for me is the amount of books I (don’t) read. Before I entered university, I used to devour books before going to sleep. Then, during university I was reading 1200+ pages per week for my History undergrad courses. When I started programming as a hobby I added online documentation, and Computer Science/Software Engineering books to the mix. But then, when I started my first job as a Software Engineer I was way too tired for reading. I spent a hefty amount of time reading during my dayjob, but it was mostly documentation.

About 8 years have passed, and I’m afraid I haven’t read that much since then. I still love reading, but as I mentioned, there’s an overabundance of distractions that I’ve fallen into that aren’t that compatible with books, specially doomscrolling before sleeping.

De-Brainrotting during my vacations

Over the last year I’ve tried to introduce new habits to avoid brainrot and keep my brain sharper: handwriting (both for pleasure and planning), making sideprojects, and writing in this blog. And while I’ve found some improvement, I still find my thinking and learning skills far away from what they were 8 years ago.

So for these vacations I wanted something different for my brain. We spent them visiting family in the country-side and celebrating my wife’s birthday on one weekend and my mother-in-law’s birthday the next one. This was the perfect opportunity to slow down and double down on new habits I wanted to grow.

These were slow vacations. I spent a lot of time in nature with my dog (I even saw a bunch of horses fighting vultures that were trying to eat the placenta of a newly born foal), socializing with family while drinking mate, and playing board games, football, and roblox (yes, roblox) with my nephews.

Books

For this vacation I took two books with me: Mary Beard’s SPQR to reminisce on my time during university, and Francisco Claro’s “De Newton a Einstein y algo más” to scratch my itch about knowing a bit more about physics. Both heavily recommended.

I had so much fun reading them that I finished them during the first of my two weeks there, so I had to buy new books: a collection of F. Scott Fitzgerald short stories, and Atomic Habits. Those were much faster to read for me so I finished them early on the second week. Surprisingly for me during that week I also found a new hobby that I wasn’t expecting to be so keen on.

Maths and Physics as a hobby

I had so much fun reading the mathematical proofs and explanations in “De Newton a Einstein y algo más” that I was left wanting more (or “algo más”). So I started going through Stewart’s Calculus: Early Trascendentals. I noticed my algebra and trigonometry was quite lacking, so I went through the books appendix on trigonometry, and their site’s resources on algebra.

Going through a maths textbook on my laptop or phone was a subpar experience, so I researched other web-native alternatives and landed on Paul’s Notes and Active Calculus. And to scratch my itch on physics I’m going through a textbook called University Physics. So far I haven’t found a web-native alternative but so far so good.

I spent the rest of my vacations working on my pre-calculus knowledge and doing maths/physics exercises.

For me this feels when I started programming as a hobby. Just learning for the sake of learning and having fun in the process. I have to admit that the remote possibility of being able to find a job in software engineering was also a motivation for learning programming, so this time it’s also different. I’m not sure how this could affect my career, specially with AI, everything career-related in Software Engineering and Data Engineering is way fuzzier than it used to be 8 years ago. So I feel like I cannot plan much ahead.

For now my goal is to some day being able to fully understand advanced physics concepts, and even be able to read new papers on some field I’d discover on this physics hobby.

Did it work?

I don’t know. But I’m having fun and I’ve noticed I’m less intellectually lazy during my day-to-day activities.

Courtesy of A24

In a world where Lindsay Clancy can inspire an onslaught of sympathetic voyeuristic fascination, why not Elizabeth Holmes? Earlier tonight, the screening of an unannounced mystery movie at the Telluride Film Festival generated every bit as much pre-screening buzz as any of the major titles programmed here. And when the movie was revealed to be “You Can See Everything,” a 174-minute documentary, co-directed by the cult comedian Nathan Fielder, about Elizabeth Holmes, the former CEO of Theranos who is now serving an 11-year prison sentence after being convicted of defrauding her investors, the sensation of “WTF? I’ve got to watch this!” did not subside as the movie went on. It only intensified.

“You Can See Everything” is just about the furthest thing from a “pure” documentary you’ve ever seen. That’s part of its appeal — it’s a one-of-a-kind reality-TV-as-hangout-movie psychodrama circus. Yet it’s been made with a spirit of true inquiry, driven by Fielder’s acidly quizzical personality (as an onscreen interviewer, he’s got the most drop-dead deadpan since Steven Wright), and it gets its hooks in you.

Fielder isn’t here to throw open the question of Elizabeth Holmes’ guilt. He wants to know what made her tick. But why, you may ask, is the comedian behind shows like “The Curse” and “The Rehearsal” making a three-hour movie about the woman who became America’s youngest self-made female billionaire by convincing the world that she’d created a revolutionary new means of blood testing? Holmes, after her conviction, stayed out of the public eye, but shortly before her prison sentence was set to begin she decided she wanted someone to document her last days of freedom. Maybe she wanted to have her say?

Whatever the motivation, Fielder, who has often operated in a nether zone between fiction and reality, was looking for a project to collaborate on with Lance Oppenheim, a documentarian he’d become friendly with. (He’s the film’s co-director.) And so, with the support of A24, and without quite knowing what they were after (or, by their own admission, what the hell they were doing), the two of them stepped into the breach. Thirty-four days before the start of Holmes’ prison sentence, in 2023, they brought a camera crew to the spacious wood-paneled beachside home in Del Mar where she was living with her partner, the California hotel heir Billy Evans, and their two children (including an infant daughter). Fielder was there to hang out, observe, and interview Holmes, trying to get inside her head. That proved to be a weirder and wilder journey than anyone might have expected.

The Elizabeth Holmes we see has a winsome soft presence, with a Cupid’s-bow grin and eyes that beam without blinking too much. She resembles the ’80s and ’90s actress Kim Greist, and has a manner that’s disarming in its gentleness. Yet when Fielder asks about the crimes she was convicted of, she pleads a strange kind of ignorance. She claims she did nothing wrong, and it’s clear she believes it. But how is that possible?

In 2015, Holmes had become a celebrity entrepreneur, based on the $9 billion valuation of her health technology company, Theranos — a company that turned out to have been built on a complete scam. For a brief time, Holmes convinced the world that she’d developed a revolutionary device for blood testing that required very small volumes of blood, such as the amount you’d get from a pinprick. She convinced top-drawer investors (including Rupert Murdoch and Larry Ellison) to pour millions of dollars into the company. For credibility, she assembled a board of directors that included such ancient luminaries as George P. Shultz, Henry Kissinger, and James Mattis. There was only one problem: the “revolutionary” technology didn’t exist.

Numerous experts, starting with the Stanford medical professor Phyllis Gardner, told Holmes that her idea wouldn’t work; you couldn’t mine voluminous amounts of data from just a few droplets of blood. But Holmes, along with her business partner, Ramesh “Sunny” Balwani (who was also her romantic partner), sold the deception anyway. For a short while they were quite “successful” at it, though in hindsight it’s clear that the castle in the air they built — all spun out of the trendy idea that medical science could be democratized — was destined to crash and burn. Because it was all made up! Holmes had spun the pinprick idea out of her own fear of needles. This was her fantasy, elevated into an invention that didn’t actually exist.

So how can she defend herself now? What she tells Fielder, with an implacable conviction that suggests a belief in her own narrative that’s nothing short of monomaniacal, is that the device patented by Theranos, which she sold as “the iPod of health care,” could, in fact, host any blood test that existed. The only thing they hadn’t quite worked out was the chemistry that would go into the tests. In other words, the device was based on the projection of a technology that didn’t exist yet. But the more Holmes talks to Fielder, the more he teases out her unhinged rationale, which is: The device works fine! We just had to figure out that pesky chemistry stuff! The strange thing is that Holmes has such a corporate mindset that she treats the chemistry stuff…that hadn’t been invented yet…as just another glitchy small problem that could easily be solved. (Have the solution on my desk Monday morning!)

She believes her version of reality. Fielder, who is not taken in by any of this (his questions remain rigorous in their skepticism), at one point gives vent to his impish side by showing Elizabeth and Billy the famous Abbott and Costello routine “Who’s on First?” When it ends, he tries to get Holmes to explain what the joke is (that “Who” is the first baseman’s name, but the name sounds like a question — hence the routine’s glorious verbal slapstick). And she can’t do it; she can’t grasp the joke. It’s not just that she lacks a sense of humor (though frankly, she kind of does). It’s that she’s got the malignant narcissist’s need to have words mean only what she decrees them to mean. She can’t hear any other meaning. Hence: her device did “work,” and still does (it can run any test!). That’s why she and Billy, years after her conviction, are still pitching Theranos to other investors (as if anyone would go near her). This is a highly rarefied level of delusional thinking.

Holmes says to Fielder that she’s never told a lie in her life. Her whole vibe is “sincerity.” (Though it’s also one of passive-aggressive unbroken-eye-contact creepiness.) The lesson of her legal case isn’t just that she was found guilty of defrauding investors, but that her faulty device, had it been put on the market, would have hurt people. That’s why the real question “You Can See Everything” is asking isn’t: Is she guilty? Rather, it’s: What makes people like Elizabeth Holmes tick? (And how many of them are out there?) Fielder and Elizabeth and Billy sit down to watch the 2022 Hulu miniseries “The Dropout,” which was based on the case, with Amanda Seyfried giving a pointed performance as Holmes. Holmes suggests that the series is a TV-movie crock. But is it?

The documentary’s first half ends with Holmes saying goodbye to her family and going off to the minimum-security Federal Prison Camp in Bryan, Texas. The prison won’t allow anyone to interview her, so that’s more or less the last we’ll see of her. And we’ve pretty much gotten the skinny on her, which is that she has a wall of denial standing between herself and the truth, and an indifference to the harm she might have caused, that borders on the sociopathic. So what is there left to discover?

Plenty, it turns out. With Holmes out of the picture, Fielder needs someone to sustain her presence. So he calls Amanda Seyfried, who sits and watches the scene we’ve just seen of Holmes sitting and watching Seyfried portray her. Are you feeling the meta of it all? That sounds like a stunt, but as the documentary goes on, and Fielder creates transcripts of things that Holmes has spoken (including a phone chat he has with her in prison), who better to read the transcripts — to turn this reality into theater, and by doing so turn it back into reality — than Amanda Seyfried? She gives an increasingly accurate performance as Holmes, getting inside her head, to the point that her acting is almost eerie in its power. Seyfried even starts to sympathize with Holmes, at least when it comes to what a glowering aggro control freak her partner Billy is. As Billy continues to make himself accessible to the filmmakers, exposing his dark side at every turn, I was afraid the movie might use him to create a kind of toxic excuse for what Holmes did.

But part of the jaunty power of “You Can See Everything” is that the film never mounts an apologia for Elizabeth Holmes. Billy may be repugnant in his cockiness, and in the way he takes control of Holmes’ image, allowing the iconic photograph of her holding a drop of blood to be used on billboards, tweeting out hostile tweets under her name, and trying to negotiate a deal with the A24 executives — yes, there’s a meeting with them in the movie, at which point we’re either in meta heaven or hell — that could result in giving him some of the profits from the movie. Does Holmes know about the deal? No. But she claims to want it that way. Her grand assertion is that she never cared about the money. And yet the money — the value of her company — became the very core of her identity. “You Can See Everything” overlaps reality, performance, deception, and mental illness until it brings you into the enigmatic space where all those things meet.

‘You Can See Everything’ Review: Nathan Fielder’s Documentary About Elizabeth Holmes Is a Shameless, Fascinating, Reality-TV-Eats-Its-Tail Look at Entitled Duplicity

Reviewed at Telluride Film Festival, Sept. 6, 2026. Running time: 174 MIN.

• Production: An A24 release. Producers: Nathan Fielder, Lance Oppenheim, Tatiana Bears, Maggie Ambrose, Jack Davis. Executive producer: Emily Osborne.
• Crew: Directors: Nathan Fielder, Lance Oppenheim. Camera: David Bolen. Editor: Adam Locke-Norton. Music: Ari Balouzian.
• With: Nathan Fielder, Elizabeth Holmes, Billy Evans, Amanda Seyfried.

We constantly work with different datasets that we normalize into our data lake. Recently we received CronosPro database files (CroBank.dat, CroIndex.dat, and CroStru.dat) that were thought to be broken because existing tooling could not parse them correctly.

Anyone who has worked with the CronosPro format, knows that its raw datafiles are notoriously difficult to convert into machine-readable formats such as CSV. A good amount of work has been put into the Cronos dump converter Cronodump alephdata/cronodump, but due to the version specific nuances of the database file structures, it doesn’t work 100% of the time. For the purpose of tackling this problem, we employed Codex for analyzing the dump structure and improved upon the Cronodump codebase to fix the parsing of the allegedly broken database.

Cronodump converter could read much of the format, but it could not decode this particular dump correctly. We had to understand how the files fit together, recover an obfuscated schema, fix several parser assumptions, and validate that parsed values still appeared under the correct schema columns.

This article explains that process from the beginning with no prior knowledge of Cronos assumed.

What is Cronos?

Cronos, also known as CronosPro, is a proprietary desktop database and information-management system. It has historically been used by organizations in Russia and other post-Soviet countries to build registries, searchable archives, document collections, and internal information systems.

A Cronos installation uses slightly different terminology from a conventional database:

Conventional term	Cronos term
Database	Bank
Table	Base
Record	Field
Field	Record ID
System Number

A Cronos database is normally represented by several related binary files:

Files	Purpose
CroStru.dat and CroStru.tad	Database structure: tables, fields, forms, and other definitions
CroBank.dat and CroBank.tad	Actual records
CroIndex.dat and CroIndex.tad	Search indexes
CroSys.dat and CroSys.tad	Information about databases known to a Cronos installation
Additional files	Forms, formulas, dictionaries, and embedded documents

The .dat files contain data. Their corresponding .tad files act as directories, telling the software where records are located inside each .dat file.

A useful mental model is:

CroStru  -> explains what the database looks like
CroBank  -> contains the actual rows
CroIndex -> helps Cronos find those rows

This distinction became central to the recovery. The record data itself was readable, but the schema needed to interpret it was protected.

What do we mean by “normalization”?

In this article, normalization does not mean relational database normalization into first, second, or third normal form.

Here it means converting a proprietary binary database into a portable and reviewable representation:

Cronos binary files
        |
        v
decoded tables and fields
        |
        v
correctly typed and aligned values
        |
        v
UTF-8 CSV files

The goal was not merely to extract strings. The result needed to preserve the relationship between:

• Tables and records
• Column names and values
• Dates and their meanings
• Text and its original character encoding
• Internal field positions
• Embedded file references

A CSV containing readable values under the wrong headers would be worse than an obvious error because it could look valid while being semantically corrupted.

Choosing an existing parser

We started with the open-source Cronodump package which provides two main commands:

• crodump inspects Cronos files and prints their internal structures.
• croconvert exports a database to CSV, PostgreSQL SQL, or HTML.

For a typical dump, conversion can be as simple as:

croconvert --csv /path/to/cronos-dump

Internally, the package performs several jobs:

1. It reads the .tad files to locate records.
2. It retrieves the corresponding bytes from the .dat files.
3. It decompresses records when necessary.
4. It decodes protected records using a KOD table.
5. It reads the schema from CroStru.
6. It uses that schema to divide CroBank records into fields.
7. It writes the resulting tables to CSV.

The repository documentation explicitly notes that parts of the Cronos format remain incompletely reverse-engineered. The parser supports many databases, but unusual version-specific details can still require investigation.

The first failure

The converter failed while reading CroStru.dat.

This meant we could not yet answer basic questions such as:

• How many tables are present?
• What are their names?
• How many fields does each table contain?
• Which field is a name, date, identifier, or file reference?
• Where does each value belong in the output?

The large CroBank.dat file contained the actual records, but without a valid schema it was effectively a stream of values separated by binary markers.

The situation looked like this:

CroStru.dat -- decoding failed --> no trustworthy column definitions
                                      |
                                      v
CroBank.dat --------------------> values cannot be interpreted safely

Before attempting to crack anything, we inspected the headers of all major files.

The dump used the Cronos 01.11 format, associated with Cronos v4. The component flags revealed an important detail:

• CroStru was KOD-encoded.
• CroBank was compressed but not KOD-encoded.
• CroIndex was compressed but not KOD-encoded.

This reduced the scope of the problem considerably.

We did not need to decode tens of gigabytes of protected record data. We only needed to recover the relatively small schema file. Once the schema was readable, the ordinary parser could interpret the much larger record file.

Compression and protection are different layers

It is useful to separate two concepts that can otherwise become confusing.

Compression changes data so that it occupies less space:

original bytes -> compression -> smaller encoded bytes

KOD protection changes byte values using a substitution table:

original bytes -> position-dependent substitution -> protected bytes

A file can be:

• Neither compressed nor KOD-encoded
• Compressed only
• KOD-encoded only
• Both compressed and KOD-encoded

The files in this dump did not all use the same combination.

That mattered because one of cronodump’s cracking methods assumes it can learn the KOD table from predictable bytes in compressed CroBank and CroIndex records. Here, those files were not KOD-encoded at all, so that assumption did not apply.

What is a KOD table?

Cronos v4 and later can protect databases by modifying a 256-entry byte-substitution table called the KOD table or KOD S-box.

At the simplest level, a substitution table says:

encrypted byte 0x00 -> decoded value X
encrypted byte 0x01 -> decoded value Y
...
encrypted byte 0xFF -> decoded value Z

Cronos adds another complication: decoding also depends on the byte’s position and the record number.

Conceptually, the algorithm is:

plaintext[i] = KOD[ciphertext[i]] - i - record_number  (mod 256)

Where:

• ciphertext[i] is the stored byte.
• KOD[...] performs a substitution.
• i is the position inside the record.
• record_number contributes another shift.
• Arithmetic wraps around at 256.

The practical consequence is that being off by even one byte changes the decoding of everything that follows.

This later explained why one schema record remained unreadable even after we recovered a plausible KOD table.

What the existing cracking tools do

cronodump includes two relevant recovery options:

crodump --strucrack ...
crodump --dbcrack ...

They use statistical properties of Cronos files rather than brute-forcing a password.

strucrack

Binary schemas contain many repeated values, especially zero bytes used in lengths, flags, padding, and integers.

strucrack groups encrypted bytes by their effective position and assumes that the most common result probably represents plaintext zero. With enough schema data, this can reveal much of the KOD table.

dbcrack

Compressed records have recognizable headers. In suitable databases, predictable bytes in those headers can be used to infer KOD mappings from CroBank and CroIndex.

Both are heuristics. They work when the source contains enough of the patterns they expect.

In our case:

• CroStru contained too little evidence for strucrack to recover every entry reliably.
• CroBank and CroIndex were not KOD-encoded, so dbcrack was not applicable.

There was also a smaller software bug: the cracking paths created a temporary argument object without a required compact property. Adding it allowed the heuristics to run:

cargs.compact = args.compact

They still did not recover a valid table automatically, but we had moved from a program error to the actual data-recovery problem.

Related upstream work

The cronodump repository contains an unmerged branch associated with pull request #22, named erdgeist-strucrack-ambigous-kods.

That work addresses the same broad weakness: strucrack may not have enough evidence to determine every KOD entry with confidence.

The pull request adds an interactive recovery workflow:

• Track confidence for each inferred KOD entry.
• Mark entries that remain unresolved.
• Detect duplicate mappings.
• Print partially decoded schema data.
• Search for likely strings such as BankName and USERINFO.
• Let the operator supply individual corrections.
• Let the operator provide known text at a particular record position.
• Improve the way strucrack is called from croconvert.

The intended process is approximately:

run automatic heuristic
        |
        v
inspect partially decoded text
        |
        v
recognize likely words
        |
        v
provide known-plaintext hints
        |
        v
resolve the remaining KOD entries

Our situation still required additional changes because KOD ambiguity was only the first problem. The upstream branch did not appear to address:

• A Cronos v4 inline-record header that shifted decoding.
• Hidden physical fields inside records.
• Incorrect handling of documented text field types.

We therefore used the same general idea, but automated the assignment differently.

Building a better statistical reference

A smaller readable Cronos component was available alongside the main dump. The cronodump repository also included a test database with known field types.

These gave us examples of what valid decoded Cronos structures look like.

We did not copy their schemas and assume the unknown database was identical. Instead, we used them to estimate general byte frequencies in Cronos schema records.

For example, valid schema data tends to contain:

• Many zero bytes
• Small binary integers
• Length-prefixed names
• Repeated structure markers
• ASCII property names
• Windows-1251 text
• Repeated field-definition layouts

For every encrypted byte in the unknown schema, we knew:

• Its ciphertext value
• Its byte position
• Its record number

For every possible KOD mapping, we could calculate the plaintext byte it would produce and score how plausible that byte was according to the reference distribution.

This produced a score table:

                    possible KOD output
                 0x00  0x01  0x02 ... 0xFF
cipher 0x00       score score score ... score
cipher 0x01       score score score ... score
...
cipher 0xFF       score score score ... score

Why this is an assignment problem

A KOD table must be a permutation.

That means:

• Every ciphertext byte has exactly one mapping.
• No two ciphertext bytes can map to the same KOD value.
• Every value from 0x00 to 0xFF must appear once.

We could not simply choose the highest-scoring value independently for each byte. Several ciphertext bytes might select the same output, producing an invalid table.

Instead, we needed to choose the best overall set of mappings while enforcing uniqueness.

This is a classic assignment problem. We solved it with SciPy’s implementation of the Hungarian algorithm:

from scipy.optimize import linear_sum_assignment

rows, columns = linear_sum_assignment(-scores)

linear_sum_assignment minimizes cost, so the scores were negated to select the maximum-likelihood assignment.

The result was a complete 256-byte KOD permutation.

At this point we had a candidate key, not yet a proven one.

Structural validation instead of “it looks readable”

Readable output alone is weak evidence. A wrong substitution can accidentally produce letters, digits, or familiar fragments.

We validated the candidate KOD table against the expected Cronos structure:

• Did records begin with valid type markers?
• Could length-prefixed names be read without running past the record?
• Did known keys such as Bank, BankId, and BankName appear?
• Did BaseNNN entries point to valid table definitions?
• Were table IDs and field counts plausible?
• Did referenced schema records decode consistently?
• Did Windows-1251 text decode into sensible field names?
• Was the final KOD table a true 256-byte permutation?

All but one schema records passed these checks, which led to the next discovery.

The first schema record had an extra header

The first CroStru record used a Cronos v4 flag value of 0x08.

Its stored data began with a 12-byte extent header:

uint64 extent_offset
uint32 extent_length

The parser treated any nonzero flag as meaning that the record bytes could be decoded directly. It therefore sent the header and payload together to the KOD decoder.

Recall that KOD decoding depends on the byte position:

plaintext[i] = KOD[ciphertext[i]] - i - record_number

Including twelve header bytes did not merely produce twelve unwanted characters. It changed the position i for every payload byte.

The whole record was consequently decoded with a position offset of twelve.

The situation was:

stored record:
[12-byte extent header][encrypted schema payload]

parser assumed:
[         encrypted schema payload          ]
 ^ decoding starts here

correct interpretation:
[skip this header][encrypted schema payload]
                  ^ decoding starts here

The parser was adjusted to recognize the v4 0x08 representation, read the extent header, and decode only the declared payload:

elif not flags or (self.isv4() and flags == 0x08):
    if self.use64bit:
        next_offset, extent_length = struct.unpack("<QL", data[:12])
        payload_offset = 12

After removing the header, the record began with:

03 04 42 61 6e 6b

At a byte level, that means:

03          schema-record marker
04          length of the following name
42 61 6e 6b ASCII text "Bank"

This was strong structural evidence that both the extent handling and the recovered KOD table were correct.

The key lesson was that a decoding failure does not always indicate a bad key. Sometimes the right decoder is being applied at the wrong boundary.

Recovering the tables

Once CroStru decoded correctly, the parser could reconstruct the logical database structure:

database definition
        |
        +-- table definition
        |       |
        |       +-- field name
        |       +-- field type
        |       +-- physical position
        |       +-- maximum length
        |
        +-- another table definition
        |
        +-- forms and other metadata

The recovered schema contained an ordinary data table and a related file table.

This allowed the converter to move from anonymous byte sequences to records with named columns.

The first CSV export was then started—but sampling it revealed another problem.

Text was exported as hexadecimal

The CSV had the correct number of columns, but many text values looked like this:

c1 c8 d7 20 c2 ce cb ce c4 c8 cc c8 d0

These were not corrupt bytes. They were Windows-1251 text represented as hexadecimal.

Cronos commonly stores Cyrillic text using Windows-1251. A previous safety change in the parser preserved unknown field types as hex rather than decoding arbitrary binary data as text. That was sensible for undocumented types, but it had also affected known textual types.

The fix was to handle documented types 1, 2, and 3 explicitly:

elif self.typ in (1, 2, 3):
    self.content = data.rstrip(b"\x00").decode("cp1251", "ignore")

The conversion path then became:

Cronos CP1251 bytes
        |
        v
Python Unicode text
        |
        v
UTF-8 CSV

Unknown binary types continued to be exported losslessly as hexadecimal.

After this fix, names and other textual values were readable, but some appeared under the wrong headers.

Why readable values were still not enough

Consider a simplified table definition:

Field A has physical position 2
Field B has physical position 4
Field C has physical position 7

The original parser assumed that because these were the three visible definitions, they corresponded to the first three stored values:

Field A <- stored value 1
Field B <- stored value 2
Field C <- stored value 3

That assumption was wrong.

Cronos records can contain internal or hidden fields that do not have ordinary visible definitions. The field definition’s idx2 value specifies its actual position in the serialized record.

The correct mapping was:

Field A <- stored value 2
Field B <- stored value 4
Field C <- stored value 7

Values 1, 3, 5, and 6 still had to be consumed even though they were not exported.

Without accounting for those gaps, the CSV looked superficially valid:

• Names were readable.
• Dates resembled dates.
• Identifiers resembled identifiers.

But the values were shifted under unrelated columns.

This is exactly the kind of error that makes output sampling essential.

Respecting physical field positions

The record parser was changed to track the current physical field position.

Before reading each visible definition, it consumes hidden fields until it reaches the definition’s idx2 position:

source_index = 1

for field_definition in table_definition[1:]:
    while source_index < field_definition.idx2 and not reader.eof():
        read_field_data(reader)
        source_index += 1

    value = read_field_data(reader) if not reader.eof() else b""
    source_index += 1

Cronos also supports complex field values prefixed by 0x1b, so field consumption had to preserve that behavior:

def read_field_data(reader):
    if reader.testbyte(0x1b):
        reader.readbyte()
        size = reader.readdword()
        return reader.readbytes(size)

    return reader.readtoseperator(b"\x1e")

After this change, bounded samples showed the expected relationships:

identifier column -> identifier-like value
name column       -> human name
address column    -> address
date column       -> valid date
category column   -> category value

Result

The final CSV conversion combined:

• The open-source cronodump package
• Ideas related to the ambiguous-KOD work in pull request #22
• Statistical KOD reconstruction
• Global assignment using the Hungarian algorithm
• Correct Cronos v4 inline-extent handling
• Windows-1251-to-Unicode conversion
• Physical field-position tracking
• Bounded semantic validation

reenigne's decoding of the 8088 microcode in 2020 opened the doors for extremely accurate emulation of the 8088 CPU.

Although I had added support for the NEC V20 in MartyPC, the V20 core was not cycle-accurate in terms of the V20's actual timings. It was an 8088 in a V20's clothing - a copy-paste of my 8088 core with V20 instructions tacked on.

This was not an ideal case, but the prospect of making my V20 core cycle-exact without the microcode seemed like it might be a discouraging slog of trial and error.

So why not get the microcode, then?

I recently commissioned InfoSecDJ to take die photography of an NEC V20 CPU (actually a second-source V20 fabricated by Sharp, but a V20 nonetheless). He did an excellent job.

The NEC V20 CPU die - InfoSecDJ

This photomosaic is extremely high resolution - 5.6 Gigapixels to be exact, an astonishing 70478x80672 resolution - too large to even fit in the JPEG image format!

You can see the entire thing at full resolution here.

The rectangular region just below the center of the die is the main microcode ROM.

The V20 Microcode ROM block

The ROM array is 258x116, containing 29,928 bits. It's evenly divisible by 29, which we know is the microcode word length of the V20, so that's a good sign. But it implies 1032 microcode words, when we were only expecting 1024. That's a bit odd, isn't it? We'll figure out the reason for that a bit later on.

Here's a close-in crop of the ROM array:

Microcode bits, zoomed in

The bright, horizontally running traces are part of the chip's metal layer. The yellowish dots are interconnects that connect the metal layer with the polysilicon layer beneath it. Note the vertical bars behind the metal layer - and note that there is occasionally a gap in the polysilicon on either side of each interconnect.

These gaps form a transistor - with the presence of a transistor indicating a 1 bit. I'll highlight the 1 bits to make that a bit easier to see.

Seeing this got me very excited - if we can visually identify the bits in the ROM, then we can extract the ROM contents. Just one slight problem - there are 29,928 bits in the array. That would be a tad bit tedious to extract by hand.

Luckily, there are tools available for this task. I loaded up the ROM image in MaskRomTool by Travis Goodspeed.

Defining bit locations with MaskRomTool

MaskRomTool allowed me to draw the rows and columns that defined the locations of the bits. Unfortunately, I found that its bit-detection methods were based on thresholding, and there wasn't enough of a difference in contrast between a bit and a non-bit to make this an effective detection mechanism. Notice the bit histogram is very compressed toward the far axis. This was not going to work. Perhaps the thresholding technique would have worked better without the bright metal layer, but I didn't want to ask InfoSec DJ to attempt removing it. Another approach was needed.

Fortunately, we could use MaskRomTool to export our defined bit positions to JSON format. I used this exported JSON file to write a Python script that extracted a square bitmap centered on each bit position, and saved it with the bit's logical column and row number in the filename.

This gave me a 105MB ZIP file of little 42x42 pixel PNG files, each containing either a bit or a not-bit. The idea behind doing this was to train a convolutional neural network to identify the bits as either 0 or 1.

If this sounds out of your depth, I had no prior experience in training CNNs before this project, and I completed it in a single afternoon. Modern Python frameworks such as PyTorch make it that easy.

There are some good CNN tutorials out there, many of them focused on the classic problem of handwriting recognition, but we have an even simpler task. We just have to sort images into two buckets, 0s or 1s. The problem is quite literally "Hotdog or Not Hot Dog."

Before we can train our CNN, we have to have a training data set. So I created a quick and dirty Python/tkinter script so I could quickly sort bits by eye into buckets as either 0s or 1s.

The quick and dirty classification tool

Shown here is a '1' bit. Can you spot it by the transistor behind the metal layer? The buttons ended up being extraneous - you just need to hit either 1 or 0 on your keyboard to classify the bit. In theory, you could do this 29,928 times and you'd have the job done in a few hours. I had originally intended for this to be my backup method in case the CNN training didn't work out - I had a few friends willing to volunteer to help, and the JSON logs that the "Bit Voter" produces can be merged to support distributed work with consensus. Fortunately, this was not needed.

Ultimately, I classified a little over 1,000 bits manually this way. Once they were sorted into two directories, we could now attempt to train a CNN model using the sorted images as input.

I won't go into great detail about how to write a CNN here - the source will be on my GitHub if you're curious. I used the excellent PyTorch library. which made the whole process a lot simpler than I could have ever imagined.

This is what a training run looks like.

 [Epoch 01] train: loss=0.6945 acc=0.7273 f1=0.0164 | val: loss=0.6924 acc=0.7876 f1=0.0000  
   val precision=0.0000 recall=0.0000 cm=[[178, 0], [48, 0]]  
 [Epoch 02] train: loss=0.6860 acc=0.7151 f1=0.3826 | val: loss=0.6257 acc=0.7965 f1=0.0729  
   val precision=0.5000 recall=0.0394 cm=[[178, 0], [46, 2]]  
 [Epoch 03] train: loss=0.3751 acc=0.8914 f1=0.7213 | val: loss=0.3900 acc=0.7655 f1=0.6327  
   val precision=0.4661 recall=1.0000 cm=[[125, 53], [0, 48]]  
 [Epoch 04] train: loss=0.1159 acc=0.9523 f1=0.9139 | val: loss=0.0495 acc=0.9912 f1=0.9773  
   val precision=0.9773 recall=0.9773 cm=[[177, 1], [1, 47]]  
 [Epoch 05] train: loss=0.0251 acc=0.9945 f1=0.9888 | val: loss=0.0460 acc=0.9867 f1=0.9744  
   val precision=0.9514 recall=1.0000 cm=[[175, 3], [0, 48]]  
 [Epoch 06] train: loss=0.0319 acc=0.9933 f1=0.9802 | val: loss=0.0438 acc=0.9823 f1=0.9659  
   val precision=0.9350 recall=1.0000 cm=[[174, 4], [0, 48]]  
 [Epoch 07] train: loss=0.0185 acc=0.9945 f1=0.9212 | val: loss=0.0274 acc=0.9956 f1=0.9891  
   val precision=0.9792 recall=1.0000 cm=[[177, 1], [0, 48]]  
 [Epoch 08] train: loss=0.0141 acc=0.9956 f1=0.9913 | val: loss=0.0271 acc=0.9956 f1=0.9891  
   val precision=0.9792 recall=1.0000 cm=[[177, 1], [0, 48]]  
 [Epoch 09] train: loss=0.0101 acc=0.9978 f1=0.9940 | val: loss=0.0447 acc=0.9867 f1=0.9735  
   val precision=0.9488 recall=1.0000 cm=[[175, 3], [0, 48]]  
 [Epoch 10] train: loss=0.0110 acc=0.9967 f1=0.9907 | val: loss=0.0437 acc=0.9912 f1=0.9773  
   val precision=0.9773 recall=0.9773 cm=[[177, 1], [1, 47]]  
 Early stopping: no val F1 improvement >= 0.0 for 3 epoch(s).  
 Best val F1: 0.9891  

If you have a CUDA-capable GPU, training is rather quick - this only took a few minutes.

The idea is that we want to maximize our accuracy - but reaching 1.0 may not be feasible, and maybe not even desirable (there's a thing called overfitting). Sometimes going on for longer just makes things worse, so we end training if we're not seeing a steady improvement.

The output of the training is a neural network model - we can then use this model to run inference on an entire input data set. Inference is just a fancy word for applying our model to actually do what we trained it to do - predict whether a given image contains a 0 bit or 1 bit.

Before we move on - a quick note to head off any potential controversies. CNNs loosely fall in the broader scope of AI from a computer science perspective, but we are not using "AI" in the modern, controversial sense that typically refers to a large language model (LLM).

When we run an inference pass, we get a confidence score for each pixel. We can use this confidence score to mark bits the model is less confident about, under some specific threshold (I used < 99% here). Here's the result of the first run, with ambiguous bits colored red:

The first output of our bit-classification CNN

I took all the ambiguous bits and manually sorted them back into the training folders, then re-ran the training, repeating until I got this result:

The final "good enough" inference run

This was pretty good - only 4 bits remain ambiguous, and it was faster just to manually verify them than to train another model.

Great, we have our 29k microcode bits and we saved hours of tedious manual labor (in exchange for hours of writing a training script in Python, but at least that is reusable!).

We still have to turn this rectangular blob of bits into a list of 29-bit microcode words. In other words, we need to reorganize the bitmap until it is 29x1032 instead of 258x116. How exactly to go about doing that is not obvious, but we can put it aside for the moment until we've decoded the matching decoder PLA.

The decode or "activation" PLA sits above the main microcode ROM block, with some intermediate circuitry sandwiched in between.

The decode PLA

The job of this PLA is to take 13 logical inputs from the left side (each input has a twin inverted signal, for a total of 26 input lines), and activate one column of the microcode array beneath it if the input signals match that column of the PLA.

What do we mean by matching?

A closer zoom might be informative:

The decode PLA, zoomed in

We have a similar arrangement of vertical metal wires, punctuated by interconnects, and gaps in the substrate forming transistors. In this case, these transistors form logical AND gates. Unlike the microcode ROM, only one transistor is ever attached to an interconnect, facing either the normal or inverted signal of an input pair. This allows the PLA to test for a 0, a 1, or to not care about that input signal at all (the case where no interconnect is present). This creates a sort of maskable Boolean logic. Since all the gates are tied together, the corresponding column of microcode will only be activated if all the gates match.

InfoSecDJ has an interesting analysis of the V20's PLA circuitry here that is worth a read.

This matching logic is very clever - almost certainly 8 of the 13 input lines are the opcode byte itself for a given instruction. Setting "don't care" bits in the PLA allows entire ranges of instructions to share the same microcode implementation, which is hugely important for efficiency's sake so that the microcode ROM doesn't take up the entire CPU die.

What's being stored here is not exactly a set of bits, but instead logic - the simple AND logic can be represented as pairs of bits, and we can extract those bits the same way we extracted the microcode ROM - drawing rows and columns in MaskRomTool, exporting the bits as images, manually sorting a thousand of them, then feeding that to our hotdog CNN.

This is our result:

The final PLA extraction

Manually decoding the first few columns of the decode PLA is promising. We essentially have:

00?00???0??00

00?00???10?00

00?000??11100

where a ? means we "don't care" about the value of the bit in that position, allowing for a range of opcodes to match.

Luckily, it appears that the 8 bits of the instruction opcode are represented in the 8 inputs starting at the fourth input.

The first row will match 64 different opcodes, starting with 00,01,02,03,08,09,0a,0b,10,11,12,13, and so on. This happens to correspond with the 8088 ISA's general ALU opcodes, which all share the same microcode on the 8088. That's a good sign.

The second line will match 04,05,0c,0d,14,15,1c,1d, and so on, corresponding with general ALU opcodes that take an immediate operand. This is surely more than coincidence. As it turns out, the matching logic for the most part is laid out in a reassuringly numerical order.

One thing to note is that there are only 257 activation lines for 1032 microcode words. This means that the "entry points" into the microcode ROM for a given instruction have to be at addresses evenly divisible by 4 - this was the case on the 8088 as well, so it's not weird to see.

Each column of the microcode contains 4 words.

The 4:1 microcode column multiplexers

Between the two blocks of microcode, a 4-way multiplexer allows two logical inputs to select one of the four words currently activated by the decode PLA. These inputs are pulled from the two low-order bits of the microcode program counter.

Given a rectangular block of microcode ROM bits, the question becomes how that block is addressed to produce a linear arrangement of microcode words. There are a number of possible permutations - from the ROM array itself, we could read from the top or bottom, and within each multiplexed row, we have a similar choice. It's also possible that lines are swapped or interleaved, so some trial and error may be necessary to produce something that looks reasonable. As it turns out, we read the microcode words in order from each column from the bottom-up.

We can tell we're on the right track if the patterns of the resulting word bitmap align with known fields. This is extremely useful if you happen to know what those known fields are. Thankfully, I had something of a clue from court documents from NEC and Intel's infamous lawsuit over the V20 microcode. Although we will see later that this diagram is not quite correct, it gave us enough of a picture to get started.

The NEC V20 microcode word format from court documents

Here you can see me musing about the reasonableness of a certain word extraction on Discord.

Working on V20 word extraction

Thankfully, finding a correct word extraction did not take long at all.

Here's what all the microcode words look like once extracted, split into five columns (each column continues from the bottom of the column to its left). I've colorized the various sections of the microcode for visual interest. I attempted to use a colorblind-friendly palette (Okabe & Ito), but apologies if this information is not visible to everyone.

The V20 microcode areas, indicated by color

Approximately two-thirds of the V20 microcode is dedicated to implementing the Intel 80186 ISA. The remainder is dedicated to either implementing NEC's own extended instruction set in the 0Fh-prefixed opcode space, or implementing the 8080 instruction set used by NEC's 8080 emulation mode. The latter accounts for nearly 1/5 of the total ROM area.

Given that we know where instructions are, and we have a rough picture of the form of microcode words, we can start determining by deduction what the various values of the source and destination operands mean. This is very similar to solving a crossword. You start at certain logical anchors - such as instructions that work with specific registers - note down guesses and assumptions as you work, and either validate or reject them.

The first four values of the first source field turn out to be segment registers - ES, CS, SS and DS. The last eight values turn out to be AX, CX, DX, BX, SP, BP, SI and DI.

I started tracking my work using an Excel spreadsheet to decode the fields and perform lookups of various field values which I kept on additional sheets.

Decoding the V20 Microcode with an Excel Spreadsheet

The Main Decoding Effort

It was at this point that I decided I could use some help. reenigne was busy with his own decoding of the 80386 microcode, so I enlisted the help of the members of the Vintage Computer Federation forums.

The thread I posted resulted in a productive collaboration with veteran VCF user dreNorterR (who had previously decoded the 80186 microcode, and so had plenty of experience in such matters). I'm not going to recount the entire thread - if you're curious to see two nerds puzzle out the secrets of the V20, it's worth a read. You can see how various fields were reverse-engineered and how we bounced ideas off each other to put the puzzle pieces together.

Unlocking the Group Decode ROM

Just staring at the microcode itself can leave one puzzled as to how certain instructions work, since the microcode itself in many cases doesn't have enough context to govern how the instructions must actually behave.

On the 8088, a separate Group Decode ROM (GDR) PLA provides additional signals that are activated by different opcodes. The 8088 GDR emits 15 signals per opcode that provide the missing context for the microcode instructions. I recommend reading Ken Shirriff's excellent blog post on the 8088 Group Decode ROM.

Being intimately familiar with the 8088, I was sure that the V20 must also have a GDR. It wasn't difficult to spot, being a large block of PLA circuitry. Many of the signals the V20's GDR emits are identical in purpose to the 8088's signals, which made decoding it more or less straightforward.

The NEC V20's Group Decode ROM PLA

Decoding the first few lines of the GDR show us some familiar patterns.

01 111100?? 00100000001000 PREFIXES  f0,f1,f2,f3
01 1111010? 00100000001000 HLT,CMC   f4,f5
01 1111?0?? 00100000000100           f0,f1,f2,f3,f8,f9,fa,fb
01 1111??0? 00100000000010           f0,f1,f4,f5,f8,f9,fc,fd
01 1111?0?0 00100000000001           f0,f2,f8,fa
01 1111?100 00100000000001 HLT,STD   f4,fc
01 00001111 00100000000001 EXT PFX   0f
01 001??110 00100000000000           26,2e,36,3e
01 01100100 00100000000001 REPNC     64
01 0110010? 00100000001000 REPX      64,65
01 0100???? 00000000011100 INC/DEC   40,41,42,43,44,45,46,47,48,49,4a,4b,4c,4d,4e,4f
01 1111?11? 01000100011011 GRP       f6,f7,fe,ff

The first mask column match matches opcodes F0, F1, F2 and F3, all of which are instruction prefixes - later on we can see the 0F opcode extension explicitly matched as well.

The True V20 Microcode Word Format

One interesting development occurred during the VCF forum collaboration - dreNorteR discovered microcode word formats that were not mentioned in the famous court documents.

As it turns out, the frequently reproduced diagram was incomplete. The left side of the microcode word can take two forms, one of which encodes an inline constant value. The right-hand side of the microcode word has four total forms, not three.

The final NEC V20 microcode word format

This division can be seen in the die photography quite clearly - 17 outputs of the microcode ROM exit the ROM in one direction, with the remaining lines exiting in the opposite direction, so clearly they had different functional divisions.

Fields like 'F', 'W', and 'E', which were left unexplained in the old diagram now have known meanings. It might have been reasonable to assume 'F' was "Update Flags" in correlation with the 8088's F field, but it actually has an entirely different meaning - Fetch.

V30

NEC had an advantage over Intel that allowed them to make a rather clever optimization. The NEC V20, like the 8088, has an 8-bit bus. The corresponding chip to the fully 16-bit 8086 is NEC's V30 CPU.

Intel designed the 8086 first, and the 8088 was derived later. This meant that the 8088 and 8086 could not be designed with the same microcode mask. NEC was able to design the V20 and the V30 at the same time, and could simply switch two lines of microcode using the metal layer that fit on top of the main microcode ROM.

In the image below, within the indicated circle, one side or arm of the metal structure was cut depending on the CPU type being fabricated. This meant both CPUs could share the same microcode mask. This also explains the discrepancy in microcode word count originally noticed!

The metal layer V20 vs V30 switch

If we zoom in, you can clearly see that a trace on the left side is cut on this V20 die where it connects to the thick post at top center. On a V30, the opposite side would be cut instead.

A zoomed-in view of the V20/V30 selection circuitry

Work Still To Do

The V20's microcode has not been 100% decoded - there are still some unidentified source values to puzzle out. There's enough of it decoded that I feel confident in beginning work on a microcode-based implementation of the NEC V20 CPU in my emulator, MartyPC. I have a hunch that the remaining mysteries will reveal themselves when we're faced with using what we know to replicate actual CPU behavior against my hardware-generated NEC V20 test suite.

Artifacts

All the resources used for decoding and my latest microcode spreadsheet are available on GitHub here.

While I’m taking a short break from low-level programming, here’s a story by a friend of mine, Sonya, about debugging a seemingly impossible bug in ARM code.

This bug hunting saga started several months ago. While developing a bare-metal hypervisor on ARM64 for postmarketOS, I hit a strange bug: if I enabled the CTR_EL0 intercept (which was the whole purpose of the HV, so could not be skipped), the phone would randomly lock itself up. After a few seconds, the watchdog kicked in and reset the system. At first I thought that the boot got slowed down so much that the system simply did not have enough time to boot, but disabling the watchdog did not help either (fortunately, the phone in question had removable battery, and I didn’t have to wait several hours for it to discharge). So, I started digging deeper.

Hypothesis #1: My emulation of MRS is wrong

On Aarch64, so-called Special Function Registers (which CTR_EL0 is one of) are accessed using MRS and MSR machine instructions:

mrs x3, ctr_el0 // read access
msr ctr_el0, x3 // write access

These instructions move data between the specified SFR and the specified general-purpose register (in this case, X3), with all other registers remaining unchanged. So the failure must have meant that I was either corrupting some of the registers I had to preserve, or not writing the real output register correctly. Thus the first two things I verified were the exception handler trampoline:

trap_from_el1:
sub sp, sp, #256
stp x0, x1, [sp]
stp x2, x3, [sp, #16]
stp x4, x5, [sp, #32]
// ...
stp x28, x29, [sp, #224]
stp x30, xzr, [sp, #240]
mov x0, sp
bl handle_trap_from_el1
mrs x1, elr_el2
add x0, x0, x1
msr elr_el2, x0
ldp x0, x1, [sp]
ldp x2, x3, [sp, #16]
ldp x4, x5, [sp, #32]
// ...
ldp x28, x29, [sp, #224]
ldp x30, xzr, [sp, #240]
add sp, sp, #256
eret

And the stack allocation:

.section .data.stack
.p2align 12
.long 0
.p2align 12
stack:

Both were correct, with no obvious signs of issues. I singlestepped the whole exception handler in QEMU and verified that it was acting exactly as expected.

To make sure things work smoothly on real hardware, I added debug prints before and after the exception handler, and it turned out that, on real hardware, the exception handler was not modifying any registers, even the intended output register.

The culprit turned out to be in this innocent invocation:

msr_accessor_sort(
    msr_accessors,
    ((uintptr_t)msr_accessors_end - (uintptr_t)msr_accessors) / sizeof(struct msr_accessor)
);

As you may know, ARM is not Icache/Dcache-coherent, which means that modifications to the data do not automatically propagate to the instruction fetches: either the modified data may not have been committed to RAM yet, or the instruction cache might be holding stale cached data from before the write. And since the buffer contained executable machine instructions, sorting the array and later trying to execute it was not going to work. So I moved the sorting to the build phase and…

The debug prints showed that the handlers worked as intended. But the system still didn’t boot.

Hypothesis #2: out-of-spec hardware

As you all know, x86(-64) hardware is only produced by two vendors – Intel and AMD, so we can expect very consistent behavior across systems.

No, OpenAI, you don’t own the em-dash.

On the ARM side the situation is different: while ARM does provide a reference implementation of the architecture, vendors are free to customize it at will, or even roll their own implementations. This means that ARM CPUs tend to have many subtle (and not so subtle) bugs, some of which were probably considered features by their developers. So the next obvious guess was that the CPU was somehow out-of-spec, and was not behaving as it was supposed to.

With that in mind, I added extra handlers to make sure that every exception is printed:

vbar_el2:
bl unknown_trap
.p2align 7
bl unknown_trap
.p2align 7
bl unknown_trap
.p2align 7
bl unknown_trap
.p2align 7
bl unknown_trap
.p2align 7
bl unknown_trap
.p2align 7
bl unknown_trap
.p2align 7
bl unknown_trap
.p2align 7
b trap_from_el1
.p2align 7
bl unknown_trap
.p2align 7
bl unknown_trap
.p2align 7
bl unknown_trap
.p2align 7
b trap_from_el1
.p2align 7
bl unknown_trap
.p2align 7
bl unknown_trap
.p2align 7
bl unknown_trap
.p2align 7

The unknown_trap routine would then print out the X30 register (lr for those of you more familiar with Aarch32), ELR_EL2, and other SFRs to determine the cause of the exception. However, none of this was actually firing.

The next best guess was that the kernel was panicking due to some wrong handling, so the next thing I did was using /proc/last_kmsg to read the crashed kernel’s logs. Unfortunately for me, the last messages I got in the log were:

[    3.113676]  (0)[153:init]fs_mgr: Running /system/bin/e2fsck on /dev/block/platform/mtk-msdc.0/11230000.msdc0/by-name/userdata
[    3.125072]  (0)[158:e2fsck]random: e2fsck urandom read with 12 bits of entropy available

This meant that the kernel did not crash cleanly. I suspected that the /dev/urandom device was responsible for the crash, and patched it out of the kernel to test that hypothesis – which made the boot go a bit further, but not much. Still a dead end.

At that point I had no idea where exactly things were going wrong, but I knew it had something to do with the intercepts, since disabling the intercepts entirely fixed the issue. So, I filtered the kernel for suspicious instructions using objdump:

$ aarch64-unknown-linux-gnu-objdump -D -b binary -m aarch64 kernel.orig | grep ctr_el0

This yielded 22 matches, which I manually patched in the binary to return the correct value. After booting the patched kernel, it was still unable to make it to Android, but adb shell worked, which meant that the patch was (at least somewhat) successful. After that I was able to reduce the patch to only a few “hot” instructions, but was still clueless about the actual reason for the hangups. At that point I suspected that my hypervisor and the kernel were somehow overwriting each other’s memory, so I decided to rewrite the handling for mrs x3, ctr_el0 (the instruction inside all the “hot” patches that actually mattered) in assembly without using any memory:

// save X0, will use it as a scratch register
msr tpidr_el2, x0

// read the Exception Syndrome Register; the value we're interested in is 0x6232c061
// (https://esr.arm64.dev/#0x6232c061)
mrs x0, esr_el2

// compare x0 to 0x6232c061
// ARM does not support 32-bit immediates in instructions, so compare bit groups one by one
sub x0, x0, #0x61
ror x0, x0, #12
sub x0, x0, #0x32c
ror x0, x0, #12
sub x0, x0, #0x62
cbnz x0, 1f // if the ESR is incorrect, go to the generic handler

// increment the saved PC by 4, to account for the instruction's length
mrs x0, elr_el2
add x0, x0, #4
msr elr_el2, x0

// restore the saved X0, perform the requested read, and return to caller
mrs x0, tpidr_el2
mrs x3, ctr_el0
eret

1:

// restore x0 before falling through to the generic handler
mrs x0, tpidr_el2

And… it worked! At that point I knew that the logic itself was correct, and it was the C handler that was somehow causing troubles, so I started bisecting further.

if(esr == 0x6232c061)
{
    asm volatile("mrs %0, ctr_el0":"=r"(regs[3]));
    return 4;
}

…worked. I looked up the address of the msr_accessor used for reading CTR_EL0, and found it to be at msr_accessors+0x28.

if(esr == 0x6232c061)
{
    regs[3] = ((uint32_t(*)(void))(msr_accessors+5/*8 bytes per element*/))();
    return 4;
}

…didn’t work. Is the relocation to blame? I checked the relocation code and tried to set the linkage base to be equal to the actual load address, but to no avail.

Then I decided to use the linker script to factor this out into a “new” symbol:

get_ctr_el0 = msr_accessors + 0x28;

if(esr == 0x6232c061)
{
    uint32_t get_ctr_el0(void);
    regs[3] = get_ctr_el0();
    return 4;
}

And this ran correctly. So now I had two versions of semantically equivalent code, only one of which worked. Time for binary bisecting!

Binary bisecting, or a poor girl’s ICE

At that point I factored the offending code out into a separate function:

static __attribute__((noinline,optimize(3))) void handle_mrs_x3_ctr_el0(uint64_t* regs)
{
    // asm volatile("mrs %0, ctr_el0":"=r"(regs[3])); // works
    // regs[3] = get_ctr_el0(); // also works
    regs[3] = ((uint32_t(*)(void))(msr_accessors+5))(); // does not work
}

And called it from the main handler:

if(esr == 0x6232c061)
{
    handle_mrs_x3_ctr_el0(regs);
    return 4;
}

The issue still reproduced, which meant that I could now focus on a single function.

At first I thought that code size could be the culprit, so I added a bunch of NOPs into the beginning of the function, but to no avail. Then I disassembled it.

Working version:

0000000040204fc0 <handle_mrs_x3_ctr_el0>:
    40204fc0:   a9be7bfd        stp     x29, x30, [sp, #-32]!
    40204fc4:   910003fd        mov     x29, sp
    40204fc8:   f9000bf3        str     x19, [sp, #16]
    40204fcc:   aa0003f3        mov     x19, x0
    40204fd0:   94000ab6        bl      40207aa8 <get_ctr_el0>
    40204fd4:   2a0003e0        mov     w0, w0
    40204fd8:   f9000e60        str     x0, [x19, #24]
    40204fdc:   f9400bf3        ldr     x19, [sp, #16]
    40204fe0:   a8c27bfd        ldp     x29, x30, [sp], #32
    40204fe4:   d65f03c0        ret

Broken version:

0000000040204fc0 <handle_mrs_x3_ctr_el0>:
    40204fc0:   a9be7bfd        stp     x29, x30, [sp, #-32]!
    40204fc4:   f0000001        adrp    x1, 40207000 <phys_ceiling_names>
    40204fc8:   f944f821        ldr     x1, [x1, #2544]
    40204fcc:   910003fd        mov     x29, sp
    40204fd0:   f9000bf3        str     x19, [sp, #16]
    40204fd4:   aa0003f3        mov     x19, x0
    40204fd8:   9100a021        add     x1, x1, #0x28
    40204fdc:   d63f0020        blr     x1
    40204fe0:   2a0003e0        mov     w0, w0
    40204fe4:   f9000e60        str     x0, [x19, #24]
    40204fe8:   f9400bf3        ldr     x19, [sp, #16]
    40204fec:   a8c27bfd        ldp     x29, x30, [sp], #32
    40204ff0:   d65f03c0        ret

I started unifying the functions in assembly, making sure that the working version still works and the broken one still doesn’t, until I arrived at this:

handle_mrs_x3_ctr_el0:
stp x29, x30, [sp, #-32]!
mov x29, sp
str x19, [sp, #16]
mov x19, x0
adr x0, get_ctr_el0 // load x0 with the address of get_ctr_el0

#if 0 // broken version
blr x0 // call the function at address stored in x0
#else // working version
bl get_ctr_el0 // call the function get_ctr_el0
#endif

mov w0, w0
str x0, [x19, #24]
ldr x19, [sp, #16]
ldp x29, x30, [sp], #32
ret

It’s easy to see that both versions are semantically equivalent – which meant that either someone interrupts the code at that exact moment and fails to restore registers, or there is a microarchitectural bug. I quickly dismissed the first possibility: on ARM64, the only higher-privileged thing than a hypervisor is the trustzone, and it had no reason to receive any interrupts, so I assumed the latter. The SoC in this phone is a MediaTek MT6735 with Cortex-A53 cores, and Alisa helped me look through the list of Cortex-A53 errata, but we did not find anything remotely similar…

And then I remembered a certain comment in Linux source. On some x86 systems, speculative access to some MMIO registers would cause the system to shut down. So I devised a thought experiment: if this was the case here, how would I prevent it? The easiest way to prevent any accesses to a physical address, including speculative ones, is to never have the address mapped, but that would interfere with the hypervisor’s own use of MMIO space (it expects a 1:1 mapping of the whole address space as a design choice). I started contemplating a “lazy paging” scenario, where I would lazily map in ranges of memory that were architecturally accessed (CPUs don’t signal page faults for speculative accesses), and then it hit me.

How do the two instructions differ? blr x0 is dynamic, bl get_ctr_el0 is static.

How are dynamic dispatches different from static branches? Dynamic dispatches employ branch prediction, static branches know where they lead ahead of time and cannot mispredict.

x86 AMD CPUs can actually mispredict an unconditional direct jump, but that’s besides the point.

What could an instruction mispredict to? …Probably a null pointer, 0x0. Not like it can spawn an address out of thin air.

What do I have in my 1:1 mapping at 0x0? …Eeeeeeh… The bootrom, which is locked out during bootloader initialization.

…And then I realized. The speculative accesses I was fighting were not data accesses, they were instruction fetches. And that meant that all I had to do is mark the pages as non-executable. Which I did and… the system booted all the way to Android, without any patches in the kernel, for the first time in half a year!

After I knew what to look for, a quick Google search for “arm64 speculative instruction fetch mmio” yielded the following gem from the official ARM documentation:

There is a subtle distinction here that is easy to miss. Marking a region as Device prevents speculative data accesses only. Marking a region as non-executable prevents speculative instruction accesses. This means that, to prevent any speculative accesses, a region must be marked as both Device and non-executable.

Not just security

Being somewhat of a hacker myself, I always considered Data Execution Prevention (DEP) to be exclusively a security measure invented to combat stack overflow attacks, so I didn’t bother to implement it: the hypervisor was never meant to run in production, so defense-in-depth would be overengineering.

But it turns out that on ARM it’s not about security at all – instead, it’s essentially an attribute bit. ARM guarantees that, for regions mapped as Device memory (newspeak for MMIO), there will be no speculative accesses. However, that only applies to data accesses – instruction fetches treat any executable memory as Normal memory. The only way to prevent a region of memory from being speculatively executable is to prevent it from being executable at all.

A possible workaround, in case you have to run code from Device memory, would be to disable the Icache for the duration of its execution. However, ARM documentation says such accesses are still illegal and discourages this.

Also, since I had to implement a way to map non-executable memory anyway, I decided to map all memory except the payload as non-executable, so my hypervisor now finally has a bit of defense-in-depth too!

Made with my own bare hands (why.)

When I started programming, my whole day revolved around development. I’d wake up, write code, and go to sleep. And that’s how it was day in and day out. My entire social circle consisted of people who also wrote code. I studied frameworks, libraries, and various programming languages.

I got into development because I was fascinated by the idea that by writing code, I could create new programs. All I needed to do was learn how to program, and I’d be able to create anything. If I learned PHP, I could build back-end applications. If I learned HTML and CSS, I could build websites.

I could choose what I wanted to create, learn the necessary technologies for it, and start building. You open an IDE, create a new project, and build it from scratch. You choose patterns, libraries, and frameworks; you read the documentation and dive into the source code. You get so immersed in the process that you don’t even notice how the days fly by. Over time, you develop your own coding style.

I wrote code, and I enjoyed writing code, but it wasn’t the kind of work I wanted to do for the rest of my life. I preferred creating something new to the actual process of writing code. I liked starting from scratch — with nothing but an idea — to create a product that people actually use. That’s why I’ve always worked exclusively at startups and never at large companies.

I’ve always dreamed of advancing my career — to become a team lead, a manager, or to take on another role that didn’t involve writing code. And whenever someone else could write the code, I’d shift away from coding toward managerial tasks. Whether I wrote the code myself or delegated it to others didn’t matter to me.

Although I got into development out of an interest in creating things, later on I did it solely because I was getting paid for it. For me, it was just a job. Even though I wrote code in my free time, I still used that approach to build my own startups, which I planned to make money from. Writing code and making money were inseparable. Over time, income from projects became the main focus, rather than simply the desire to create projects.

But there’s another type of person. These are true programmers who write code regardless of income. They write code simply because they can’t help but write it. It doesn’t matter to them how much money they make from it. They might work on open-source projects that will never generate any income for them.

They enjoy writing code, solving development challenges, designing, figuring out the logic, fixing bugs, and understanding the entire system as a whole. They enjoy the process itself. And they’ll never hand that over to AI agents.

I love building startups, and they love writing code. For me, building projects is a way to make money. For them, programming is art.

That’s why all my arguments about how AI helps me build projects are completely irrelevant to true artists. They don’t need AI to help them with anything.

And that’s why there’s often a negative reaction to the claim that AI will soon replace all programmers. For artists, this claim itself is illogical. Even if some technological breakthrough occurs and AI can truly work autonomously and solve problems hundreds of times faster than humans, people will still continue to write code by hand, line by line. AI will never be able to take away their ability and desire to write code.

I don’t use AI when writing. It produces soulless, bland text. AI slop that no one wants to read. I want to come up with the wording myself, proofread and edit the text, come up with a title, and delete unnecessary sentences and entire paragraphs. I like it when new ideas for the text come to me as I’m writing. The writing process itself — the thinking that goes on in the moment — is important to me, not just the result. If I asked an AI to write a text on the topic “Programming is art”, that thought process wouldn’t happen. I wouldn’t experience the same emotions that come with writing it myself. For me, writing is art that I pursue regardless of money. I think people who enjoy writing code feel the same way.

An international team including Nobel Prize-winning physicist Professor Sir Roger Penrose has observed a long-predicted effect of gravity on a falling quantum object for the first time. The result shows that a fundamental principle at the heart of Einstein’s theory of gravity remains consistent with the behaviour of matter in the quantum world. The study, led by Ben-Gurion University of the Negev, The University of Ulm and the University of Oxford, has been published in Science Advances.

Artistic image of blue glowing quantum waves in space abstract background

A new study involving Oxford physicists provides an experimental connection between quantum physics and Einstein's theory of gravity. Image credit: sakkmesterke, Getty Images.

For more than a century, physicists have relied on two extraordinarily successful descriptions of nature. Quantum mechanics explains the strange behaviour of atoms and other tiny objects. Einstein’s theory of gravity explains how objects fall and how gravity shapes the Universe. Yet physicists still do not fully understand how the two fit together.

Now, an international team has performed an experiment that probes the point where they meet. In the study, the researchers observed a distinctive change in the quantum properties of atoms as they fell under gravity. Crucially, the effect they measured is the same one predicted when Einstein’s equivalence principle, a cornerstone of his theory of gravity, is applied to a quantum object.

The equivalence principle states that for an observer in free fall, gravity should locally disappear. Someone falling freely in a lift, for example, would experience weightlessness. Whilst this theory has survived extraordinarily precise tests involving ordinary matter, it was unclear how this could be experimentally tested with quantum objects, which can behave as waves and effectively travel along more than one path.

At the heart of the experiment is a new apparatus the researchers call the Quantum Galileo Interferometer in honour of Galileo’s work on gravity. It allowed them to do something unusual: effectively split the quantum wave associated with an atom into two paths, hold one in place while allowing the other to fall freely, and then reunite them to see how gravity had changed the falling wave. As this work required a new type of quantum interferometry, quantum theory specialist Professor Wolfgang Schleich from the University of Ulm developed the quantum understanding of the apparatus used.

Putting an Einstein principle to a quantum test

The experiment was carried out at Ben-Gurion University using clouds of rubidium atoms cooled to just above absolute zero and manipulated close to the surface of a specially designed atom chip.

The 2D MOT apparatus which feeds the science chamber with cold atoms. Credit: Or Dobkowski.

The experimental team, including PhD student Or Dobkowski, first used microwave pulses to put the ultracold atoms into a quantum superposition, effectively allowing each atom to travel along two different paths at once. They then used tiny electrical wires on the chip to generate precisely controlled magnetic fields. One part of the atomic wave responded to this magnetic field, allowing the researchers to apply an upward force that exactly counteracted the downward pull of gravity. In effect, this part was held stationary relative to the laboratory and the Earth**.**

The other part was pushed upwards with a precisely controlled magnetic pulse, then switched into a state almost unaffected by the magnetic field so that it could fall freely under gravity – following a ballistic trajectory, similar to a ball thrown into the air.

At the end of the fall, the researchers used another precisely controlled magnetic pulse to bring the two parts back together. When the two waves were reunited, they interfered with each other. That interference allowed the researchers to measure the tiny difference in quantum phase accumulated while one was falling and the other was held still.

The phase measured in the new experiment is the same as the one predicted when Einstein's principle is applied to such a quantum wave. The result therefore provides an experimental connection between quantum physics and Einstein's theory of gravity.

Although previous experiments have used quantum particles to measure gravity, the researchers say this is the first direct measurement of the predicted quantum phase of a freely falling object.

Lead author Professor Ron Folman (Ben-Gurion University of the Negev) said: 'This is a unique paper, in the sense that it combines a hard experiment with a far-reaching theoretical interpretation, about one of the most fundamental questions in physics: How can gravity (described by Einstein’s theory of relativity) and quantum theory, be unified into one understanding of the universe? These two pillars of modern physics have so far eluded all attempts at a unified theoretical framework, but this complex experiment gives more hints as to how such a unification may be achieved.'

— Study co-author Professor Vlatko Vedral, Department of Physics, University of Oxford

“We have no consistent theory telling us why quantum physics should fail. This experiment pushes quantum mechanics into one of its most intriguing frontiers, gravity, and shows that, once again, its predictions hold.”

— Study co-author Professor Vlatko Vedral, Department of Physics, University of Oxford

The atom chip used in the experiment (fabricated at Ben-Gurion University of the Negev). In the experiment the chip was upside-down and the atoms manipulated just under it. Credit: Ben-Gurion University of the Negev.

The result does not unite quantum mechanics and gravity, nor does it show that gravity itself is quantum. Instead, it demonstrates that Einstein’s equivalence principle remains consistent with quantum mechanics in the regime tested.

Also, the study does not overturn an argument made by study co-author Professor Sir Roger Penrose (University of Oxford) that quantum mechanics could break down for sufficiently massive objects held in quantum superpositions for long enough times. Whilst the present experiment did not reach the masses or timescales needed to test this idea, the research team hope the technique will be a step towards experiments with much heavier objects, including nanodiamonds, that could investigate this possibility. Such an experiment is now underway in the same group at Ben-Gurion University of the Negev.

The international study included researchers from Ben-Gurion University of the Negev; the University of Oxford; the University of Southampton; German Aerospace Center, the Institute of Quantum Technologies, Ulm; Universität Ulm; and Texas A&M University.

The study ‘Observation of the quantum phase of free fall and the consistency with the equivalence principle’ has been published in Science Advances.

For more information about this story or republishing this content, please contact *[email protected]*.