A hacker collective pulled down a Flock camera and dumped its data. The files included thousands of videos and logs showing that the device captured 1.6 million images of 50,000 vehicles in 21 days.

Hackers Got Inside a Flock Camera. Its Data Shows How the System Really Works

Photo-Illustration: Jobanny Cabrera; Getty Images

Hackers ripped down a Flock camera above a roadway, made a near-complete copy of the data stored inside it, and shared the files with 404 Media and WIRED, revealing in new detail how exactly Flock Safety’s cameras track the movements of both vehicles and people. The hackers say they are also publishing details on how they managed to obtain the software, in the hopes that other people may copy them.

The breach provides an unprecedented look inside a system that Flock has described as protected by on-device encryption. The hackers were able to copy the camera’s storage and recover an encryption key stored on the device, which unlocked videos of thousands of vehicle detections. The hackers shared the material with 404 Media and the transparency nonprofit Distributed Denial of Secrets, which shared the data with WIRED. 404 Media and WIRED then analyzed those files as part of a joint investigation.

While much of the automatic license plate reader’s most sensitive storage remained encrypted and inaccessible, the joint analysis of the recovered data shows that software running on the device explicitly detects people as well as vehicles, license plates, and bicycles. The camera can produce dozens of images of a single passing vehicle and, according to several weeks of recovered logs, generated more than a million images. Its computer-vision software also sometimes isolated bumper stickers and other graphics, including, in one case, an American flag patch on a motorcyclist’s saddlebag.

The act of removing the camera and dumping its software shows that some people are not content with just destroying or removing the cameras. Across the country, multiple people have been arrested for allegedly tampering with or otherwise sabotaging Flock’s cameras. In response, some towns have announced that they are going to stop using Flock’s cameras altogether, and in one case, a police department even made a fake, 3D-printed Flock camera case in order to bait potential vandals.

“Why just destroy them when we can reverse engineer them and find the secrets of those spying on us?” one of the hackers, from a collective calling itself stegan0gram, said in an interview. “We liberated hardware in the field, disarmed them, and proceeded with reverse engineering of the cameras and associated solar equipment.”

Flock’s cameras photograph passing vehicles and send the images and other data to the company’s servers. There, Flock’s system presumably reads the license plate and can identify characteristics such as the vehicle’s color, make, and model. Flock then makes these time-stamped records searchable by whichever local agency owns or has access to the cameras. But in many cases, Flock’s system also allows other police departments from all over the country to search those cameras too, as part of the company’s national network. In Alpharetta, Georgia, for example, WIRED found that records from the city’s Flock cameras were accessible to more than 2,000 agencies, including police departments, colleges, airports, and, inexplicably, the Office of Inspector General for the federal General Services Administration.

This national network has been a selling point for Flock but also a deep source of controversy. 404 Media revealed that local cops were performing lookups in the national network on behalf of Immigration and Customs Enforcement, including in areas that banned working with immigration authorities or transferring license plate data out of state. 404 Media also revealed that a cop in Texas searched Flock cameras nationwide for a woman who self-administered an abortion. Those stories, among others, triggered a national conversation about whether people want Flock cameras, or automatic license plate readers more generally, in their communities.

And in the case of stegan0gram, the answer is clearly no.

The hackers said they were able to access the Android system on the camera and found two partitions—sections of its hard drive, essentially. A few of these were unencrypted, the hackers said, including one called “vendor” and another called “media.” The latter contained an encryption key that unlocked another part, which contained much of the media—the videos and stills—the camera took.

In early 2025, security researcher Jon “GainSec” Gaines reverse engineered a Flock license-plate reader and documented flaws that could be used to gain root-level access. After Gaines disclosed his findings, the company acknowledged the findings but downplayed their severity, writing that the flaws required physical access to the device and that even someone who gained access to a camera “would still not be able to gain access to footage,” because images remained on the device only briefly after being transmitted to the cloud.

404 Media and WIRED analyzed the camera’s contents. The device’s processor is similar to those used in midrange smartphones, and it runs about 20 Flock-built apps that handle everything from detecting motion and taking pictures to classifying objects, uploading data, and receiving remote updates.

According to the code, when something moves into view, the camera takes a rapid series of photos. A typical passing vehicle generated about 28 images, though some produced more than 100. The camera uses different exposures to capture both the license plate and the wider scene, then scans the images, selects and crops useful frames, and sends them with other data to Flock over the cellular network. The camera itself does not appear to read the plate or identify the vehicle’s make, model, and color. That appears to happen on Flock’s servers.

According to our analysis, the camera’s logs recorded about 21 days of activity across several periods. During those windows, the device photographed roughly 50,200 vehicles and generated about 1.6 million images. On a typical day, it logged around 3,300 vehicles, with a high of 4,454. Those figures would vary considerably depending on where a camera is installed and how much traffic passes in front of it. The camera was almost certainly operating outside those periods, but older logs had been overwritten or were no longer recoverable from the device.

The software running on the camera explicitly detects people, something which is typically overlooked in discussions around Flock cameras. When it spots a person, it records where they appear in the image and how confident it is in the detection.

To test what the software could actually see, WIRED extracted the models from the camera’s files and ran them against test images and footage recovered from the device. The models readily detected people, including a selfie of a reporter. WIRED then ran them across 27,321 short videoclips stored on the camera. The clips were MP4 files, each about one to two seconds long, recorded at 1,024 by 768 pixels without audio. They were separate from the rapid bursts of higher-resolution still images the camera also takes as vehicles pass. The models detected people in 11 of the clips, all of them riding motorcycles. The small number is likely due to the camera’s position above a roadway, pointed down at passing traffic where pedestrians were unlikely to appear.

The tests also showed how broadly the camera’s license plate detector could interpret what it saw. In some cases it mistook bumper stickers, dealership frames, and other graphics for license plates and cropped them out as if they were plates. In one video of a passing motorcycle, the detector cropped an American flag patch on the rider’s saddlebag as if it were a plate.

Flock insists its cameras do not perform face recognition. WIRED and 404 Media found no evidence of any face-recognition capabilities in the camera’s software beyond ones included by default in the Android operating system. Those capabilities did not appear to be enabled or in active use.

In August, WIRED obtained frontend code for Flock’s police software, now called OS Investigate and previously known as Nightshift, and reconstructed portions of the tool. That software showed how Flock can use the records generated by its cameras, along with police files and commercial data, to identify drivers, surface vehicles that repeatedly travel together, and search for people based on patterns of movement. The data provides a view of the other end of a system.

A Flock spokesperson said in a statement: “The unauthorized removal and tampering of a Flock camera is illegal.” When asked specifically about the encryption key stored on the camera, the company added, “Flock takes security seriously and maintains a public Vulnerability Disclosure Policy for security researchers to report potential vulnerabilities directly to us. We received no report through that process, and based on the limited information provided, we do not have enough detail to assess the claims being made. If the individuals identified legitimate vulnerabilities, we encourage them to submit their technical findings through our vulnerability reporting process so our security team can review them and take any appropriate action.”

One of the hackers said, “Being investigated is a legit concern and something we are trying to avoid. I'm sure our actions have attracted some attention as it is, but we are careful and try to keep a low profile.”

Noel Pichardo, a former Pawtucket, Rhode Island, police officer who became an outspoken critic of Flock after challenging his department’s use of the cameras, says he understands the activists’ frustration but worries that sabotaging devices could ultimately strengthen the case for them. “I think that type of vigilantism will only crystallize the police and the state at large in their belief that this tool is necessary,” Pichardo says. “The longer the state continues to ignore the groanings of their constituents who are against this type of surveillance, the more this will happen.”

The camera’s logs also show the camera struggling with storage. Its logs recorded more than 27,000 “no space left on device” errors while trying to save full-resolution images, along with tens of thousands of related errors, crashes, and reboots. At the same time, about every two minutes, code checked that the camera was still running and logged the message, “Who’s a good boy?!” More than 12,000 of those messages appear in the recovered logs.

When the camera did restart, another service left a final message in the logs: “A reboot was requested! ¡Adiós, Amigos!”

...or how to make Qwen learn query optimization via agentic reinforcement learning

A reinforcement-learning policy update

Four RL rollouts commence for a single query. Qwen produces a candidate strategy per rollout and sends it off to Postgres for measurement against its own default plan. Scalar rewards are assigned to each rollout, which flow backwards to update Qwen's weights.

Weights nudged toward the faster plans.

How good are query optimizers, really?

Leis et al. asked this exact question in 2015. Then, they asked it again 10 years later.

Despite an enormous body of research spanning a decade since their original exploration, they found that query optimizers continue to leave much to be desired.

I was surprised when I first learned about this. A Postgres database should know everything about the stuff that lives in its tables, no? How hard can it be?

As it turns out: enormously hard. In fact, one particular task a query optimizer needs to do, join ordering, is known to be NP-hard.

So query optimizers are hard. What’s not as hard is verifying whether a query plan an optimizer picks is good or not. Put simply, a good query optimizer produces plans that run fast, and a bad one produces slow plans. Language models are particularly good at learning how to do tasks with easily verifiable outputs. Because there’s a single axis to optimize for—execution time of a query—the problem beautifully reduces to reinforcing the behaviors that guide a model to produce faster query plans.

What follows is a breakdown of an experiment I ran to explore the question: can a small, open-weights model be post-trained via supervised fine-tuning (SFT) and agentic reinforcement learning (RL) to produce Postgres query plans that beat Postgres’s default plans?

The answer to our question is a resounding yes. Highlights include:

• Attaining a 44.7% latency reduction across 113 join-heavy queries from a 4B model initially unable to produce a query plan for 99 of them
• Constructing a Postgres measurement rig that minimizes Linux page cache contention noise across concurrent containers
• Designing a custom GRPO variant for scoring RL rollouts in an inherently noisy environment
• Splitting RL across two machines: vLLM and the trainer on a rented 2x H100 node and four Postgres containers running on my desk
• Running off-policy distillation across half a thousand GPT-6 Astra agent trajectories

Let’s start from the beginning.

Inside a query optimizer

Consider the following slice of the IMDb dataset:

-- An IMDb title (movie, series, episode, etc.) [~1M rows]
title (
  id              integer PRIMARY KEY,
  title           text,
  production_year integer,
  kind_id         integer -- FK -> kind_type
)

-- Movie <> company junction table [~2M rows]
movie_companies (
  id              integer PRIMARY KEY,
  movie_id        integer, -- FK -> title.id
  company_id      integer, -- FK -> company_name.id
  company_type_id integer, -- FK -> company_type.id
  note            text
)

-- A company's name, origin, etc. [~100k rows]
company_name (
  id           integer PRIMARY KEY,
  name         text,
  country_code text     -- '[us]', '[jp]', ...
)

-- Lookup table of company roles for a title [4 rows]
company_type (
  id   integer PRIMARY KEY,
  kind text -- 'production companies', 'distributors', ...
)

-- Lookup table for what a title _is_ [7 rows]
kind_type (
  id   integer PRIMARY KEY,
  kind text -- 'movie', 'tv series', 'episode', ...
)

Let’s say I’m trying to answer the question: “Which Japanese companies put out the most titles in the 2000s?” We might write the following query:

SELECT cn.name,
       COUNT(*) AS titles
FROM   title AS t,
       movie_companies AS mc,
       company_name AS cn
WHERE  t.id = mc.movie_id
  AND  mc.company_id = cn.id
  AND  cn.country_code = '[jp]'
  AND  t.production_year BETWEEN 2000 AND 2009
GROUP  BY cn.name
ORDER  BY titles DESC
LIMIT  10;

Running this query outputs 10 Japanese companies with the number of titles they were associated with between 2000 and 2009, sorted from highest to lowest.

But how did Postgres get these results?

The path Postgres took to get this data for us is not a foregone conclusion, and it has everything to do with what we call selective predicates (i.e. the filtering conditions in a WHERE clause).

To illustrate this, let’s imagine our same query without the Japanese company filter or the date range filter:

SELECT cn.name,
       COUNT(*) AS titles
FROM   title AS t,
       movie_companies AS mc,
       company_name AS cn
WHERE  t.id = mc.movie_id
  AND  mc.company_id = cn.id
GROUP  BY cn.name
ORDER  BY titles DESC
LIMIT  10;

mc can only join with cn via mc.company_id = cn.id, and t can only join with mc via t.id = mc.movie_id.

These constraints produce two valid join trees:

The two join trees for our query. The lower join runs first; the result is an input into the root join.

The cardinality of a table or query result is the number of rows it contains. Assume the relevant tables have the following cardinalities:

1. cn=100k
2. mc=2m
3. t=1m

Taking into account our joins, we get the following cardinalities:

• (cn⋈mc)=2m, then ⋈t=2m
• (t⋈mc)=2m, then ⋈cn=2m

Regardless of the order in which these three tables are joined, the same 2m rows are always passed into the second join.

Now let’s add back our selective predicates:

1. cn′=5k (assuming 5% of our 100k companies are Japanese)
2. mc=2m (does not change)
3. t′=200k (assuming 20% of our 1m titles were made in the 2000s)

• (cn′⋈mc)≈100k, then ⋈t′≈20k
• (t′⋈mc)≈400k, then ⋈cn′≈20k

The first join ordering filters the 2m movie_companies entries down to the 5% slice of companies that are Japanese. Assuming uniform distribution (we’ll discuss later why we assume this), this join results in approximately 100k rows. Joining the result with the filtered title table keeps only the 20% of those rows from the 2000s.

The second join ordering filters the 2m movie_companies entries down to the 20% slice of titles that were made in the 2000s. The same uniformity assumption holds, so the first join results in 400k rows, meaning we’re passing 400k rows into the second join.

We do 4x the work if we picked the second join ordering.

Unfortunately, it doesn’t stop there.

A combinatorial explosion

Each join can use any of:

1. Hash join
2. Merge join
3. Nested-loop join

Factoring commutativity back in now, there are 4 different outer/inner join orientations, resulting in 8 possible combinations.

Lastly, each table can be scanned in different ways. Considering just four types of scans:

1. Sequential
2. Index
3. Index-only
4. Bitmap

• 2 Join trees: which pair of tables joins first.
• × 2² Orientations: each of the 2 joins can swap which input is outer and which is inner.
• × 3² Algorithms: each of the 2 joins picks hash, merge, or nested loop.
• × 4³ Scans: each of the 3 tables is either read sequentially or via index, index-only or bitmap scans.
• = 4,608

There are 4,608 different ways to run this query. This is actually an undercount. Plans can run in parallel, aggregates can be hashed or sorted, etc.

It’s also worth noting that Postgres doesn’t evaluate all of these plans. It uses dynamic programming (and a genetic algorithm for queries involving 12+ joins) to prune the search space.

To make matters worse, every join combinatorially explodes the search space:

SELECT cn.name,
       COUNT(*) AS titles
FROM   movie_companies AS mc,
       company_name AS cn
WHERE  mc.company_id = cn.id
  AND  cn.country_code = '[jp]'
GROUP  BY cn.name
ORDER  BY titles DESC
LIMIT  10;

• 1 Join tree: with two tables there is only one way to join them.
• × 2¹ Orientation: 1 join means there are only 2 orientations.
• × 3¹ Algorithm: the join algorithm can be a hash join, merge join or nested loop.
• × 4² Scans: each of the 2 tables is either read sequentially or via index, index-only or bitmap scans.
• = 96

SELECT cn.name,
       COUNT(*) AS titles
FROM   title AS t,
       movie_companies AS mc,
       company_name AS cn
WHERE  t.id = mc.movie_id
  AND  mc.company_id = cn.id
  AND  cn.country_code = '[jp]'
  AND  t.production_year BETWEEN 2000 AND 2009
GROUP  BY cn.name
ORDER  BY titles DESC
LIMIT  10;

• 2 Join trees: the ways 3 tables can be joined up, before any swapping of inputs.
• × 2² Orientations: each of the 2 joins can swap which input is outer and which is inner.
• × 3² Algorithms: each of the 2 joins picks hash, merge, or nested loop.
• × 4³ Scans: each of the 3 tables is either read sequentially or via index, index-only or bitmap scans.
• = 4,608

SELECT MIN(t.title) AS movie_title
FROM keyword AS k,
     movie_info AS mi,
     movie_keyword AS mk,
     title AS t
WHERE k.keyword LIKE '%sequel%'
  AND mi.info IN ('Bulgaria')
  AND t.production_year > 2010
  AND t.id = mi.movie_id
  AND t.id = mk.movie_id
  AND mk.movie_id = mi.movie_id
  AND k.id = mk.keyword_id;

• 8 Join trees: the ways 4 tables can be joined up, before any swapping of inputs.
• × 2³ Orientations: each of the 3 joins can swap which input is outer and which is inner.
• × 3³ Algorithms: each of the 3 joins picks hash, merge, or nested loop.
• × 4⁴ Scans: each of the 4 tables is either read sequentially or via index, index-only or bitmap scans.
• = 442,368

SELECT MIN(t.title) AS movie_title
FROM company_name AS cn,
     keyword AS k,
     movie_companies AS mc,
     movie_keyword AS mk,
     title AS t
WHERE cn.country_code ='[de]'
  AND k.keyword ='character-name-in-title'
  AND cn.id = mc.company_id
  AND mc.movie_id = t.id
  AND t.id = mk.movie_id
  AND mk.keyword_id = k.id
  AND mc.movie_id = mk.movie_id;

• 25 Join trees: the ways 5 tables can be joined up, before any swapping of inputs.
• × 2⁴ Orientations: each of the 4 joins can swap which input is outer and which is inner.
• × 3⁴ Algorithms: each of the 4 joins picks hash, merge, or nested loop.
• × 4⁵ Scans: each of the 5 tables is either read sequentially or via index, index-only or bitmap scans.
• = 33,177,600

SELECT MIN(lt.link) AS link_type,
       MIN(t1.title) AS first_movie,
       MIN(t2.title) AS second_movie
FROM keyword AS k,
     link_type AS lt,
     movie_keyword AS mk,
     movie_link AS ml,
     title AS t1,
     title AS t2
WHERE k.keyword ='10,000-mile-club'
  AND mk.keyword_id = k.id
  AND t1.id = mk.movie_id
  AND ml.movie_id = t1.id
  AND ml.linked_movie_id = t2.id
  AND lt.id = ml.link_type_id
  AND mk.movie_id = t1.id;

• 56 Join trees: the ways 6 tables can be joined up, before any swapping of inputs.
• × 2⁵ Orientations: each of the 5 joins can swap which input is outer and which is inner.
• × 3⁵ Algorithms: each of the 5 joins picks hash, merge, or nested loop.
• × 4⁶ Scans: each of the 6 tables is either read sequentially or via index, index-only or bitmap scans.
• = 1,783,627,776

SELECT MIN(a1.name) AS writer_pseudo_name,
       MIN(t.title) AS movie_title
FROM aka_name AS a1,
     cast_info AS ci,
     company_name AS cn,
     movie_companies AS mc,
     name AS n1,
     role_type AS rt,
     title AS t
WHERE cn.country_code ='[us]'
  AND rt.role ='writer'
  AND a1.person_id = n1.id
  AND n1.id = ci.person_id
  AND ci.movie_id = t.id
  AND t.id = mc.movie_id
  AND mc.company_id = cn.id
  AND ci.role_id = rt.id
  AND a1.person_id = ci.person_id
  AND ci.movie_id = mc.movie_id;

• 696 Join trees: the ways 7 tables can be joined up, before any swapping of inputs.
• × 2⁶ Orientations: each of the 6 joins can swap which input is outer and which is inner.
• × 3⁶ Algorithms: each of the 6 joins picks hash, merge, or nested loop.
• × 4⁷ Scans: each of the 7 tables is either read sequentially or via index, index-only or bitmap scans.
• = 532,030,685,184

SELECT MIN(an.name) AS cool_actor_pseudonym,
       MIN(t.title) AS series_named_after_char
FROM aka_name AS an,
     cast_info AS ci,
     company_name AS cn,
     keyword AS k,
     movie_companies AS mc,
     movie_keyword AS mk,
     name AS n,
     title AS t
WHERE cn.country_code ='[us]'
  AND k.keyword ='character-name-in-title'
  AND an.person_id = n.id
  AND n.id = ci.person_id
  AND ci.movie_id = t.id
  AND t.id = mk.movie_id
  AND mk.keyword_id = k.id
  AND t.id = mc.movie_id
  AND mc.company_id = cn.id
  AND an.person_id = ci.person_id
  AND ci.movie_id = mc.movie_id
  AND ci.movie_id = mk.movie_id
  AND mc.movie_id = mk.movie_id;

• 4,698 Join trees: the ways 8 tables can be joined up, before any swapping of inputs.
• × 2⁷ Orientations: each of the 7 joins can swap which input is outer and which is inner.
• × 3⁷ Algorithms: each of the 7 joins picks hash, merge, or nested loop.
• × 4⁸ Scans: each of the 8 tables is either read sequentially or via index, index-only or bitmap scans.
• = 86,188,970,999,808

SELECT MIN(cn.name) AS producing_company,
       MIN(miidx.info) AS rating,
       MIN(t.title) AS movie
FROM company_name AS cn,
     company_type AS ct,
     info_type AS it,
     info_type AS it2,
     kind_type AS kt,
     movie_companies AS mc,
     movie_info AS mi,
     movie_info_idx AS miidx,
     title AS t
WHERE cn.country_code ='[us]'
  AND ct.kind ='production companies'
  AND it.info ='rating'
  AND it2.info ='release dates'
  AND kt.kind ='movie'
  AND mi.movie_id = t.id
  AND it2.id = mi.info_type_id
  AND kt.id = t.kind_id
  AND mc.movie_id = t.id
  AND cn.id = mc.company_id
  AND ct.id = mc.company_type_id
  AND miidx.movie_id = t.id
  AND it.id = miidx.info_type_id
  AND mi.movie_id = miidx.movie_id
  AND mi.movie_id = mc.movie_id
  AND miidx.movie_id = mc.movie_id;

• 20,340 Join trees: the ways 9 tables can be joined up, before any swapping of inputs.
• × 2⁸ Orientations: each of the 8 joins can swap which input is outer and which is inner.
• × 3⁸ Algorithms: each of the 8 joins picks hash, merge, or nested loop.
• × 4⁹ Scans: each of the 9 tables is either read sequentially or via index, index-only or bitmap scans.
• = 8,955,727,561,359,360

SELECT MIN(n.name) AS voicing_actress,
       MIN(t.title) AS jap_engl_voiced_movie
FROM aka_name AS an,
     char_name AS chn,
     cast_info AS ci,
     company_name AS cn,
     info_type AS it,
     movie_companies AS mc,
     movie_info AS mi,
     name AS n,
     role_type AS rt,
     title AS t
WHERE ci.note IN ('(voice)',
                  '(voice: Japanese version)',
                  '(voice) (uncredited)',
                  '(voice) (English version)')
  AND cn.country_code ='[us]'
  AND it.info = 'release dates'
  AND n.gender ='f'
  AND rt.role ='actress'
  AND t.production_year > 2000
  AND t.id = mi.movie_id
  AND t.id = mc.movie_id
  AND t.id = ci.movie_id
  AND mc.movie_id = ci.movie_id
  AND mc.movie_id = mi.movie_id
  AND mi.movie_id = ci.movie_id
  AND cn.id = mc.company_id
  AND it.id = mi.info_type_id
  AND n.id = ci.person_id
  AND rt.id = ci.role_id
  AND n.id = an.person_id
  AND ci.person_id = an.person_id
  AND chn.id = ci.person_role_id;

• 242,160 Join trees: the ways 10 tables can be joined up, before any swapping of inputs.
• × 2⁹ Orientations: each of the 9 joins can swap which input is outer and which is inner.
• × 3⁹ Algorithms: each of the 9 joins picks hash, merge, or nested loop.
• × 4¹⁰ Scans: each of the 10 tables is either read sequentially or via index, index-only or bitmap scans.
• = 2,558,960,455,762,575,360

SELECT MIN(kt.kind) AS movie_kind,
       MIN(t.title) AS complete_us_internet_movie
FROM complete_cast AS cc,
     comp_cast_type AS cct1,
     company_name AS cn,
     company_type AS ct,
     info_type AS it1,
     keyword AS k,
     kind_type AS kt,
     movie_companies AS mc,
     movie_info AS mi,
     movie_keyword AS mk,
     title AS t
WHERE cct1.kind = 'complete+verified'
  AND cn.country_code = '[us]'
  AND it1.info = 'release dates'
  AND kt.kind IN ('movie')
  AND mi.note LIKE '%internet%'
  AND mi.info IS NOT NULL
  AND (mi.info LIKE 'USA:% 199%'
       OR mi.info LIKE 'USA:% 200%')
  AND t.production_year > 2000
  AND kt.id = t.kind_id
  AND t.id = mi.movie_id
  AND t.id = mk.movie_id
  AND t.id = mc.movie_id
  AND t.id = cc.movie_id
  AND mk.movie_id = mi.movie_id
  AND mk.movie_id = mc.movie_id
  AND mk.movie_id = cc.movie_id
  AND mi.movie_id = mc.movie_id
  AND mi.movie_id = cc.movie_id
  AND mc.movie_id = cc.movie_id
  AND k.id = mk.keyword_id
  AND it1.id = mi.info_type_id
  AND cn.id = mc.company_id
  AND ct.id = mc.company_type_id
  AND cct1.id = cc.status_id;

• 1,490,850 Join trees: the ways 11 tables can be joined up, before any swapping of inputs.
• × 2¹⁰ Orientations: each of the 10 joins can swap which input is outer and which is inner.
• × 3¹⁰ Algorithms: each of the 10 joins picks hash, merge, or nested loop.
• × 4¹¹ Scans: each of the 11 tables is either read sequentially or via index, index-only or bitmap scans.
• = 378,099,722,048,923,238,400

SELECT MIN(chn.name) AS character_name,
       MIN(mi_idx.info) AS rating,
       MIN(t.title) AS complete_hero_movie
FROM complete_cast AS cc,
     comp_cast_type AS cct1,
     comp_cast_type AS cct2,
     char_name AS chn,
     cast_info AS ci,
     info_type AS it2,
     keyword AS k,
     kind_type AS kt,
     movie_info_idx AS mi_idx,
     movie_keyword AS mk,
     name AS n,
     title AS t
WHERE cct1.kind = 'cast'
  AND cct2.kind LIKE '%complete%'
  AND chn.name IS NOT NULL
  AND (chn.name LIKE '%man%'
       OR chn.name LIKE '%Man%')
  AND it2.info = 'rating'
  AND k.keyword IN ('superhero',
                    'marvel-comics',
                    'based-on-comic',
                    'fight')
  AND kt.kind = 'movie'
  AND mi_idx.info > '8.0'
  AND t.production_year > 2005
  AND kt.id = t.kind_id
  AND t.id = mk.movie_id
  AND t.id = ci.movie_id
  AND t.id = cc.movie_id
  AND t.id = mi_idx.movie_id
  AND mk.movie_id = ci.movie_id
  AND mk.movie_id = cc.movie_id
  AND mk.movie_id = mi_idx.movie_id
  AND ci.movie_id = cc.movie_id
  AND ci.movie_id = mi_idx.movie_id
  AND cc.movie_id = mi_idx.movie_id
  AND chn.id = ci.person_role_id
  AND n.id = ci.person_id
  AND k.id = mk.keyword_id
  AND cct1.id = cc.subject_id
  AND cct2.id = cc.status_id
  AND it2.id = mi_idx.info_type_id;

• 11,932,560 Join trees: the ways 12 tables can be joined up, before any swapping of inputs.
• × 2¹¹ Orientations: each of the 11 joins can swap which input is outer and which is inner.
• × 3¹¹ Algorithms: each of the 11 joins picks hash, merge, or nested loop.
• × 4¹² Scans: each of the 12 tables is either read sequentially or via index, index-only or bitmap scans.

Estimating, not counting

Postgres is in a tough spot here. It would be reasonable to think it could simply count cardinalities and pick the plan that minimizes the number of rows passed through to successive joins.

But this would imply Postgres can count cardinalities during query planning. It can’t. In order to know this, it would need to actually run each join and count the resulting rows. This defeats the whole point of a fast query optimizer. A query optimizer does not aim to be exact in its cost minimization… it aims to be good enough across many types of queries.

Instead, Postgres uses statistics to estimate cardinalities. The planner queries the pg_statistic table, getting back common values for each column and their frequencies, and a histogram for the rest. Things get a bit more complicated when you tack on joins. Postgres doesn’t know how the rows in one table are distributed over the other. To get around this, it assumes that the frequency of a given value in the first table can simply be applied over the second table. This is the uniform distribution assumption I mentioned earlier.

Assuming a uniform distribution is fine as a heuristic, but when it fails, it fails hard. Looking back at an earlier join ordering, (cn′⋈mc)≈100k, then ⋈t′≈20k, we filtered 2m movie_companies entries on the assumption that 5% of them were from Japanese companies. But what if the 5% of companies that are Japanese were actually responsible for 50% of the movies? The first join would produce 1m rows! The cost model says pick the first join ordering; in reality, the second one is actually better since it only sends 400k rows through to the second join.

One bad estimate in an early join can cascade through the rest of the join tree, corrupting all other estimates.

How to steer an elephant

Postgres always picks the plan with the lowest cost, and we can’t change its cost model without modifying its source code, so how can we actually steer it to pick different plans that have higher costs?

Enter pg_hint_plan.

pg_hint_plan is a beautifully simple third-party extension: just by adding structured “hints” as comments above SQL statements, you can nudge Postgres towards plans that use the instructions provided in the hint. For example:

/*+
  HashJoin(a b)
  SeqScan(a)
*/
EXPLAIN
SELECT *
FROM pgbench_branches b
JOIN pgbench_accounts a ON b.bid = a.bid
ORDER BY a.aid;

The hint mandates usage of a HashJoin for joining pgbench_accounts and pgbench_branches, and doing a sequential scan of the pgbench_accounts table; the actual query plan follows suit nicely.

Formulating our problem

Given that we can influence Postgres to pick different—and potentially better—query plans using pg_hint_plan hints, the question we’re starting with is:

Can a language model learn to produce hints that result in better query plans?

Useful research

What might make this a worthwhile problem to solve?

My first idea was to give the model the query and the exact same set of information Postgres’s planner has. This amounts to seeing if we could build a better cardinality estimator. I came to the conclusion this is not a worthwhile avenue to explore; we would be fighting decades of cardinality estimation research. Furthermore, the inference latency alone would far outweigh any learned usefulness compared to Postgres’s ultra-fast query optimizer.

The second idea—and what I believe is the correct formulation—lies in a specific database usage pattern: heavy analytic workloads. If queries are getting run thousands of times using sub-optimal default Postgres plans, efficiency gains are being left on the table. Instead, a model could be trained to find a better way to run a specific query. The training process might require execution of that query tens to hundreds of times upfront, but the amortized cost across all runs of the query would be drastically lower.

The goal isn’t to try and beat Postgres on the time/efficiency Pareto frontier for one-off queries, but we may be able to beat it on queries that run over and over again.

A model and its harness

I decided to start with a small 4B model because it would be easiest to train/inference myself on the 2x RTX 3090 rig (affectionately named FLOPper) I have at home.

Around the time I started this project, the Qwen 3.8 family of models was released, unfortunately without a 4B variant. However, I came across a Qwen 3.8 4B distillation from a small lab in Germany called Empero and was intrigued. They used Qwen 3.8’s 2.4T model as a teacher model to distill learnings into Qwen 3.5 4B, producing empero-ai/Qwen3.8-4B-Distill. This distilled model is not outright better than its base 3.5 model; it performs better on MMLU tasks and slightly worse on GSM8K tasks. In other words, this distillation performs better when evaluated on breadth of general knowledge, and slightly worse on multi-step mathematical reasoning. As to which is better for our task, I do not know; I decided to stick with the distilled model either way.

With the model locked in, I built a lightweight agent harness, qo-agent, that would orchestrate hint production. It was given the following six tools:

1. inspect_relation — Lists a table’s columns with types and nullability, index definitions and estimated rows and bytes
2. get_column_stats — Gets Postgres planner statistics for 1-8 columns of a relation
3. get_plan — Gets the default plan’s estimates or a submitted candidate’s stored plan
4. evaluate_candidate — Validates a proposed plan action and then executes it for timing/plan diagnostics
5. keep_default — Returns Postgres’s default plan itself as the candidate and ends the search
6. finish — Takes as input a submitted candidate ID or the default plan and ends the search

To take advantage of structured outputs, the agent was instructed to produce PlanAction JSON objects. Calls to evaluate_candidate then compiled PlanAction objects into hints and prepended them to the original query.

A sample agent trajectory:

Agent → get_plan("default")
Tool  ← Default plan: (t ⋈ mc) ⋈ cn, hash joins, estimated rows per node
Agent → evaluate_candidate({
  "leading": { "left": { "left": "cn", "right": "mc" }, "right": "t" },
  "joins": [{ "relations": ["cn", "mc"], "force": "hash" }]
})
Tool  ← c1 · valid · novel plan · 118 ms · 0.91× default · 2 attempts left
Agent → evaluate_candidate({
  "leading": { "left": { "left": "t", "right": "mc" }, "right": "cn" },
  "joins": [{ "relations": ["t", "mc"], "force": "nestloop" }],
  "scans": [{ "relation": "mc", "force": "index" }]
})
Tool  ← c2 · valid · novel plan · 87 ms · 1.24× default · 1 attempt left
Agent → evaluate_candidate({
  "leading": { "left": { "left": "t", "right": "mc" }, "right": "cn" },
  "joins": [{ "relations": ["t", "mc"], "force": "hash" }]
})
Tool  ← c3 · valid · novel plan · 100 ms · 1.08× default · 0 attempts left
Agent → finish({ "selected_candidate_id": "c2" })
Tool  ← Finished · selected c2

A sample trajectory where the agent is permitted to submit up to three candidates.

Benchmarks

An agent is useless without something to benchmark its performance against. Fortunately for us, the hard work of creating these benchmarks was already done.

The Join Order Benchmark

Leis et al. introduced the Join Order Benchmark (JOB) in How Good Are Query Optimizers, Really?. They used it to evaluate cardinality estimation and join-order optimization using our familiar IMDb dataset.

It consists of 113 queries spread across 33 query templates. Query templates differ via their relational skeleton. They reference different tables and connect them with different join predicates. You can think about them as a structural family of questions that can be answered. Queries derived from templates preserve the tables used and the join graph topology but change selection predicates.

Looking at an example:

SELECT MIN(t.title) AS movie_title
FROM   company_name AS cn
JOIN   movie_companies AS mc ON mc.company_id = cn.id
JOIN   title AS t ON t.id = mc.movie_id
JOIN   movie_keyword AS mk ON mk.movie_id = t.id
JOIN   keyword AS k ON k.id = mk.keyword_id
WHERE  cn.country_code = :country_code
  AND  k.keyword = 'character-name-in-title';

Query template 2 — “What is the alphabetically first title of a movie associated with a company from country X and tagged with the keyword character-name-in-title?”

…and here are two real queries from JOB derived from this template:

SELECT MIN(t.title) AS movie_title
FROM   company_name AS cn,
       keyword AS k,
       movie_companies AS mc,
       movie_keyword AS mk,
       title AS t
WHERE  cn.country_code = '[de]'
  AND  k.keyword = 'character-name-in-title'
  AND  cn.id = mc.company_id
  AND  mc.movie_id = t.id
  AND  t.id = mk.movie_id
  AND  mk.keyword_id = k.id
  AND  mc.movie_id = mk.movie_id;

Query 2a — “What is the alphabetically first such movie title associated with a German company?”

SELECT MIN(t.title) AS movie_title
FROM   company_name AS cn,
       keyword AS k,
       movie_companies AS mc,
       movie_keyword AS mk,
       title AS t
WHERE  cn.country_code = '[us]'
  AND  k.keyword = 'character-name-in-title'
  AND  cn.id = mc.company_id
  AND  mc.movie_id = t.id
  AND  t.id = mk.movie_id
  AND  mk.keyword_id = k.id
  AND  mc.movie_id = mk.movie_id;

Query 2d — “What is the alphabetically first such movie title associated with a U.S. company?”

The Cardinality Estimation Benchmark

Another relevant benchmark is the Cardinality Estimation Benchmark (CEB), introduced in Flow-loss: Learning Cardinality Estimates That Matter. It uses the same IMDb database and is a much larger benchmark consisting of ~13.6k synthetically generated queries organized across 16 query templates. CEB’s definition of a template is looser than JOB's. Two CEB templates can share the same join graph, differing only in their selectivity predicates. In JOB, every template's join graph is unique.

Train time, test time

Due to its size, CEB was a good fit for training the model. JOB would be used to validate the model’s performance.

You might be wondering if it makes sense to both train and test on IMDb. If it works well, hasn’t the model just learned this specific database well?

I would argue this is precisely the point. We want our model to learn IMDb well. Given our problem formulation, if this agent is continually getting used for a company’s analytic workloads across its specific databases, we need not generalize to all databases.

The real issue is making sure we’re not overfitting to JOB query templates during training over CEB. The model should learn IMDb in a way where given any query, even for structural query families it hasn’t seen before, it’s still capable of producing a good plan. In practice, this means we need to prune CEB queries that have the same shape as any of the JOB queries.

Query topology mapping

Let’s define a query’s “topology” as its structural join-graph (de-aliased table names as nodes and joins as edges). The join graph excludes all selectivity predicates; we’re only interested in joins here.

CEB queries sharing a topology with a JOB query would be removed from the training set. I wrote a small script to convert all JOB and CEB queries to their topologies and checked if there was any overlap. There wasn’t, so no filtering was required.

How to muffle an elephant

Before getting into benchmarking the agent and doing training runs, we have to talk about how Postgres was actually run, because it directly impacts the training process.

First, some facts:

• FLOPper has a CPU with 16 physical cores, 64 GB of RAM and a 2 TB NVMe SSD
• The slice of IMDb we’re using is 8.5 GB on disk
• Postgres caches pages of data retrieved during query execution into a buffer
• The operating system has its own filesystem cache doing the same thing one level down

If we run the exact same query on Postgres 20 times in a row, it won’t take the same amount of time each run. In day-to-day work, this isn’t a big deal. But the whole thesis, and the training process itself, relies on measuring whether one way of running a query is faster than the Postgres default. This means we need to do everything in our power to de-noise Postgres.

First, I needed to understand just how noisy Postgres query executions are.

I started by building a “calibration” capability into my experimentation workflow. The calibration process was simple: run N Docker containers built from a Postgres image, each given a fixed slice of CPU cores and RAM to use. I set N=4 to begin; anything lower might make future training far too slow, and anything higher might lead to more CPU contention, which means more noise. Each container was given 4 cores to use and capped at 8 GB of memory.

On startup, each container initialized Postgres with identical settings and loaded the IMDb data. Calibration then opened a thread pool of size four and pushed all 113 queries onto a shared queue. Whenever a container finished measuring a query, it pulled the next one off the queue.

The actual measurement process had two phases:

1. Run the query a few times to “warm it up”
2. Then run the query 20 more times and record each execution time

So what does it mean to warm a query up? We need to bust out some OS fundamentals to understand.

Whenever Postgres executes a query, it asks the operating system (in our case, Linux) for pages of data. Linux first checks its own filesystem cache, the page cache. If the pages are present, Linux sends them over; else it reads them from disk, stores them in its cache and then sends them over. Postgres, in turn, keeps received pages in its own shared_buffers cache for easy reuse. When shared_buffers begins to overflow, Postgres evicts pages. If it needs those pages again, it must ask Linux once more.

Every time there’s a cache hit in shared_buffers for a page, Postgres increments a counter called “shared hit blocks” (SHBs). If it has to ask Linux, it increments “shared read blocks” (SRBs).

Postgres conveniently reports both counters if we run EXPLAIN with the BUFFERS option. For example, running EXPLAIN (ANALYZE, TIMING OFF, BUFFERS, FORMAT JSON) outputs something like:

{
  "Plan": {
    "Node Type": "Aggregate",
    "Shared Hit Blocks": 1800786,
    "Shared Read Blocks": 52990
  },
  "Execution Time": 189.2
}

These counters give us some notion of the “warmness” of a query. After each warmup run, we compared its hit and read counts to the previous run’s. If both were within 2% of each other (and the plan hadn’t changed), we called the query warm and started measuring. A query needed at least two warmups to have something to compare, and was cut off at five regardless. The idea was that if the counters stopped moving, the data could be considered settled and cache churn would be minimized during the 20 measurements.

I set shared_buffers to a conservative 128 MB and ran the first calibration.

Half the queries were declared warm after only two runs. Not bad… at least until I dug deeper. The SRB counts weren’t dropping to zero; rather, they were hovering steady at some large number. With only 128 MB of shared_buffers against an 8.5 GB database, Postgres was consistently missing its own cache on every execution and asking Linux for more pages. “Stable” did not mean “resident.”

Linux’s page cache is fast, so this isn’t the end of the world. Unfortunately, a new problem emerged when I actually looked at the 20 measurements taken for various queries. Let’s look at one query in particular, job-13b.

14 of the 20 landed between 186 and 204 ms. The other 6 landed between 227 and 253 ms, somewhere between 14% and 26% slower. The query wasn’t even uniformly noisy, it just had two different speeds at different times, and a third of the time it ran at the slower speed.

I initially wanted to quantify noise using the coefficient of variation:

The CV tells us the “wobble” of a measurement. If a query takes 100 ms and has a CV of 5%, we could say it wobbles by about 5 ms. For job-13b, the CV was 10.3%. It wasn’t great. CV is also not a great measurement to use here. Because it’s built on the mean, it’s easily influenced by a few outlier runs.

We don’t actually care as much about how spread out the 20 runs are. We do care about how often this causes our measurement criteria during training runs to get fooled.

To fool an agent

Bear with me here as I skip ahead a little bit in order to provide more color on what exactly we needed to measure.

To de-noise during actual agent runs, I couldn’t just run the agent’s proposed plan a single time. Instead, I ran three interleaved (candidate, default) pairs sequentially. Three was picked somewhat arbitrarily to provide some measure of variability while being small enough to prevent agent evaluation runs from spending most of their time in Postgres. Once the three candidate/default execution time tuples were obtained, the medians of both the three candidates and the three defaults were taken and expressed as a ratio of each other to determine the final speedup or slowdown. If the two medians differed by less than an arbitrarily declared 5%, it was a tie. Outside of that tie zone, a candidate could be declared as a speedup or a slowdown.

Now let’s go back to our earlier job-13b example. We had 14 executions in one clump, and 6 in another slower clump. The median of three strategy sounds good until you realize that if, in theory, at least two of the three measurements landed in that “slower” clump, the median would bias towards the less frequent slower clump.

Imagine a candidate plan that executes identically to the default. No real difference exists, so the correct reward is zero. Draw three timings for the “candidate” and three for the “default” out of the 20 we observed. There are 1,140 ways to draw three from 20; for job-13b, 230 of them contain at least two slow runs, so one side’s median lands in the slow clump ~20% of the time.

That’s a totally phantom 14-26% speedup or slowdown that we would show to our model as signal ~20% of the time. Dangerous!

So we can’t just rely on CV as the golden number to minimize, as two queries with the exact same CV can fool the measurement reward at different rates depending on whether the spreads are a uniform blur or two clumps sitting more than 5% apart. The actual number to minimize is this fooling rate itself.

I wrote a small script to compute the fooling rate directly from raw calibration data. It worked by sliding a window of six sequential runs across the 20. For each window, we took interleaved pairs of size two to represent an interleaved (candidate, default) pair. A window of size six gives us pairings like: (t1, t2), (t3, t4), (t5, t6). In any given pair, t_n and t_{n+1} can alternate roles of being the candidate query, or the default query. That means for each pair, there are two possibilities, and therefore for each window of three tuples, there are 8 possibilities. 20 measurements means we’ll slide this window 15 times, so we have 120 total possibilities for a given query.

We derive two metrics from these raw numbers. First, we calculate the no-op error rate for a given query as the ratio of the 120 simulated possibilities that do differ by more than 5% against the number that don’t. We sum these percentages up across all 113 JOB queries and then divide by 113. This number, which we’ll call the “mean no-op error rate,” gives us the percentage likelihood that the reward may get fooled for any JOB query when doing our three paired measurements strategy. Second, we sort the no-op error rates for all 113 queries, lowest to highest. The number that is 90% of the way to the end of this sorted list is reported as the “p90 query,” and gives us a measure of the fooling rate for the worst-offending queries.

At 128 MB for shared_buffers and four concurrent containers, the “fool rate” script produced the following mean no-op error rates and p90 query numbers:

The numbers aren’t good. One in twenty no-op plans get rewarded, and one in ~10 queries gets fooled more than 13% of the time.

We can do better.

Tuning Postgres

I focused on two memory-related settings Postgres exposes:

1. shared_buffers decides how much of the database Postgres can keep in its own cache
2. work_mem decides how much memory a single sort/hash operation can get before spilling to disk

I ran four calibrations:

Surprisingly, work_mem had no effect on noise at all, and shared_buffers carried all of the weight!

With 2 GB of shared_buffers, the median query ended warmup with its SRB counter at exactly zero: its working set was fully resident in Postgres’s own cache. The no-op error rate dropped by roughly 4x, and the 90th percentile query went from being fooled 13%–20% of the time to almost never. Our two-clump query, job-13b, went from a CV of 10.3% to 0.9%, with all 20 runs landing within 7 ms of each other.

One neat benefit emerged that I wasn’t initially chasing: the default plans themselves got faster. The summed runtime of all 113 JOB queries fell from 95 seconds to 60 seconds, just from cache residency. In other words, actually taking our measurements for both candidates and defaults would now be significantly faster, meaning the training process would take less time.

I locked in 2 GB shared_buffers and 4 MB work_mem for the rest of the project.

Baselines and metrics

I used two metrics for benchmarking agent performance.

Geometric mean speedup

The geometric mean speedup gives all queries equal weight. For example, in a two-query sample, if query 1 runs 2x faster than its baseline, and query 2 runs 0.5x faster than its baseline, then S_geo=1.00x. It doesn’t matter if query 1’s baseline took 5 minutes and our candidate took 2.5 minutes, but query 2 only regressed from 25s to 50s, as they are equally weighted.

Total workload speedup

Total workload speedup treats the entire query set as one batch. We simply add all the baseline times and divide by the sum of the candidate times. In our above example, S_workload=1.4x.

Both metrics tell different stories. The total workload speedup is a measure of practicality. A data analyst building out a suite of analytics queries wants to decrease the overall runtime across the batch. But from a model training standpoint, the total workload speedup could be entirely influenced by a single query plan the agent chanced upon; the rest of the batch could be degenerate. This implies the model hasn’t actually learned anything interesting; it just got lucky. Because the geometric mean speedup cares not for absolutes, it gives us a measure of actual learning across the batch: values above 1x imply that the average query is executing faster.

A frontier intelligence control

Before running the untrained 4B model through the qo-agent harness, I wanted to validate this problem was actually solveable by today’s frontier models. If a model like GPT-6 Astra or Qwen 3.8 2.4T couldn’t improve upon the default Postgres query plan, I couldn’t really expect the 4B model to either.

I took a small sample of 10 JOB queries and benchmarked them on both Astra and Qwen 3.8 2.4T running through the qo-agent harness.

Given the difference between the single-candidate scores and the 5-candidate scores, the agent was clearly capable of doing in-context learning across sequential executions of its candidates. This gave me the confidence to stick with an agentic multi-turn approach rather than try and train the 4B model to get really good at one-shotting a plan.

During a run of the agent, each candidate was warmed once and then measured once. After exhausting the candidate attempts budget, the model was only presented with a single tool to call, finish, and the model was told to select the best scoring candidate (or keep the default plan). After the candidate was selected, three interleaved (candidate, default) pairs were run and passed through a clipper:

S_i = clip(median(D_i) / median(C_i), 0.1, 10)

The clipper constrained the result of the division between the two medians to be between [0.1,10]. These clipper values were picked somewhat arbitrarily; I found they prevented the geometric mean speedup from getting overly influenced by an extreme speedup or an extreme regression.

Conclusion: frontier intelligence is capable of agentically doing query optimization.

The vanilla 4B baseline

We’re now ready to evaluate the untrained 4B model on JOB and see how it does!

The same 5-candidate plan budget per agent trajectory configuration was employed. The results were dismal:

Only the last three rows contribute to the score, leaving 15 valid trajectories out of 113. Nine of those 15 result in a score of 1.00x by construction (the plan was identical to the default, or the model chose to keep the default). That left just six candidate plans that were:

1. Structurally intact
2. Valid
3. Novel

Five of the six plans resulted in speedups of 1.02x–1.30x, and one of them landed at 0.05x.

The most common issues seen were:

• The PlanAction was not a valid object
• Join trees did not contain every relation exactly once
• Actions were wrapped in an extraneous action key
• Leading trees had subtrees that were not actually connected in the query’s join graph
• The model called specific tools at the wrong time, or called tools that didn’t exist

Not only was the model terrible at this task, it couldn’t even grok the harness wrapped around it either.

Off-policy distillation via supervised fine-tuning

I first needed to get the 4B model to speak the “language” of the qo-agent harness. I would make it good at query optimization after.

We can use supervised fine-tuning (SFT) to do this. Specifically, we can do off-policy distillation.

Off-policy distillation is a training method by which a student model (sometimes referred to as a policy) learns to imitate outputs produced by a teacher model. It’s called “off-policy” because the training data is not generated by the student model/policy itself. It’s remarkably simple. A complete teacher trajectory (sometimes referred to as a demonstration) is shown to the student model. For every token in the trajectory, the probability the student model gave to generating that token results in a per-token loss. Averaging these per-token losses leads to a demonstration-level loss. Standard backpropagation via chain rule then lets you compute gradients for all trainable parameters in the student model, and the configured optimizer can nudge parameter values in a way where loss gets minimized in a single pass.

What’s super neat about off-policy distillation is that we don’t need a lot of data for it to work well. Because each demonstration provides us thousands to tens of thousands of token predictions, our student model’s weights adapt quickly.

How do we actually get these demonstrations though? We could write them all by hand, but that would take far too long. One step up would be writing a tool to randomly generate valid-looking trajectories. But these ignore that we have the best teacher of all already available: smarter, larger models.

The strategy is simple: generate a bunch of trajectories by running a smart model through the qo-agent harness, and distill those trajectories into our student 4B model.

Rendering, loss masking, and unrolling

There are a few complexities to unpack.

Imagine we decide to use GPT-6 Astra as our teacher model. It produces trajectories in OpenAI’s Responses API format. Our Qwen model doesn’t understand this format; we need to transpile the human-friendly Responses API JSON format into a model-friendly token format. This is where the concept of rendering comes in. Rendering libraries can take in a trajectory’s text and convert it to a raw sequence of tokens a specific model actually understands.

Another consideration with agent trajectories is determining which tokens our model should actually be predicting. The model never produces certain tokens in a trajectory, like the system prompt, any user prompts or the results of a tool call after a harness executes it. The model should still see these tokens when predicting the next token though; they’re still part of the context, but we should only compute losses for tokens the model is responsible for predicting. We can employ a strategy called loss-masking here. A loss-masking library lets us label the parts of a teacher trajectory that are context-only, versus the parts our student model is responsible for predicting.

Finally, when fine-tuning over agent trajectories, we generally don’t include the entire trajectory as a single trainable unit. Instead, the trajectory is broken up into a set of (context, reply) pairs. The context in these pairs is additive and includes previous model replies.

One trajectory with three assistant replies becomes three training examples. Each example includes as context everything preceding an assistant reply, including the model's earlier responses.

Low-rank adaptation (LoRA)

Our 4B model has 4.66 billion tunable parameters. If we wanted to update all of these parameters in a single pass during SFT, we would need a GPU with at least 64 GB of VRAM. I wanted to test my hypotheses first on my consumer-grade RTX 3090s, each of which carries 24 GB of VRAM, so I needed something more parameter-efficient here.

Low-rank adapters, or LoRAs, are the canonical way to do this. At a high-level, they work by freezing the model’s weights as they are, instead letting you train a much smaller pair of matrices that when multiplied together, result in an adjustment to selected weights in your original model.

4.66 billion weights come in at 9.32 GB in bf16. The tiny LoRA I actually ended up training was only 42.5 MB and contained only 21.2 million trainable parameters.

Teacher model selection

We previously learned that frontier models could operate well in qo-agent; the next decision was picking between GPT-6 Astra or Qwen 3.8 2.4T.

It seems like Astra wins on all fronts. However, Astra’s biggest drawback is that when run through the API, reasoning tokens aren’t supplied. Instead, the API gives you a short summary in place of reasoning tokens. On the other hand, Qwen 3.8 2.4T is an open-weights model and is happy to provide all of its reasoning tokens.

I was wary about training off of trajectories containing only reasoning summaries. The How to Steal Reasoning Without Reasoning Traces paper talked about this exact thing: training off reasoning summaries resulted in the student model’s performance decreasing. To combat this, the authors devised a new method: trace inversion. Trace inversion calls for synthetically expanding a reasoning summary into what the raw reasoning tokens may have looked like. Although they’re not exactly the ones Astra actually produced during inference, the longer reasoning blocks led to improved performance when transferred over to a smaller model. This provided some level of comfort; if I picked Astra and performance suffered, I could experiment with trace inversion.

The other consideration was context lengths. Given I was only doing SFT off a single RTX 3090 to start, I needed any given trajectory to not exceed ~50k tokens in sequence length. If it did, the training process might OOM given the 3090’s limited 24 GB of VRAM. All Astra trajectories fit under that budget, but some of the Qwen 3.8 2.4T ones didn’t.

I decided to try out SFT on the Astra traces. If performance suffered, I could explore trace inversion; if that didn’t work, I could rent beefier GPUs for training, bump our global context limit, and use Qwen 3.8 2.4T traces instead.

I began by generating 120 Astra trajectories over a random slice of queries from CEB, with reasoning summaries enabled. 100 trajectories would be used for training, and 20 would be used for a held-out validation set. All trajectories were rendered via Prime Intellect’s renderers library into Qwen format, loss-masked appropriately and unrolled and packed into usable training demonstrations. I then used Prime Intellect’s prime-rl library to run SFT, training the smaller ~21 million parameter LoRA. 100 Astra trajectories became 382 training rows after unrolling and packing. I allowed training to run for just a single epoch, meaning every example was trained on exactly once, and used a batch size of one (each demonstration updated the weights). Finally, I evaluated the resulting LoRA over JOB:

The one-epoch adapter against the untrained model on all 113 JOB queries. A query counts as scored when its trajectory ended with a measured candidate, a duplicate of the default plan, or the default itself. Wins and regressions are scored queries more than 5% faster or slower than the default.

Promising! The adapter learned the harness.

I could now either generate more fresh trajectories, or do more epochs over the dataset we already had. Given the latter is cheaper, I decided on more epochs.

It was around this point that I got impatient and wanted the training process to run even faster, so I rented a 2x H100 node on Lambda.

Now training on an H100 instead of a single RTX 3090, I did two more training runs over the existing LoRA; a second epoch and then a third:

Two and three epochs over the same 100 Astra trajectories, evaluated on JOB. Before epochs two and three, the qo-agent harness was upgraded to allow the model to keep the default plan after a search, which is why many more queries are scored. The two- and three-epoch rows were evaluated under identical settings.

Two epochs improved our results, but three epochs regressed them! This was especially interesting because validation loss didn’t budge at all during the second epoch:

A very good lesson that a flat validation loss doesn’t necessarily mean the model has stopped learning useful behavior.

I still felt we had more to learn from SFT though before proceeding with RL. I did zero filtering on the training trajectory dataset, and hadn’t carefully audited if I was missing any capabilities. It turns out I was, mostly around the model’s ability to construct valid Leading trees.

I generated another 320 Astra trajectories; 300 for training, and 20 for validation. I filtered out just six training trajectories where Astra opted to keep the default plan without even trying a single candidate. I did two more epochs in two separate runs:

Continuing the two-epoch adapter on the 300 new trajectories, evaluated on JOB.

Our 4B model not only learned the harness; it was now genuinely making good calls on various JOB queries! Fortunately, training on reasoning summaries didn’t harm performance.

Making the 4B model good at query optimization

The model now spoke the “language” of the qo-agent harness, and we got some free performance gains out of SFT too. It was time to make it very good at query optimization.

Agentic reinforcement learning

Agentic RL differs from SFT in that we actually run the current policy over training queries inside the agent harness N times. Each run (also referred to as a rollout) results in a final output that’s scored against some verifiable criteria. Lastly, each rollout’s score is then weighted relative to the other same-query rollouts. A positive “advantage” is reinforced by making the model’s weights more likely to produce that trajectory in future runs, and a negative advantage is penalized; the weights are updated to be less likely to produce that trajectory in future runs.

Designing per-rollout rewards and relative advantages

The initial reward algorithm was simple:

The speedup was calculated as the default plan’s median execution time divided by the selected candidate plan’s median execution time.

For every evaluated candidate that was invalid, we subtracted 0.1 from the natural log of the speedup ratio. We subtracted a further 0.05 if the rollout resulted in a plan that shared the same fingerprint as the default Postgres plan. Finally, if the trajectory ended with no valid candidate at all, a flat 3 was subtracted from the reward in lieu of any of the 0.1 or 0.05 subtractions.

The first few RL runs I did using this reward resulted in a model that was terrified of producing invalid plans due to the extremely harsh -3 condition. The model played it safe instead, returning Postgres’s default plan over and over again, accepting the smaller 0.05 reward hits.

GRPO exacerbated this issue. The plain GRPO algorithm converts multiple rollout rewards into relative “advantages”:

A_i = r_i - (1/G) Σ_{j=1}^G r_j

GRPO as prime-rl implements it: simply subtract the group’s mean reward from each rollout’s reward. The GRPO paper also divides by the group’s standard deviation.

Let’s say we perform four rollouts for a given query resulting in the following plans, execution speeds and rewards:

Four rollouts of one query under the first reward and prime-rl GRPO. Nothing in the group beat Postgres, but two rollouts still receive positive advantage because they beat the group's mean.

We’re reinforcing bad behavior by telling the model it’s okay to produce plans that end up being equivalent to Postgres’s default plan!

Both the busted reward algorithm and GRPO needed to be swapped out for something that could actually score advantages relative to the default plan’s execution time.

The reward was updated as follows:

• Anything within 5% of the default execution time counts as zero, and real gains/losses are shrunk by 0.05.
• The speedup is clipped to [0.1x, 10x] before the log, so one extreme plan cannot dominate a group.
• If a candidate plan times out, rollout i is scored as if it took exactly the duration of the timeout amount. Because this is generous, we tack on an additional 0.1 fee.
• If the rollout kept or finished with the default, the quality is exactly zero.
• Trajectories ending without valid candidates have no quality value.

Open answers to a reader's letter.

A reader recently wrote me a long letter with lots of questions about learning programming in this age of LLMs. After a bit of back-and-forth, I got permission to quote extensively from the letter in order to attempt some answers in public.

None of my answers I consider particularly rigorous; the situation is so uncertain that I can only answer to the best of my abilities, but I don't claim them to hold any kind of immutable truth.

"I'm trying to understand how people who deeply understand software think about learning and competence in the age of AI. I'm approaching it almost as a historian would: asking people directly how they make sense of a technological transition while actually living through it.

"About a year ago I became fascinated by AI-assisted programming. Despite having no formal CS background, with LLMs I managed to build a fairly large TypeScript/JavaScript system involving APIs, PostgreSQL, LLM pipelines, research automation and multi-model workflows. At first it felt almost magical: AI seemed to collapse the distance between having an idea and being able to build it.

"But now I'm trying to turn that system into a real production product, and I'm struggling. I fix one error with AI, then another appears, then another part behaves in a way I don't fully understand. After months of refactoring I had an uncomfortable realization: I may have built a system that is above my own level of understanding. When everything works, that gap is almost invisible. When it doesn't, it becomes very real.

"Sometimes I genuinely don't know what to do next without asking another model. That made me wonder whether I spent a year building a product, or partly building the appearance of one: something sophisticated enough to work, but which I don't yet understand deeply enough to truly own.

"I'm not anti-AI at all. I'm fascinated by these systems and want to work with them professionally. But I'm unsure what the right relationship with them should be."

Indeed, I'm not sure either, but before proceeding, I find it most transparent to reveal my position. I haven't yet decided on AI, but I lean toward disliking it, knowing full well that it may be unstoppable.

I do work and experiment with it, and it often impresses me. At other times, it frustrates me. It's usually when it impresses me the most that I resent it maximally.

When it's bad, it can be frustrating, but then at least I can absorb an ember of warmth in the illusion that what I've spent more than thirty years learning is still relevant. When it's at its best, I sometimes think: Where do I sign up for the Butlerian jihad?

My position on LLMs is only partly based on my own socio-economic status. I'm old enough, and have had enough success already, that all other things being equal, I can survive unemployment. I'm not sure, on the other hand, than any knowledge-based society can.

It may be that LLMs will take programmer jobs before they take other white-collar jobs. After all, programming may be a discipline where verification is easier than, say, insurance claims management. Still, if we reach a point of mass unemployment among knowledge workers, I'm not sure society as we know it will survive.

I usually don't talk much about my background as an economist, but in this context I find it relevant to mention. As an economist, I can't imagine that mass unemployment of 30-40% will not have a significant impact on the economy.

I'm painfully aware of the arguments that this has happened before: There may be job loss, but the advance of technology leads to new jobs we can't even imagine today. It was like that with the introduction of the stocking frame, the steam engine, the internal combustion engine, computers, etc. This is only partly true: Yes, new jobs were created, but often not for those people who lost their jobs. Coal miners didn't just become programmers overnight.

The same kind of argument was used when China was admitted to the World Trade Organization. And indeed, lots of new jobs were created, just not in the Western world.

So, based on lived and historical experience, I'm sceptical of arguments that all will be fine.

But I sincerely hope that I'm wrong. I love to program, and wouldn't mind doing it for another ten years. Perhaps more importantly, I have young adult children. I hope that there's a world for them, too.

"So I'd really like to know how you think about this. Are you glad you learned programming fundamentals before LLMs existed? If you were starting today, would you still seriously study languages, data structures, databases, networking, operating systems, debugging and architecture? Do you think AI can let people become capable of building much faster than they become capable of understanding?"

Am I glad that I learned programming before LLMs? Yes, of course. Those skills served me well for thirty years.

If I was starting today, I'd seriously consider learning carpentry, metalworking, gun-smithing, or something else that requires hand-eye coordination. I know that advances are made in robotics, too, but replacement of manual labour seems to lie farther in the future.

But to address the question: I am, personally, currently learning data structures, language semantics, etc. as part of a university programme. I do that because I'm curious, however, and not because I expect to get much monetary reward out of it.

Do I think that AI enables people to develop faster than they can keep up? This remains to be seen. Software developers have already, for decades, been working on top of abstractions they didn't understand. If you were a web developer, you didn't know much about compiler programming. If you were a compiler programmer, you didn't know much about integrated circuit design. And if your job was to engineer integrated circuits, you wouldn't know much about the levels of abstraction above you.

A good rule of thumb was: Understand the level of abstractions directly below the one you work in, as well as the one above. That would enable you to troubleshoot most problems.

"And how do you personally deal with that? When AI can solve something immediately, how do you decide when to use it and when to work through the problem yourself? If you were in my position, with a substantial AI-built project but weak foundations underneath it, would you step back and systematically learn those foundations, keep building and learn as problems appear, or combine the two?"

That's two radically different questions, because I no longer have a weak foundation in software development. Even if I were dealing with something far from what I usually do, I can ramp up leveraging what I already know. Let's imagine that someone tasked me with maintaining an application written exclusively in RISC-V assembly code. That's the most alien software environment I can imagine for myself. Adapting to such a development environment would be difficult for me, but still not as difficult as it would be for someone new to programming in general. Believe it or not, I have written small exercise programs in RISC-V, as well as an exercise compiler that compiled to RISC-V.

But what if I had virtually no software background?

Well, once upon a time, I was in exactly that situation. When I started my career, for years I balanced a knife's edge of getting things done while learning on the job. Beginning in 1999, I wrote COM components in C++, not understanding much of what I was doing. Somehow, I still made it work, even to a degree that I managed to eliminate any obvious memory leaks.

I was, however, never happy just slapping things together without understanding how they worked. So I did, as suggested by the question, step back to systematically learn fundamentals. This worked well for a career launched in the mid 1990s. Will it work well today?

I'm not so sure: Reaching a level of competency high enough to recognize your past confidence as clearly lying on the too-ignorant-to-realize-it portion of the Dunning-Kruger curve took decades. Do you have that much time today?

Granted, with LLMs, you can learn faster, because you can ask more directed questions. Thirty years ago, I would buy books in the hope that they would contain some helpful material. This still meant slogging through a lot of learning material not immediately relevant to the task at hand.

Still, I doubt that it's possible to significantly speed up human learning. The bottleneck is hardly the teachers nor the materials, but how fast a human brain can absorb new knowledge.

"One last thing I would be especially grateful to hear about is how you learned programming yourself, and how you learn new technical things today. How did you approach learning a new language earlier in your career? Books, projects, reading other people's code, exercises, debugging, something else? And if you had to learn a completely new programming language today, with AI available, how would you do it?"

The short answer to the first question: Slowly, based on much trial and error, occasionally backed by a book.

Apart from a very early false start with COMAL 80, my first programming projects was to (re)calculate bifurcation diagrams and the Lorenz attractor for my master's thesis in economics. Reaching for what I had, I wrote them in QBasic, learning from the samples that shipped with it, as well as occasionally asking a friend.

While I'm glossing over many details, in the 1990s and 2000s, I mostly learned from examples and documentation. While I did buy a book about C++, I don't think I ever finished it, and I picked up various Basic dialects as well as C# exclusively from documentation and example code.

That said, although I never read a book to learn C#, books were instrumental in teaching me both F# and Haskell. I have, over the years, relied heavily on books to educate myself, but as my Goodreads profile reveals, I love books in general.

How do I learn a completely new programming language today? Again, my experience is useless to someone new to programming in 2026: I've now seen so many programming languages that if I run into a new one, I can usually pick it up from perusing existing code and looking up the few things that aren't immediately clear.

But that's presupposing that the language in question is 'normal'. If I had to get back into APL, I'd at least have to find a tutorial.

You may have noticed that I don't much use LLMs for learning. LLMs don't hallucinate; they bullshit, and I'm deeply distrustful of anything they tell me. This is not to say that I don't use LLMs, but I tend to ask them questions that yield verifiable answers. Can I make this Haskell expression more succinct? Any useful answer to such a question is a code suggestion that either works, or doesn't work; is shorter, or isn't. That's easy to verify.

What should I learn next? does, on the other hand, not yield a verifiable answer. I tend to not to ask such questions of LLMs.

In conclusion, you could say that I prefer asking LLMs falsifiable questions.

Day to day, I think a surprising amount of engineering productivity comes from small nuggets of knowledge: being aware that a language feature exists; knowing that an unexplained tcp delay is probably related to the TCP_NO_DELAY setting and Nagle’s algorithm; knowing the right git incantation to get out of a pickle; or knowing a trick with sed to rewrite a file.

In one sense, this is self-evident: anything you know is going to be made up of smaller pieces of knowledge. Of course those smaller pieces of knowledge matter.

But I think there are some nuggets of knowledge that are particularly valuable and don’t require a lot of supporting mental infrastructure. You don’t need to know any python to use python3 -m http.server to start a simple server in a directory, but it might still make your work marginally easier. Let me share a few examples:

• You probably know that ctrl + r allows searching your terminal’s command history, but if you install fzf, you can set it up so that ctrl + r does a fuzzy search. If you want even more power, atuin replaces your shell history with a searchable SQLite database. per-directory-history lets you switch back and forth between searching for commands that have been run in a specific directory or searching all previous commands. Finally, you can configure how much history to store: stackoverflow question.

• You can SELECT without a FROM. This can be useful for testing out how a function in your database actually works or reminding yourself how SELECT TRUE <> NULL works.1

• PostgresSQL and MySQL both support explain analyze which will actually run the query you’re trying to optimize and give you a ton more information about its performance.

• In regular expressions, \b, the word boundary assertion, makes it easy to look for the beginnings or ends of words.

• You can use logarithms with metrics to get a sense of the distribution of values for a field you’re interested in:

  const bucket = Math.floor(Math.log10(userInGroupCount))
  metrics.increment("my_metric", { bucket });
  

• Modern JS now supports Array.flatMap, Object.entries, and Promise.withResolvers.

• In NodeJS, you can keep a connection open to an external resource by creating an https.Agent and then providing it to your http requests: fetch(url, {method, agent}). This can have a dramatic impact on latency.

• git log -S pattern (”git pickaxe”) can give you all commits that added or removed a string in a codebase. It’s amazingly useful especially with older codebases! (git log -G pattern is similar, but will also show when that line was moved)

• Similar to cd -, you can use git checkout - to check out your previous HEAD.

• You probably don’t need find. A lot of find commands can be replaced with globs like **/*.md. Most shells support this out of the box, but with bash, you need to turn this on with shopt -s globstar.

• In a similar vein, most folks will probably want to use rg (ripgrep) rather than grep, ack, or ag.

• zsh’s advanced autocompletion features aren’t turned on by default:

  if type brew &>/dev/null; then
      FPATH="$(brew --prefix)/share/zsh/site-functions:${FPATH}"
  fi
  autoload -Uz compinit
  compinit
  

You might have already known all of these things! Or you might work in a domain that makes all of these little tricks totally useless. Even if this particular set of tricks isn’t useful for you, I bet you have your own stash of tricks that you’ve accumulated over the years that makes your work easier.

At a company, I think even more knowledge tends to be this sort of small high-leverage nugget:

• To debug $PROBLEM, use $DATA_SOURCE.
• $PERSON knows a ton about $AREA and they’re happy to help if you get stuck
• There are good docs about $HARD_THING $OVER_HERE.
• When $THING happens, it means we should manually scale out.
• To do a rolling restart of a service, run $THIS_COMMAND.
• This $UTIL makes $THAT_PROBLEM easy to script.

At a previous company, I shared a trick on slack every day with the engineering team, both technical and company-specific, and folks found them pretty useful. Even if you knew 9/10 tricks, that 10th doc or technique might save you some time! And one trick per day was the right number to avoid overwhelming people with knowledge, and it could occasionally spark useful discussion. If you’re a more senior engineer at your company, you might think about doing something similar.

1. I think I first saw this technique on Julia Evan’s blog, and I think her writing often perfectly encapsulates the idea behind this post: "small bits of knowledge are powerful! and fun! and approachable!!" ↩︎

I find it very useful to be able to do everyone else’s job — it helps you learn how they work and what they need, and it helps even more when you need their job done, but they won’t do it themselves.

If you made something new and those you made it for can’t be bothered to start using it, go ahead and integrate it into their system. It worked for Intel back when they made their first 32-bit CPU and had their people add support for that CPU in Microsoft’s compiler — I’ve met someone from that Intel team. For some reason, many resent the idea of working on someone else’s system, and think of it as charity uncalled for in the workplace. Well, did Intel do Microsoft’s job to help poor struggling Microsoft, or did they do it to benefit from Microsoft’s success?

If a manager doesn’t actually manage anything, this is fine as long as you can go to his people and effectively manage them — without calling it that and without taking credit for it, of course, but you can usually find people in his org who’ll work with you, on the theory that they’re supposed to. It’s true that many of them have long figured out that they’re really supposed to follow orders passed down the hierarchy and do nothing else, even if everything around them is on fire. But some never figure this out, and most managers fail to punish at least some of these slow learners of theirs, so they’re yours to work with.

If you need a feature in something you use, it’s often much easier to persuade someone to take your patches adding this feature than to get the work scheduled within their high-velocity Scaled Agile planning process (of course the plan is already finalized for this quarter as well as the next — it’s our well-run planning process to which we owe our velocity — but in a few weeks, we can discuss the priority of this versus the other features relevant for the quarter after the next one.) Sadly, merging changes might have gotten harder recently, with your well thought-out patch looking no different than random LLM output at first glance, but it shouldn’t be a problem once they get to know you.

Like I said, a lot of people think this is twisted, and why would they do that. I believe that not only do 20% of the people do 80% of the work, but that all this work only achieves its ultimate goals thanks to the <5% of the people who do stuff that someone else is supposed to do, but won’t, for reasons which are perfectly legitimate in the organization’s view of reality, even though the cumulative effect of such legitimate reasons is the certain death of the whole place.

I also believe that you will be well-rewarded for doing everyone else’s job in the many cases where the organization is in a shape bad enough to need this (which is most of them) but is still healthy enough to eventually appreciate it (and if it can’t, it’s on its last breath.) You’ll also see indirect rewards from being one of the few people actually understanding how the place works and what it takes to get something done there.

(If anything, it’s refusing to get into various areas over the years that I personally regret — areas which I totally could get into, but didn’t, on the theory that it’s too far from what I do, not to mention boring. In hindsight, how very stupid. When you drown in preventable problems as a result of thinking someone else’s job will “just get done” when it obviously wasn’t going to, it’s no longer boring, but it might well be too late.)

But not like this

There are 2 seemingly related things which look juicy to senior people but could actually play out badly, that people don’t reject as a twisted form of charity but rather love very much, because they get more headcount. In these cases, you aren't doing someone’s job so that they will be done thanks to you, but you do a similar thing in parallel to them, or instead of them.

The first case is doing something in-house instead of buying it, or using a standard free version. Some places do too much buying and prefer external products even when they suck, because “standards are good” or because it’s easier to spend money than to hire people. But other places do too much in-house building, because in those ones, it’s impossible to approve any expense, but reasonably easy to grow your headcount. If we look at it from the perspective of improving expected outcomes as opposed to doing the easiest thing in a given place, buy vs build is a very hard decision specific to each case, where you need to learn a lot of details so as to honestly weigh your own deficiencies vs the deficiencies of prospective vendors and the structural issues of the market.

The second common variant is, instead of centralizing a service so that everyone in the company uses it, every department does it on their own. Department managers like it because it’s one less party not reporting to them to depend on (and nothing scares managers more than needing things from people not reporting to them.) And the people managing the department managers like it because they don’t need to deal with the centralized service department allegedly failing — and having to figure out if that’s really true and how to fix it, or if it’s an excuse made up by the other departments for failing at their own job. Here, too, if outcomes are what concerns you rather than what's easiest for senior leadership, standardizing on something across the company vs having everyone roll their own variant is a hard decision to make.

Thanks to Dan Luu for reviewing a draft of this post. He commented, “from the opening, I thought it was going to be about the thing where founders supposedly do a better job if they know how to do the job of the person they're hiring, which is why it’s useful to scale the company up from nothing. That's what some founders say, anyway. A friend of mine who's a successful second-time founder (who I think has good judgment) said that it's much easier to hire well if you've done the job yourself, which sounds reasonable/plausible. I guess you could get the same value out of having the right trusted people who've done the job but, empirically, people seem more likely to get who they trust wrong than right.”

This makes sense to me, though I am yet to scale a company of my own to be able to confirm this. I feel it does work this way when growing a team, as long as you know how to manage people who do their job better than you would; but by the time the team grows to hundreds of people, it will often have some people on it whose job you can’t do, and if it’s not one team of several but an independent company, this will happen much more quickly. So I would think that lacking the complementary ability — to manage people whose job you couldn’t possibly do, which is not easy — is what will become the limiting factor.

Introduction

Last year I had to write code which converted texture data between gaming platforms. Going into it, I seriously underestimated the complexity of texture memory layouts.

I'm currently working on a team at 505 Games porting a custom PC game engine to consoles. One of the many challenges faced has been around texture conversion. I knew there were a lot of complexities and subtleties to it. I recognized most of them in isolation. But, it wasn't until I had to write working code which handled all these details in tandem that the full complexity really sank in. And so, I thought this would make for an interesting blog post!

This blog post will explain the complexity of texture memory layout, along with why it is necessary. This will be done through the lens of someone trying to debug the calculation of memory addresses for each part of a console texture. Due to the subject matter, this post will have to get a little bit more technical.

I will assume an understanding of programming fundamentals, memory layouts, as well as familiarity with basic video game technology and digital imagery.

Throughout this article we'll use a 1024x1024 RGBA texture using BC7 as an example. For illustration purposes, let's use a simple wood texture.

Figure 1. 1024x1024 Example texture

Converting this texture between platforms requires us to keep track of a lot of details. For the purpose of this article I will explain the following: block compression, texel ordering, mips, and texture tiles. There are other details such as pitch, depth, and texture array index. These mainly require simple offsets which are slightly annoying but do not add any interesting theory, so I will ignore them.

Here is what that image would look like if we ignore all of these complications and interpret the result as a simple stream of color values:

Figure 2. 1024x1024 chaotic rainbow noise

Platform specifics

If this post is about texture formats, why am I talking about memory locations? All the details we will go through have some very important similarities and distinctions between platforms. In particular, textures are decomposed into the same hierarchy of building blocks across platforms. However, at every level of the hierarchy the order of these elements in memory may differ between platforms. This means that given a memory area to hold the texture, the act of converting a texture between platforms can be seen as a problem of calculating, for each of these elements, its expected memory address after conversion.

We will primarily discuss most of these components in a platform-agnostic way. Where a distinction is necessary, we will rely on the view taken by DirectX 12.

Idealized view

Given a general understanding of digital images, how might one expect a 2D texture to be represented in memory? Images have a number of color channels, each having a specific bit depth. A common bit depth is 8, meaning one byte per channel. This gives us values in the range [0,255] per color. An image comprises a grid of color samples, typically across two dimensions: width and height. For ordinary images these samples are called pixels. For textures we call them texels. Given this, the naive assumption would be that an RGBA image with 32-bit color depth is just a stream of texels sweeping from left to right, row by row, with each texel represented as four consecutive bytes: one for each channel. While some simple textures really do work this way, this is unfortunately very far from how most textures are represented in modern video games.

The reason is performance. There's a number of orthogonal techniques applied, each of which complicate texture representation but improve rendering performance. Most (e.g. texel ordering & texture tiles) exist to improve cache locality. That is, doing our best to store data as close as possible to all other data we expect to need at the same time. Good locality drastically decreases the amount of time wasted waiting for memory transfers - which is one of the most expensive operations in modern computer hardware. While block compression also helps with locality, it primarily improves memory transfer speed by decreasing texture size in memory. Finally, we have mips which drastically reduce rendering cost by downsampling the entire texture in multiple steps ahead of time - allowing us to avoid the many expensive texture samples we'd otherwise need any time we want to average all texels in a surrounding area.

From the perspective of someone debugging texture loading across platforms, every one of these techniques is another wrinkle to keep track of.

A single texel

Let us take a look at how an actual texel is represented in a typical modern video game texture. For simplicity, let us assume the original image is RGBA with 8 bits per channel as above. This gives us a total of 32 bits (4 bytes) per texel if uncompressed. However, most modern textures use block compression. Block compression shrinks the memory size of textures to decrease data transfer times.

The most common block compression format these days is BC7. This is a very complex format with more quirks than I can explain in this article - I don't even know them all in detail. At its core, BC7 is a lossy compression format leaning on clever assumptions about similarity of adjacent texels. It stores texels in 4x4 blocks. Each block specifies pairs of reference colors called end points. Each texel then gets its color by specifying an index identifying an interpolated value between these end points. A single block is 16 bytes large and describes 16 texels - which amortizes to a single byte per texel; a compression factor of 4x for an RGBA texture with 8-bit channels. Despite this large compression factor it is very hard for the human eye to tell the difference between BC7 compressed textures and uncompressed textures - even in side by side screenshots. This is great news for game developers who need to optimize streaming of large numbers of big textures. Another advantage of compressing texels in blocks is that a lot of effects require sampling multiple adjacent texels. Making nearby texels closer in memory increases cache locality and speeds up memory access.

Here you can see what our above texture looks like if we correctly treat our texture as a series of 16 byte chunks each representing a 4x4 BC7 block. We can now see blocks of the correct colors, though out of order.

Figure 3. 1024x1024 jumbled mix of texels out of order

Unfortunately, block compression adds a fair bit of complexity to our conversion code. This is because it changes texture memory size and texture indexing. The biggest gotcha is that not all textures are compressed, so your code has to consistently do the right thing in both cases.

Code which would otherwise read/write a single texel now has to check whether to deal with a texel or block. For non-compressed textures width and height are just an index across each dimension. Compressed textures need to iterate across blocks of 4x4 texels at a time. Compression also affects how to calculate the size of a texture in memory. Memory size depends on resolution, number of channels, channel bit depth, and potential compression mode.

Debugging a block

The amount of clever tricks employed by BC7 is bad news for debugging. It makes a memory dump practically inscrutable without tooling. Graphics debuggers contain built-in tools to visualize and analyze textures. Unfortunately, that often doesn't help when the result looks like the above figures. It is also non-trivial to write visually interpretable debug information to a compressed texture. If you write raw values ignoring compression you'll get a mess of meaningless colors.

What really complicates visual debugging is that any accidental offset which changes your byte alignment renders all data visually incoherent. This is because it affects which parts of the blocks are read as end points and which are interpreted as indices. If you write debug information because you have a problem with your texture address computations it can even be a challenge to find where this information ended up.

Luckily, I found a few useful tricks for debugging.

In most cases, you can simply replace each 16 byte block (128 bits) with 4 separate 32-bit integers of your choice. For instance, this is just enough to store x, y, z coordinates and mip level. That way, you can easily identify the exact memory offset by reading the memory dump of any given texel block and comparing its written values to the actual coordinates.

In some cases I needed to write larger chunks of data to specific texture locations matching some debug criteria. In these cases you can write 16 bytes of all zeroes to get a block of 4x4 texels which is technically invalid but guaranteed to render as black. I've been using a series of black blocks as visual markers. Just enough to get a few visible blocks even with alignment issues. Between these markers I write my raw debug data values. This way I can visually identify debug portions in a texture view, find the memory location of the corresponding texel, then use memory dumps to read the actual values between the markers.

Texel ordering

It's easy to imagine texture memory as a byte stream sweeping texel by texel, row by row. Unfortunately, such a layout is not ideal. We want to do everything we can to increase locality of neighboring texels. That is, texels which are visually close to each other should also be close in memory. This matters because nearby texels will often be sampled together. To that effect, textures often use different memory layout patterns. Such a pattern is called a swizzle. When iterating over all texels in a texture, the swizzle allows you to transform a texel index to the coordinates of the corresponding texel. The most well-known swizzle is probably the Morton order. This swizzle means the order of texels in memory isn't left to right, row by row, but rather moves in a fractal Z pattern across the texture. Use the slider to see the difference. The left and right view show the same image each multiplied by a linear and z-ordered mask respectively. Meaning that 4x4 texel elements go from black toward their original color as their index increases.

Figure 4. A comparison of linear (left) vs Morton (right) texture ordering. Use the slider to change between the two.

There are other possible orders, and which one is needed depends on the details of your specific texture. This choice may also differ for the same texture across platforms. Meaning you may need to completely re-order the texels when converting a texture from one platform to another. In the general case, your conversion code has to take into consideration both which ordering is used for the source and target platform.

As with block compression, this re-ordering makes debugging more complex than it already is. Correctly aligned texture memory using the wrong swizzle will make your image look like 4x4 puzzle pieces all jumbled up. See Figure 3 above.

All of this is getting quite complicated, but with a bit of grit we can sort it out. Of course, it gets worse.

Mips

Textures aren't actually single images. Each texture contains a series of increasingly scaled down versions of the same image, called mips. The largest version (mip 0) has the size of the original image. Each consecutive mip is half the width and height of the previous one. The reason for this is that when rendering, we want the resolution of sampled texels to match the resolution of the render target. For a textured surface further from the camera, each pixel will overlap several texels. This means we need to sample a larger area of the texture, which is relatively expensive. Not doing so will cause aliasing artifacts such as shimmering and Moiré patterns. The solution is to create mips ahead of time. When rendering a textured surface the shader can then use the mip where texel size most closely matches the render target pixel size. Real-world renderers use more advanced texture filtering techniques, but they all rely on mips to precompute area sampling.

Mips are largely laid out in well-defined order one after the other in memory. However, whether they are stored in ascending or descending order varies between platforms. This means you cannot simply iterate over them and increase both source and destination pointer in lockstep. For each mip you need to figure out where in memory it starts for the source and target platform.

Figure 5. Illustration of the 256x256 mip next to all higher mips.

Tiles

The next complication is tiles. Again, we want to optimize cache locality when working on a texel and its surroundings. Another way this is done is by splitting texture memory into tiles. These tiles each hold a square piece of the underlying mip. This layout also allows games to stream only the visible parts of large textures. Within each tile, adjacent texels are block compressed and swizzled as described above. Tiles are laid out linearly in memory from top left to bottom right. A tile is typically 64KiB. This means a 1024x1024 texture mip using BC7 is made up of 16 tiles in a 4x4 grid, with each tile containing a 256x256 texel square. With BC7, each tile contains 4096 blocks of 4x4 texels each, laid out according to whatever swizzle pattern we're using.

Figure 6. 1024x1024 texture with each tile masked by a different tint

Thinking of how to store this in memory, it's easy to consider a hierarchical view as described above. Textures are made of mips. Mips are made of tiles. Tiles are made of blocks or texels. Of course, the smallest mips will be much smaller than a full tile, so giving them a full tile each would be very wasteful. For our example texture we could fit the last 8 mips in a single 64KiB tile! Doing this for every texture the memory savings quickly add up. Hence, most platforms pack all mips smaller than a full tile into as few tiles as possible. This is called the mip tail. In fact, figure 5 shows precisely the 256x256 mip which fills a single tile next to all higher mips, showing that together they fit in a single packed tile.

This means we need to handle both mips spanning multiple tiles and tiles containing multiple packed mips.

Non-contiguous tiles

While everything above is nearly sufficient to convert a contiguous linear texture to console specific memory layout, there is a final wrinkle. In a modern game engine textures are often streamed tile by tile into a shared area of texture memory (called a heap in DirectX12). Each tile of a texture is mapped to its own memory area within this heap. With different mips getting streamed in and out on demand we may get fragmentation of texture memory. This means different tiles of a texture may not end up in a contiguous area of texture memory. Which in turn means if we are writing data to texture memory we need to create a translation from tile index of a texture to the specific address where this tile is mapped. Then we add an offset within the given tile to get actual memory address. A simpler take would be to base our calculations on the base address for the first texel of our texture plus a global offset. But for non-contiguous textures this would overwrite tiles from other textures and in turn leave some of our own tiles uninitialized. This is similar to an actual bug I caused during this project, and it took me quite some time to realize the faulty assumption!

Summary

Combining all of these gives you a rough view of how a texture is laid out in memory and how it may differ between platforms. Putting it all together in my conversion code took surprisingly much work. Along the way I kept bumping into special cases and specific textures which broke implicit assumptions I'd made. Hopefully, you can now share my appreciation for this aspect of the complexity which goes into making your games run just a little bit faster.

I've spent the last month adding features and improving performance in z486_MiSTer, mostly working through games from the first half of the 1990s. Looking a few years ahead brought me to another change I wanted to explore: the arrival of 3D graphics cards.

The first one that left a strong impression on me was the Voodoo. The game was Need for Speed II SE. Smooth textures, fog, and the speed of the whole thing made it feel like a new generation of PC gaming. Could I recreate that on an FPGA now that the z486 CPU exists?

The result of this detour is zSST, a SystemVerilog implementation of the 3dfx Voodoo Graphics, or SST-1. Combined with my z486 CPU and the surrounding PC hardware, it forms z486 XL: a DOS PC with Voodoo graphics running in the programmable logic of a Xilinx KV260 board. Tomb Raider now runs with its original 3dfx renderer.

A teal Gouraud-shaded teapot rendered by the zSST simulation

zSST rendering the Utah teapot in simulation.

Tomb Raider Level 2 running with zSST on a KV260, captured over HDMI

Tomb Raider on the complete FPGA PC.

zSST implements most of the central Voodoo features: prepared triangles, texture filtering and mipmapping, depth and alpha tests, fog, blending, dithering, framebuffer access, and buffer swaps. It supports both the fixed-point and floating-point setup interfaces. Hardware game testing is still concentrated on Tomb Raider; broader compatibility and later Voodoo generations are work for another day.

The CPU and renderer run at 100 MHz on the KV260. That board has enough logic, DSP blocks, on-chip memory, and DDR bandwidth for the combined design. The DE10-Nano does not have room for this graphics addition. The KV260 uses its onboard DDR; there is no external SDRAM module to add.

Starting from the programming model

Fortunately, there is plenty of material to work from. 3dfx released the Glide source in 1999, before NVIDIA acquired its core graphics assets in December 2000. The surviving Glide source and SST-1 specification explain how software prepares triangles, configures the pixel pipeline, and manages textures and framebuffers.

The specification is a behavioral target, rather than a circuit diagram. It tells what should happen when software writes a register, but leaves many implementation choices open. 86Box provides useful references for complicated rendering behavior. The earlier MAME Voodoo work is another part of this preservation history. SpinalVoodoo supplied particularly useful Glide traces and reference screenshots for testing.

From triangles to 3D, one pixel per clock

Voodoo Graphics turns triangles into pixels, leaving much of the 3D work to the host CPU. Its command interface is surprisingly compact: five main command registers drive the accelerator.

Both triangle commands launch the same rendering pipeline. Other registers hold coordinates, gradients, and render state, while memory-mapped regions provide texture uploads and direct framebuffer access. The main drawing primitive is simply a prepared triangle.

For game developers, Glide presents a friendlier interface:

void grDrawTriangle(const GrVertex *a, const GrVertex *b, const GrVertex *c);

Before this call, the host CPU transforms the 3D geometry, computes vertex lighting, clips it, and projects it onto the screen. Glide then prepares the screen-space triangle and its parameter gradients—the increments used to interpolate values across its surface—and writes the triangle command to start rendering. Unlike later GPUs such as the GeForce 256, SST-1 has no hardware transform-and-lighting engine.

That still leaves plenty of work for the accelerator. The rasterizer finds which pixel centers lie inside the triangle and interpolates their color, depth, and texture coordinates. The texture unit fetches and filters texels; the framebuffer unit combines colors, applies visibility tests and fog, blends with the existing image, and writes the result.

A triangle rasterizer feeding a pipeline with several different pixels in flight simultaneously

A pixel takes several stages to finish, while new pixels can keep entering.

The original card divides this work between two ASICs: the FBI, or Frame Buffer Interface, and TREX, the texture mapping unit, usually called the TMU. At a 50 MHz graphics clock, the advertised peak is one textured, depth-tested output pixel per clock: 50 million pixels per second.

One pixel per clock does not mean that a pixel finishes in one clock. It means that different stages can work on different pixels simultaneously: while one pixel is being textured, an earlier one can be blended and another written out. Once the pipeline is full, it can ideally accept and finish a pixel every clock, provided memory keeps up.

That is the appeal of a fixed-function pipeline. A software renderer executes many instructions for each pixel; dedicated hardware overlaps that work across a steady stream of pixels. Voodoo brought richly textured 3D games to life at a fluid 30 FPS or more—a big part of what made it so popular.

Building the pixel pipeline

Compared with an x86 CPU, the arithmetic path is pleasantly regular. Let's follow a pixel from its interpolated parameters through texturing and color operations to the framebuffer, starting with how the numbers are represented.

Fixed point behind a floating-point interface

Floating-point arithmetic is central to modern GPU programming. SST-1 sits at an interesting transition: software can submit floating-point values, but the rendering machinery largely operates in fixed point—integers with an implicit scale factor.

Here 12.4 means twelve bits before the binary point, including the sign, and four fractional bits. A screen coordinate of 10.5 is therefore stored as the integer 168: multiply by 16 to encode it, divide by 16 to recover the value. Those fractional bits let the rasterizer handle vertices between pixel centers.

The fvertex, fstart, and floating-point gradient registers accept IEEE single-precision values. SST-1 converts them into its internal fixed-point representation, and zSST follows that contract. Once the triangle is prepared, advancing along a scanline mostly means adding a precomputed increment to each interpolated parameter. Much of the pixel-by-pixel work becomes simple integer addition.

Four texels for one pixel

For perspective-correct texturing, the TMU interpolates S/W, T/W, and 1/W, then divides the first two by the third to recover texture coordinates. This keeps a floor or wall texture in perspective as the surface recedes. The TMU also selects a mip level: a smaller version of the texture for pixels that cover a larger area of its surface. This reduces aliasing and shimmering in the distance.

Bilinear filtering then combines four neighboring texels—the pixels of the texture—around the sample position. First blend the top pair horizontally, then the bottom pair, and finally blend vertically between those two results. The fractional position determines the weights, producing a smooth transition between texel colors instead of an abrupt jump from one to the next.

Four neighboring texel centers and two horizontal interpolations followed by a vertical interpolation

Four texture reads produce one filtered sample.

In zSST, a four-stage front end pipelines the perspective and level-of-detail calculations. Address generation and cache lookup supply the texels, and two registered decode stages turn their stored formats into colors for filtering and texture combining. Palette-based and NCC-encoded textures need different decoding rules, but ultimately feed the same pixel stream.

Color, tests, fog, and blending

Once texture and framebuffer data are available, zSST's FBI pixel path uses six registered stages:

These stage boundaries are chosen to meet the FPGA's clock target. The SST-1 specification describes the operations but does not reveal the original ASIC's exact pipeline registers. Splitting fog lookup from fog application, for example, keeps a long arithmetic path out of a single clock while retaining the ability to accept one pixel per clock.

The result is written to the back buffer. A retrace-synchronized buffer swap then displays the finished image without switching buffers halfway through scanout. The original Voodoo was a 3D-only add-on, passing the ordinary VGA card's output through when inactive. z486 XL makes the analogous selection between the PC's VGA output and zSST's display output inside the FPGA system.

The hard part: feeding it from memory

The zSST pixel pipeline proved relatively straightforward to implement, at least compared with z486's CPU pipelines. Keeping it fed turned out to be much harder. A bilinear sample needs four texels from separate addresses. A depth-tested, blended pixel also needs the existing depth and color, followed by writes of the new values. Performing those accesses one at a time quickly destroys throughput. I ended up spending more time designing, tuning, and debugging the memory system than the arithmetic pipeline.

How the original card supplied the pixels

Diamond Monster 3D board photograph, with TMU and texture RAM above FBI and framebuffer RAM Matching schematic: TMU and FBI each connect to four EDO chips over a 64-bit interface; arrows show texture work, filtered color, PCI commands, and display output through the RAMDAC

The board and its division of labor, with matching chip positions in both views. Connections are schematic; click either image to enlarge.
Photo: Konstantin Lanzet; crop: Pittigrilli, Wikimedia Commons. Photo license: GFDL 1.2 or later. Schematic: nand2mario.

The division of labor is visible on this Diamond Monster 3D. The upper 3dfx chip is the TMU, the lower one the FBI, each with four EDO RAM chips to its right. The upper group holds textures; the lower group holds color and depth/alpha buffers.

FBI and TMU each have a dedicated 64-bit memory path. On the texture side, four-way interleaving lets the banks read independent addresses, supplying the four neighbors for bilinear filtering in parallel. The specification (p. 13) promises the same throughput as point sampling, without storing duplicate texels.

But what if two neighboring texels land in the same chip? The trick is to distribute texels in a repeating two-dimensional pattern, rather than split the image into four large regions. Assign a bank to each combination of even or odd column and row, and the reason becomes clear:

An alternating A/B/C/D bank layout: an aligned 2×2 window and a boundary-crossing window both contain all four banks, allowing one independent read per bank

Moving the sample changes the banks' positions, not their number. Coordinates are (column, row); bank letters illustrate the principle, not physical SST-1 chip numbers.

Every 2×2 window contains A, B, C, and D—even the orange window crossing both horizontal and vertical block boundaries. Two consecutive columns have opposite parity, as do two consecutive rows. All four combinations occur exactly once, so each bank supplies one texel with no conflict.

Texture edges and small mip levels need a little more care. SST-1 uses power-of-two texture dimensions, so wrapping preserves the alternating pattern for dimensions of two or more. At clamped edges, or in mip levels only one texel wide or high, some samples reuse the same texel. The central insight remains: fast bilinear filtering depends on arranging memory so that the arithmetic receives all its inputs together.

The FBI applies a similar idea to color and depth/alpha memory. Its interleaved path supports a peak of one rendered pixel per clock, or two pixels per clock for clears. Working on adjacent pixels together spreads the read/write cost across a scanline. Fabien Sanglard's two-pixel explanation offers a useful reconstruction of this behavior, though the exact ASIC bank schedule is not documented in the programming guide.

At 50 MHz, each 64-bit path has a theoretical bandwidth of 400 MB/s: 800 MB/s in total, but reserved for different jobs. The TMU cannot borrow idle FBI bandwidth, or vice versa. These dedicated buses and carefully arranged banks remind me of the NES- and SNES-era designs I explored in projects such as SNESTang: getting the most out of memory means designing around exactly when and where each value is needed.

What changes on an FPGA SoC

Voodoo's memory layout explains how it kept the pipeline busy, but I cannot simply transplant that design to the KV260. The board has much more memory bandwidth, yet no dedicated EDO memory attached to either rendering unit. Instead, the FPGA accesses shared DDR through the Zynq processing system's AXI ports. Linux, the FPGA PC, and display scanout all compete for that memory. The goal is the same—keep the pixel pipeline fed—but the way to achieve it has to change.

Separate FBI and TMU EDO paths on SST-1 compared with shared DDR and three AXI clients on KV260

Original SST-1 has dedicated texture and framebuffer buses. zSST shares its renderer port and uses buffering to tolerate DDR latency.

Our KV260 measurements show why bandwidth alone is not enough. A 128-bit port at 100 MHz has a theoretical bandwidth of 1.6 GB/s. With one request outstanding, a 4 KiB read reaches 1,370 MiB/s, but a 64-byte read reaches only 189 MiB/s. The first data typically takes about 280 ns to arrive—roughly 28 clocks at 100 MHz—with occasional much longer waits.

Measured DDR bandwidth rises with burst length while time to first data stays near 280 nanoseconds

Single-outstanding board measurements. Long bursts amortize latency; small requests need concurrency.

A renderer that waits for each small read before issuing the next will spend most of its time idle. zSST needs enough independent work in flight to cover those waits.

Caches, replay, and a reorder buffer

Keeping the pipeline fed requires both fewer DDR accesses and less time spent waiting for them. The first step is caching. Nearby screen pixels often sample overlapping parts of a texture, so recently fetched texels can be reused from on-chip RAM. zSST's texture cache holds 8 KiB in 64-byte lines; each fetch also brings in neighboring texels that subsequent pixels are likely to need.

A cache miss still takes many clocks, but independent texture samples need not wait for it. zSST keeps up to eight cache-line fetches outstanding, using a replay queue to park samples with missing data and retry them when it arrives. Meanwhile, samples whose texels are already cached can proceed. Prefetching gets a head start on future reads.

Now a later cache hit can finish before an earlier miss. A 64-entry reorder buffer, or ROB, collects those results and releases them in their original order. The principle is familiar from CPUs: do useful work during a long wait, then restore order before passing the results downstream.

The framebuffer side uses separate 4 KiB color and depth/alpha read caches, while write combiners pack neighboring 16-bit updates into 128-bit requests. Here, ordering matters: blending or depth testing may need a value that an earlier pixel has changed but not yet written to DDR. Forwarding supplies the pending value directly. Framebuffer updates take effect in order, and state changes that require completed work wait for it to drain. Memory requests can overlap, but later pixels must still see the effects of earlier ones.

Texture and framebuffer requests proceeding in parallel, joining by tag, and retiring through the FBI pixel stages and write combiner

Caches reuse nearby data; queues overlap memory requests; ordered retirement preserves the result.

FBI and TMU share the renderer's 128-bit AXI port, HP2. The PC uses HP0 and display scanout uses HP3, keeping their request queues separate even though all three ultimately share DDR.

Evaluation results

I measure the renderer separately from the complete PC. The simulation benchmark sends commands through zSST's front end and exercises the TMU, FBI, shared arbiter, and a DDR timing model. Read data arrives after at least 26 clocks, with deterministic variation and occasional longer delays; writes are also rate-limited. The full-renderer tests allow 32 outstanding reads.

At 100 MHz, zSST reaches 78.5 million pixels per second (MPix/s) for textured triangles, and 72.8 MPix/s with depth testing and blending. Voodoo 1's published estimates at its native 50 MHz are 43 and 37 MPix/s for comparable feature sets. This is not an apples-to-apples benchmark: the triangle workloads differ, and the original estimates also include fog, mipmapping, and Gouraud shading. I do not have a Voodoo 1 to run the same test on both. The comparison shows the approximate fill-rate range, not a measured speedup over the original card.

Native-speed fill-rate comparison: textured zSST 78.47 versus SST-1 43 MPix/s; textured with depth and blend zSST 72.83 versus SST-1 37 MPix/s

100 MHz zSST simulation versus the published 50 MHz SST-1 estimates. The tests cover similar feature classes, but use different workloads.

High fill rates do not automatically translate into high game FPS. On the board, Tomb Raider Level 2 produced 237 displayed buffer swaps in about 20 seconds—roughly 12 per second, measured from swaps rather than an engine FPS counter. Preliminary measurements point to a CPU bottleneck: it still has to run the game, prepare geometry, and submit commands. Shared DDR contention may also contribute. There is plenty left to optimize in the complete machine.

For non-Voodoo games, the current 100 MHz z486 XL runs maximum-detail Doom at 38.5 FPS and Quake 1.06 at 8.1 FPS. That is roughly 20% faster than the 85 MHz DE10-Nano build—about 23% for Doom and 19% for Quake. A 512 KiB write-back L2 cache in UltraRAM helps the CPU make better use of DDR.

In the integrated XCK26 build, zSST accounts for about 29,500 LUTs, 28,100 flip-flops, 14 RAMB36 blocks, 8 RAMB18 blocks, and 97 DSP slices. The combined PC and graphics design meets timing at 100 MHz.

Closing

The rewarding part is seeing original Glide software drive hardware I built in RTL. I expected the rendering arithmetic to be the hard part; getting data to it efficiently took more work. Voodoo's carefully interleaved EDO and zSST's caches and queues solve the same problem under very different constraints: a fast pixel pipeline is only useful when it has something to do.

Both zSST and z486 XL are available open source. If you already have a KV260, the z486 XL SD image provides the Linux support and application needed to launch your own DOS disk images.

Credits: Thanks to SpinalVoodoo for the Glide traces and reference screenshots, and to 86Box for its implementation references. Fabien Sanglard's The story of the 3dfx Voodoo1 is an excellent introduction to the original card's memory system.

Before television shows like The Office and Parks and Recreation cemented the mockumentary in the minds of millions, there was Christopher Guest. He didn’t invent the genre, but he’s widely recognized as one of its most influential practitioners, and for my money, there’s none better. I’ve watched Waiting for Guffman and Best in Show more times than I can count. But the one that has stuck with me the most, the one I quote at the slightest provocation, is This Is Spinal Tap.

If you’ve seen it you already know where this is going (and if you haven’t, you now have weekend plans). The film is a fictional documentary about an aging English rock band named Spinal Tap, whose members are everything we picture when we picture over-the-top rock stars. In one of its more memorable scenes, the guitarist (Nigel) gives the filmmaker (Marty) a tour of his most prized gear, in particular showing off an amplifier unlike any other: its dials don’t stop at ten. That leads to what might be the single most quoted exchange in the entire movie:

Nigel: “You see, most blokes, you know, will be playing at ten. You’re on ten here, all the way up, all the way up, all the way up, you’re on ten on your guitar. Where can you go from there? Where?”

Marty: “I don’t know.”

Nigel: “Nowhere. Exactly. What we do is, if we need that extra push over the cliff, you know what we do?”

Marty: “Put it up to eleven?”

Nigel: “Eleven. Exactly. One louder.”

This is .NET 11. It’s one louder, with another year’s worth of performance work having gone into making the runtime and libraries that much faster. Of course, the premise of Nigel’s special amplifier is ludicrous, as is exemplified in the subsequent few lines of dialog:

Marty: “Why don’t you just make ten louder and make ten be the top number and make that a little louder?”

Nigel: (pauses) “…these go to eleven.”

In contrast, .NET 11 is actually one higher, one louder. The sections that follow are full of real improvements. A bounds check removed, an allocation that no longer happens, a lock that isn’t taken, a loop that runs in fewer cycles than it did a year ago, a comparison folded to a constant here, a redundant check hoisted out of a loop there, a couple of instructions fused into one, a syscall sidestepped, an array copy handed off to SIMD, and on and on. That’s how real performance work goes, accumulating gain after gain, each compounding on the last, until the whole thing is measurably, provably louder. And so, in this post, as I’ve done in past years with .NET 10, .NET 9, .NET 8, .NET 7, .NET 6, .NET 5, .NET Core 3.0, .NET Core 2.1, and .NET Core 2.0 before it, we’ll take an unhurried tour through hundreds of them.

This is a long one. It’s meant to be. Grab your hot beverage of choice, settle in, and let’s turn it up.

Benchmarking Setup

As in previous years, the post is chock full of micro-benchmarks that demonstrate the individual improvements. Almost all of them use BenchmarkDotNet, and each is written to be self-contained so you can try it out yourself.

Start by ensuring you have both .NET 10 and .NET 11 installed (most of the benchmarks compare the same code running on both versions) and create a new console project in a fresh benchmarks directory:

dotnet new console -o benchmarks
cd benchmarks

Replace the contents of the generated benchmarks.csproj with the following, which multi-targets both versions so that BenchmarkDotNet can build for each:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFrameworks>net11.0;net10.0</TargetFrameworks>
    <LangVersion>preview</LangVersion>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
    <AllowUnsafeBlocks>true</AllowUnsafeBlocks>
    <ServerGarbageCollection>true</ServerGarbageCollection>
    <SystemPackageVersion Condition="'$(TargetFramework)' == 'net10.0'">10.0.12</SystemPackageVersion>
    <SystemPackageVersion Condition="'$(TargetFramework)' == 'net11.0'">11.0.0-rc.1.26425.128</SystemPackageVersion>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="BenchmarkDotNet" Version="0.16.0-preview.1" />
    <PackageReference Include="System.IO.Hashing" Version="$(SystemPackageVersion)" />
    <PackageReference Include="System.Runtime.Caching" Version="$(SystemPackageVersion)" />
    <PackageReference Include="System.Numerics.Tensors" Version="$(SystemPackageVersion)" />
  </ItemGroup>

</Project>

For a given benchmark to test, copy its complete contents over everything in Program.cs and then run it. Each benchmark includes as a comment at the top the exact command to use. In most cases, it’s:

dotnet run -c Release -f net10.0 --filter "*" --runtimes net10.0 net11.0

which builds in Release and runs the benchmark against both .NET 10 and .NET 11, emitting a side-by-side comparison. The other common form, used when a benchmark is comparing two coding approaches on a single runtime (rather than the same code across two runtimes) is:

dotnet run -c Release -f net11.0 --filter "*"

The usual disclaimer applies: these are micro-benchmarks, many measuring operations so short that a blink would miss them. Your results will vary with your hardware, OS, runtime configuration, what else your machine happens to be doing at that exact moment, and whether Mercury is in retrograde.

Every line of managed code ultimately ends up at the just-in-time compiler, so let’s start there.

JIT

Of all the places to improve .NET’s performance, few have as broad an impact as the just-in-time (JIT) compiler. C#, F#, and Visual Basic are typically compiled first to intermediate language (IL), and the JIT ultimately turns that IL into the native instructions the CPU executes. A JIT improvement can therefore benefit application and library code wherever the optimized pattern occurs, often with no source changes or recompilation of the application itself. Even removing a single instruction or proving one check unnecessary can add up when the code is on a very hot path.

Deabstraction

We as developers love our abstractions. They let us write clean, reusable, object-oriented code, but we don’t want to pay for every abstraction at run time. The runtime can often undo an abstraction when it proves the effects aren’t observable. It can look at a virtual call and determine which concrete method it’ll invoke, look at a heap allocation and recognize that the object never leaves the current stack frame, or look at an interface cast and reuse a type fact already established earlier in the method. This process is called “deabstraction.” .NET has improved steadily in this area for years, and that continues in .NET 11.

Every time you write interface in C#, you’re creating a contract, a promise that any type implementing that interface can be substituted for any other. That flexibility is enormously valuable because, for example, it’s what lets us write IEnumerable<T> and have it work equally well over arrays, lists, other collections, LINQ, custom iterators, and so on. But the CPU doesn’t know anything about these contracts; it just knows how to execute instructions. Turning “call whatever method this interface reference points to” into actual machine instructions requires special machinery. Consider this example:

// dotnet run -c Release -f net11.0 --filter "*"

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;
using System.Runtime.CompilerServices;

BenchmarkSwitcher.FromAssembly(typeof(Benchmarks).Assembly).Run(args);

[DisassemblyDiagnoser, HideColumns("Job", "Error", "StdDev", "Median", "RatioSD")]
public class Benchmarks
{
    private Animal _animal = Environment.TickCount >= 0 ? new Dog() : new Cat();

    [Benchmark]
    public int Speak() => _animal.Speak();

    public abstract class Animal
    {
        public abstract int Speak();
    }

    private sealed class Dog : Animal
    {
        [MethodImpl(MethodImplOptions.NoInlining)]
        public override int Speak() => 1;
    }

    private sealed class Cat : Animal
    {
        [MethodImpl(MethodImplOptions.NoInlining)]
        public override int Speak() => 2;
    }
}

At compile time, all else equal, the JIT doesn’t know whether _animal is a Dog or a Cat. It generates code that loads the instance’s “method table pointer” (its object type handle), sometimes called a “vtable pointer”, stored at the beginning of every .NET object, indexes into the method table at the known slot for Speak, and calls the function pointer found there:

; x64
mov     rcx, [rcx+8]   ; load _animal
mov     rax, [rcx]     ; load method table
mov     rax, [rax+40]  ; load vtable chunk
call    qword ptr [rax+20]

For this one call to Speak, we pay three dependent memory dereferences and an indirect call because the processor doesn’t know for certain in advance where the call is going (it might guess, or “speculatively execute”, but it has to be prepared for the possibility it was wrong), and because the call target is indirect, the JIT can’t inline the callee. Whatever Speak does, its code can’t be folded into the calling method.

That’s a performance problem. Those indirections have overhead, but the bigger cost is the lost opportunity to inline. Inlining not only saves function call overhead, more importantly it opens the callee’s code up to the same optimizations that are operating on the caller, such as constant propagation, dead code elimination, bounds check elimination, further devirtualization, etc. That means a series of small virtual calls that each look innocent can, when devirtualized and inlined, collapse into a handful of instructions that would be unrecognizable and way cheaper when compared to the original source code. Without inlining, each callee is an opaque box; with it, the JIT can see through the layers.

We as .NET developers constantly rely on the JIT’s sophisticated heuristics for inlining that weigh the IL size of the callee, the exact work the callee is performing, the call frequency of the method, the expected benefit from constant arguments, and dozens of other factors. For virtual calls, the JIT needs to know what the actual target of the call will be; it needs to “devirtualize”. In some cases, it can determine that statically, where it has exact-type knowledge. For example, if the JIT can prove that animal is always a Dog, whether because it was just allocated with new Dog():

Animal animal = GetSomeAnimal();
animal.Speak();
...
static Animal GetSomeAnimal() => new Dog(); // inlineable

or because the variable’s type is a sealed class:

Dog animal = GetSomeAnimal();
animal.Speak();
...
sealed class Dog { ... } // impossible for `animal` to be anything other than a `Dog`

or with NativeAOT and whole-program compilation, if it sees that Animal is abstract and the only type in the whole application that derives from Animal is Dog:

Animal animal = GetSomeAnimal();
animal.Speak();
...
abstract class Animal { ... }
class Dog : Animal { ... } // no other such derived type

or other such validation, it can emit a call to Dog.Speak() directly, and the inliner can take its shot.

But for other cases where it can’t prove this with static analysis, the JIT turns to profile-guided optimization (PGO). PGO sounds fancy, but it’s conceptually simple. With “tiered compilation”, when a method is first invoked, it can be compiled “just in time” with few-to-no optimizations (this is referred to as Tier 0). The JIT can include in this compilation additional probes (think “printf debugging”) that let it track a bunch of interesting information about the nature of the code, recording what actually happens when it runs: which branches are taken, what are the concrete types that show up at virtual call sites or cast attempts, and so on. If the method is invoked enough or loops enough times, the runtime can ask the JIT to produce a new optimized version (referred to as Tier 1). That compilation can then factor in all of the learnings gathered as part of that profiling.

The JIT, of course, still needs to generate code that’s always correct. Even if a dynamic profile says animal was Dog 100% of the time, that doesn’t guarantee it’ll always be Dog in the future; it could be that the first 1000 calls passed in a Dog but the 1001st call is going to pass in Dolphin. How can the JIT incorporate this learning then? By emitting a run-time check. The Dog path can get a direct call, which may then be inlinable, and the other path keeps the original virtual call as the fallback. The speed comes from making the common case tiny, while correctness comes from leaving the uncommon case intact.

// Approximately what the JIT generates
if (animal?.GetType() == typeof(Dog))
{
    ((Dog)animal).Speak();  // devirtualized, inlinable
}
else
{
    animal.Speak(); // original virtual call, hopefully rare
}

This “guess and verify” pattern, called “guarded devirtualization” (GDV), accounts for many of the biggest throughput wins in real workloads. It’s applicable not only to virtual dispatch but also to interface dispatch, which also happens to be a bit more expensive than virtual dispatch because a type can implement any number of interfaces and that means the interface slots don’t simply map to fixed vtable positions.

Deabstraction can also make object creation more efficient when it reveals what kind of object is involved. In general, objects in .NET are allocated on the garbage collected heap, tracked by the garbage collector (GC), and collected when no longer reachable. Heap allocation is typically fast, often effectively just bumping a pointer. However, when there’s not enough space available to bump the pointer, it can get much more expensive, including needing to incur a garbage collection. Every allocated object also effectively incurs the amortized cost of all collections, as every allocated object eventually needs to be cleaned up.

“Escape analysis” is the compiler technique that lets us ask whether this object ever “escapes” the current method. If an object reference to a newly allocated object provably doesn’t escape, then the JIT can more efficiently allocate it. It needn’t store it on the GC heap, because nothing could possibly need to reference that object again, so it can instead allocate the object on the stack, making both allocation and cleanup essentially free. Stack allocation is even faster than heap bump-pointer allocation; it’s just decrementing the stack pointer, which is typically already in a register. And more importantly it means zero GC impact, because the stack frame is freed atomically on function return.

The JIT’s been progressively expanding escape analysis over the past several .NET releases, with .NET 9 and 10 seeing significant investments in stack-allocating delegates and closures, Nullable<T> temporaries, and small helper objects. The key theme is that every false positive escape, every time the JIT incorrectly concludes an object may escape when it really doesn’t, represents a heap allocation that could have been avoided, and we want to whittle away at that false positive list. In .NET 11, the JIT trims that list in several ways.

We’ll start with nullable boxing. Consider this benchmark:

// dotnet run -c Release -f net10.0 --filter "*" --runtimes net10.0 net11.0

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;

BenchmarkSwitcher.FromAssembly(typeof(Benchmarks).Assembly).Run(args);

[MemoryDiagnoser(false), HideColumns("Job", "Error", "StdDev", "Median", "RatioSD")]
public class Benchmarks
{
    private int? _nullableNull;
    private int? _nullableValue = 42;

    [Benchmark]
    public object? BoxNullableNull() => (object?)_nullableNull;

    [Benchmark]
    public object? BoxNullableValue() => (object?)_nullableValue;

    [Benchmark]
    public string? FormatNullableInt() => Format(_nullableValue);

    private static string? Format<T>(T value)
    {
        if (value is IFormattable formattable)
            return formattable.ToString(null, null);

        return null;
    }
}

Method Runtime Mean Ratio Allocated Alloc Ratio BoxNullableNull .NET 10.0 2.095 ns 1.00 – – BoxNullableNull .NET 11.0 1.764 ns 0.84 – – BoxNullableValue .NET 10.0 9.213 ns 1.00 24 B 1.00 BoxNullableValue .NET 11.0 4.126 ns 0.45 24 B 1.00 FormatNullableInt .NET 10.0 9.583 ns 1.00 24 B 1.00 FormatNullableInt .NET 11.0 1.987 ns 0.21 – 0

dotnet/runtime#122167 expands nullable boxing inside the JIT, exposing the temporary box to escape analysis; previously, a runtime helper hid it. For a null input, there’s no allocation on either version, because nothing gets boxed. And on both versions, BoxNullableValue returns the boxed object, meaning the object escapes, so the 24-byte allocation remains. However, for FormatNullableInt, the JIT in .NET 11 can now see that the temporary 24-byte box doesn’t escape and eliminates that heap allocation entirely.

Escape analysis improved further for enumerators, through a mechanism called Conditional Escape Analysis (CEA). Support for CEA was introduced in .NET 10, but .NET 11 extends the set of patterns that this analysis can safely recognize. The existing escape analysis asks whether a reference created by an allocation can flow somewhere the JIT can no longer track, such as an unknown call. If it can, the object must remain on the heap. That analysis is necessarily conservative and largely flow-insensitive: if an object might be passed to an interface call on any path, it doesn’t try to prove that the path containing that call is mutually exclusive with the path containing the allocation.

Unfortunately, that’s exactly what GDV produces when it optimizes a foreach over an IEnumerable<T>. As noted earlier, GDV turns an interface call into a type check with two branches: a fast branch for the likely collection type and a fallback branch containing the original interface call. Devirtualization and inlining along the fast branch will often reveal an enumerator allocation for the collection type, while later enumerator guards retain fallback calls such as IEnumerator<T>.MoveNext. The existing analysis sees those calls and concludes that the locally allocated enumerator might escape. CEA instead records the relationship between the fast-path allocation and the enumerator local tested by the later guards. If every apparent escape occurs only behind a failed type check, the JIT can clone the region into a hot version where those checks are known to succeed. In that clone, the object can’t reach the fallback calls, so it can be stack-allocated and often promoted into separate scalar locals. The original region remains as the general slow path.

One case .NET 10 didn’t handle, though, was a GetEnumerator() implementation that returns the result of another GetEnumerator() call. A collection expression converted to IEnumerable<int>, for example, uses a compiler-generated read-only-array wrapper with exactly this structure: the wrapper’s GetEnumerator() delegates to the underlying array’s GetEnumerator. With dotnet/runtime#122946, the JIT in .NET 11 handles this “chaining”:

// dotnet run -c Release -f net10.0 --filter "*" --runtimes net10.0 net11.0

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;

BenchmarkSwitcher.FromAssembly(typeof(Benchmarks).Assembly).Run(args);

[MemoryDiagnoser(false), HideColumns("Job", "Error", "StdDev", "Median", "RatioSD")]
public class Benchmarks
{
    private static readonly IEnumerable<int> s_readOnlyStatic = [1, 2, 3, 4, 5];
    private readonly IEnumerable<int> _readOnlyInstance = [1, 2, 3, 4, 5];

    [Benchmark]
    public int ReadOnlyStatic()
    {
        int sum = 0;
        foreach (int item in s_readOnlyStatic) sum += item;
        return sum;
    }

    [Benchmark]
    public int ReadOnlyInstance()
    {
        int sum = 0;
        foreach (int item in _readOnlyInstance) sum += item;
        return sum;
    }
}

Method Runtime Mean Ratio Allocated Alloc Ratio ReadOnlyStatic .NET 10.0 2.665 ns 1.00 – – ReadOnlyStatic .NET 11.0 2.666 ns 1.00 – – ReadOnlyInstance .NET 10.0 13.874 ns 1.00 32 B 1.00 ReadOnlyInstance .NET 11.0 2.674 ns 0.19 – 0

ReadOnlyStatic, whose static readonly field the JIT can effectively treat as a constant, was already optimized in .NET 10. In .NET 11, the instance-field case also loses its 32-byte enumerator allocation and converges on the same throughput.

dotnet/runtime#121918 from @MichalPetryka fixes another way an address could unnecessarily make an object appear to escape. The IL constrained. prefix lets one generic callvirt sequence work for both value types and reference types: it can avoid boxing a value type, while for a reference type it dereferences the receiver and performs normal virtual dispatch. ObjectEqualityComparer<T>.Equals, used in the following benchmark by EqualityComparer<T>.Default, contains such a call to value.Equals(other). The receiver was represented as an indirect read through the address of a local. Merely taking that address marked the local as exposed, preventing the newly allocated Value from being considered for stack allocation. The receiver is now represented as a direct value load instead, and the 24-byte heap allocation disappears.

// dotnet run -c Release -f net10.0 --filter "*" --runtimes net10.0 net11.0

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;
using System.Collections.Generic;

BenchmarkSwitcher.FromAssembly(typeof(Benchmarks).Assembly).Run(args);

[MemoryDiagnoser(false), HideColumns("Job", "Error", "StdDev", "Median", "RatioSD")]
public class Benchmarks
{
    private static readonly Value s_other = new(42);

    [Benchmark]
    public bool Equals() => EqualityComparer<Value>.Default.Equals(new Value(42), s_other);

    private sealed class Value(int value)
    {
        private readonly int _value = value;

        public override bool Equals(object? obj) => obj is Value other && _value == other._value;

        public override int GetHashCode() => _value;
    }
}

Method Runtime Mean Ratio Allocated Alloc Ratio Equals .NET 10.0 3.874 ns 1.00 24 B 1.00 Equals .NET 11.0 1.786 ns 0.46 – 0

While CEA can move a non-escaping object off the GC heap, sometimes the JIT can go further and prove an allocation need not exist at all. Generic code provides a common source of such opportunities through boxing. For example, the ArgumentNullException.ThrowIfNull method accepts an object value. That means when you have a method like this:

static void Test<T>(T value)
{
    ArgumentNullException.ThrowIfNull(value);
    ...
}

when T is constrained to a non-nullable struct, boxing is incurred, in order to pass value as object. ThrowIfNull here is a nop if value is non-null (since the method is simply if (value is null) Throw();), and previous releases successfully optimized away that boxing in optimized code. However, in Tier 0, that optimization wasn’t applied, and ThrowIfNull would end up allocating. While this wouldn’t negatively impact steady-state throughput, it would lead to annoying noise in profiling, as well as additional overhead during startup, where such use wasn’t yet promoted out of Tier 0. In .NET 11, dotnet/runtime#129392 adds support for this in Tier 0 as well.

On the virtual-dispatch side, multiple PRs contribute to improving generic virtual methods (GVMs). dotnet/runtime#120866 from @hez2010 stops eagerly spilling ldvirtftn call targets into a temporary, and lets generic virtual target resolution move ahead of argument setup when legal. dotnet/runtime#122023 from @hez2010 then enables the JIT to devirtualize non-shared GVMs, carrying the generic context needed to turn the indirect dispatch into a direct, and potentially inlineable, call. And dotnet/runtime#128702 from @hez2010 extends that support to shared GVMs and default interface implementations that require an instantiating stub. These optimizations can increase total code size when the newly direct calls are inlined, but that’s generally the desired trade: more of the actual work becomes visible to the optimizer. Consider the following benchmark:

// dotnet run -c Release -f net10.0 --filter "*" --runtimes net10.0 net11.0

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;
using System.Runtime.CompilerServices;

BenchmarkSwitcher.FromAssembly(typeof(Benchmarks).Assembly).Run(args);

[MemoryDiagnoser(false), HideColumns("Job", "Error", "StdDev", "Median", "RatioSD")]
public class Benchmarks
{
    [Benchmark]
    public int NonShared() => ((IProcessor)new Processor()).SizeOf(42);

    [Benchmark]
    public int Shared() => ((IProcessor)new Processor()).SizeOf("hello");

    private interface IProcessor
    {
        int SizeOf<T>(T value);
    }

    private sealed class Processor : IProcessor
    {
        public int SizeOf<T>(T value) => Unsafe.SizeOf<T>();
    }
}

Casting a freshly allocated Processor to IProcessor incurs an interface generic virtual call in the IL, but the JIT is now able to see the receiver’s exact type, even in the shared string case, such that .NET 11 devirtualizes and inlines both calls. That in turn exposes Unsafe.SizeOf<T>() as a constant and proves that the short-lived Processor doesn’t need to be allocated at all.

Method Runtime Mean Ratio Allocated Alloc Ratio NonShared .NET 10.0 6.678 ns 1.00 24 B 1.00 NonShared .NET 11.0 1.764 ns 0.26 – 0 Shared .NET 10.0 7.166 ns 1.00 24 B 1.00 Shared .NET 11.0 1.764 ns 0.25 – 0

Building on that, dotnet/runtime#123183 from @hez2010 enables ReadyToRun compilation to resolve and devirtualize more non-shared generic virtual calls that would otherwise remain indirect, and dotnet/runtime#130202 from @hez2010 extends that support to NativeAOT. NativeAOT represents some generic virtual targets as “fat pointers” (pointers that are more than just an address, typically an address and associated metadata, and that in this case carry both a code address and generic context); by deferring that transformation until after exact-type devirtualization has had a chance to run, the JIT can turn an interface call site with a single known target to a non-shared GVM into a direct call that may then be inlined.

Type information also needs to survive the transformations the JIT performs internally. If the JIT spills a reference expression into a temporary while restructuring a tree, losing the expression’s exact class information can turn a call that was devirtualizable back into an opaque virtual call. That’s what happens here in .NET 10: Value gets boxed and SetValue is invoked through IValue. dotnet/runtime#128485 from @hez2010 preserves the class handle and exactness on the temporary. With that information still available, .NET 11 devirtualizes and inlines the call, eliminating the box and its 24-byte allocation.

Separately, dotnet/runtime#127433 relaxes the inliner’s budget heuristics for callees on [Intrinsic] types like Span and Vector. These types intentionally expose many small, composable methods that serve as gateways to JIT-recognized operations. If a wrapper remains as a call, the caller pays the call overhead and optimizations around it see an opaque boundary. If it inlines, the importer can replace its body with an intrinsic node and optimize that node together with the surrounding indexing, bounds checks, and vector operations. Giving such wrappers more favorable budgeting therefore keeps more of them inlineable and exposes more of the actual operation to the rest of the optimizer.

One of the core abstraction-enabling mechanisms in .NET is delegates: they let us pass around objects representing functions to be invoked, carrying with them associated required state. Deabstraction enables avoiding paying for the overheads associated with delegates in some cases. For the rest, we still want those delegates to be as cheap as possible. dotnet/runtime#99200 from @MichalPetryka simplifies CoreCLR’s delegate representation, removing one pointer-sized field from every delegate object. That saves 8 bytes per delegate in a 64-bit CoreCLR process. dotnet/runtime#129304 from @MichalPetryka improves Native AOT’s delegate layout separately by reordering its existing four fields so related values are adjacent. The updated layouts also give equality and hash-code operations more direct access to the method identity they need.

// dotnet run -c Release -f net10.0 --filter "*" --runtimes net10.0 net11.0

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;

BenchmarkSwitcher.FromAssembly(typeof(Benchmarks).Assembly).Run(args);

[MemoryDiagnoser(false), HideColumns("Job", "Error", "StdDev", "Median", "RatioSD")]
public class Benchmarks
{
    private static readonly Target s_target = new();
    private static readonly Func<int> s_first = s_target.GetValue;
    private static readonly Func<int> s_second = s_target.GetValue;

    [Benchmark]
    public Func<int> ClosedInstance() => s_target.GetValue;

    [Benchmark]
    public bool DelegateEquals() => s_first.Equals(s_second);

    [Benchmark]
    public int DelegateGetHashCode() => s_first.GetHashCode();

    private sealed class Target
    {
        public int GetValue() => 42;
    }
}

Method Runtime Mean Ratio Allocated Alloc Ratio ClosedInstance .NET 10.0 7.395 ns 1.00 64 B 1.00 ClosedInstance .NET 11.0 6.844 ns 0.93 56 B 0.88 DelegateEquals .NET 10.0 3.254 ns 1.00 – – DelegateEquals .NET 11.0 2.215 ns 0.68 – – DelegateGetHashCode .NET 10.0 5.623 ns 1.00 – – DelegateGetHashCode .NET 11.0 3.741 ns 0.67 – –

dotnet/runtime#129410 from @MichalPetryka follows up on the CoreCLR layout by placing the target object and method pointer next to each other. Those are commonly consumed together during invocation, and the adjacency enables paired loads on architectures such as Arm64.

Runtime Async

For more than a decade, async and await have let us write asynchronous code that looks remarkably similar to synchronous code: we can put a try/catch around an await, use local variables on either side of it, return a value and generally reason about the method in source order. When execution reaches an await for something that isn’t yet complete, however, the method can’t simply leave its current stack frame in place and wait for the operation to finish. The thread needs to be freed up to do other work, while the work after the await, including whatever local state it will need later, must survive somewhere. In C#, the compiler has traditionally been responsible for transforming the method into a representation that enables that continuation.

I went into the history and mechanics of that transformation in How async/await really works. The very short version is that the compiler traditionally replaces an async method with a small entry method and a generated state machine whose MoveNext method contains the transformed user code. Parameters, locals that need to survive an incomplete await, spilled expression values, awaiters, the current state number, and a method builder all become fields on a heap-allocated object. The generated MoveNext method runs the user’s code until an awaiter reports that it isn’t yet complete. It stores enough information to know where and with what values to resume, registers MoveNext as the continuation, and returns. When the operation completes, MoveNext is invoked again, jumps to the right location based on the saved state number (think goto and a label), retrieves the result from a value-producing awaiter, and continues. If every awaiter is already complete, MoveNext can run all the way through synchronously. When the method completes or throws, the builder publishes the result, cancellation, or exception through the returned Task, Task<T>, ValueTask, or ValueTask<T> (or, in the rare case, a custom task-like type).

For example, consider this tiny method:

static async Task<int> ReadLengthAsync(Stream stream, CancellationToken cancellationToken)
{
    var buffer = new byte[4096];
    int bytesRead = await stream.ReadAsync(buffer, 0, buffer.Length, cancellationToken);
    return bytesRead;
}

While the code that gets generated for this changes over time and differs between debug and release builds, the lowering by the C# compiler has looked something like this:

[AsyncStateMachine(typeof(<ReadLengthAsync>d__0))]
static Task<int> ReadLengthAsync(Stream stream, CancellationToken cancellationToken)
{
    <ReadLengthAsync>d__0 stateMachine = default;
    stateMachine.builder = AsyncTaskMethodBuilder<int>.Create();
    stateMachine.state = -1;
    stateMachine.stream = stream;
    stateMachine.cancellationToken = cancellationToken;
    stateMachine.builder.Start(ref stateMachine);
    return stateMachine.builder.Task;
}

struct <ReadLengthAsync>d__0 : IAsyncStateMachine
{
    public int state;
    public AsyncTaskMethodBuilder<int> builder;
    public Stream stream;
    public CancellationToken cancellationToken;

    private TaskAwaiter<int> awaiter;

    public void MoveNext()
    {
        int result;
        try
        {
            TaskAwaiter<int> localAwaiter;

            if (state != 0)
            {
                byte[] buffer = new byte[4096];
                localAwaiter = stream.ReadAsync(buffer, 0, buffer.Length, cancellationToken).GetAwaiter();
                if (!localAwaiter.IsCompleted)
                {
                    state = 0;
                    awaiter = localAwaiter;
                    builder.AwaitUnsafeOnCompleted(ref localAwaiter, ref this);
                    return;
                }
            }
            else
            {
                localAwaiter = awaiter;
                awaiter = default;
                state = -1;
            }

            result = localAwaiter.GetResult();
        }
        catch (Exception e)
        {
            state = -2;
            builder.SetException(e);
            return;
        }

        state = -2;
        builder.SetResult(result);
    }
}

That’s quite a lot of generated code for three lines of C#. The compiler has to make decisions before the program runs about the state-machine layout, which values might need to survive, how many awaiter fields are required, and how all the suspension points fit into one MoveNext dispatch. The runtime and JIT have optimized the resulting pattern heavily over the years, including combining the task, state machine, continuation, and ExecutionContext into a single allocation, but by the time the JIT sees the IL, the transformation has already happened, leaving it with a very complicated system to try to optimize.

.NET 11 introduces a new way to split that responsibility, a reimplementation of the async/await infrastructure referred to as “runtime async”. Rather than the C# compiler being responsible for the transformation, the JIT is. The C# compiler emits a much smaller suspension-aware IL contract for each eligible async method and marks the method as async in metadata. The runtime and JIT then do the work that depends on runtime knowledge: creating the externally visible Task or ValueTask, recognizing direct async calls, deciding which values are actually alive at each suspension point, laying out continuation objects, and generating the control flow that suspends and resumes the method. Effectively, the transformation moves from C# to the runtime, where more information is available to optimize it.

The programming model hasn’t changed. This is still C# async/await; await still obeys the awaiter pattern, exceptions and cancellation still surface through the returned task-like object, ConfigureAwait still has its usual meaning, synchronous completion is still synchronous completion, and on and on. An explicit goal for the feature has been 100% behavioral compatibility: whether an async method is lowered by the language compiler or by the runtime is an implementation detail, and any observable semantic difference is a bug.

In .NET 11, application code opts in with a compiler feature switch:

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>net11.0</TargetFramework>
    <Features>$(Features);runtime-async=on</Features>
  </PropertyGroup>
</Project>

Note that there’s no new C# syntax involved, so LangVersion=preview isn’t required, nor is EnablePreviewFeatures. While this is opt-in at the application layer, most of the in-box shared framework is already built this way for .NET 11. The async/await performance goal for .NET 11 is parity with .NET 10, and in general runtime async is already as good as or better than the older implementation in many important paths. It isn’t yet fully optimized, though, and there are known cases where it still produces less efficient code. I’d encourage you to experiment in .NET 11 with opting-in your applications and services; just make sure to measure. My hope is that it’ll be on by default starting in .NET 12.

Moving the transformation from the C# compiler to the runtime has the added benefit of reducing binary size. As noted, the traditional lowering emits an entry method, a generated state-machine type, fields for captured state, and a MoveNext body, for every async method. Runtime async leaves a much smaller method body for the runtime to transform. The following tiny app contains ten Task<int>-returning async methods, each awaiting the next, and compiles the same source once with compiler lowering and once with runtime async:

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net11.0</TargetFramework>
    <AssemblyName>SizeProbe</AssemblyName>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
    <Features Condition="'$(RuntimeAsync)' == 'true'">$(Features);runtime-async=on</Features>
  </PropertyGroup>
</Project>

// dotnet build -c Release -p:RuntimeAsync=false -o classic --no-incremental; dotnet build -c Release -p:RuntimeAsync=true -o runtime --no-incremental; Get-Item .\classic\SizeProbe.dll, .\runtime\SizeProbe.dll | Select-Object Directory, Length

Console.WriteLine(await Benchmarks.Layer0());

public class Benchmarks
{
    public static async Task<int> Layer0() => await Layer1();
    private static async Task<int> Layer1() => await Layer2();
    private static async Task<int> Layer2() => await Layer3();
    private static async Task<int> Layer3() => await Layer4();
    private static async Task<int> Layer4() => await Layer5();
    private static async Task<int> Layer5() => await Layer6();
    private static async Task<int> Layer6() => await Layer7();
    private static async Task<int> Layer7() => await Layer8();
    private static async Task<int> Layer8() => await Layer9();

    private static async Task<int> Layer9()
    {
        await Task.Yield();
        return 42;
    }
}

Lowering SizeProbe.dll Ratio Compiler 10,752 bytes 1.00 Runtime async 5,632 bytes 0.52

For a method such as:

static async Task<int> CallerAsync() => await CalleeAsync();

with runtime async enabled, the C# compiler generates IL like the following:

; MSIL
.method private hidebysig static
    class System.Threading.Tasks.Task`1<int32> CallerAsync() cil managed async
{
    call class System.Threading.Tasks.Task`1<int32> CalleeAsync()
    call int32 System.Runtime.CompilerServices.AsyncHelpers::Await<int32>(
        class System.Threading.Tasks.Task`1<int32>)
    ret
}

There is no generated <CallerAsync>d__0 type, no IAsyncStateMachine, no MoveNext, no AsyncTaskMethodBuilder<int>, and no AsyncStateMachineAttribute. Previously, async on a C# method evaporated at compile time. Now, the method has a new MethodImpl async bit, represented in IL assembly syntax by that async modifier, and the body calls helpers in System.Runtime.CompilerServices.AsyncHelpers.

At first glance the ret looks impossible because the declared signature returns Task<int> while the value on the IL evaluation stack is an int. This clearly isn’t a normal calling convention. The VM can give a Task-returning method two related identities, or MethodDescs, where one has the normal signature the rest of managed code sees, Task<int> CallerAsync(). The other is the AsyncCall variant, which effectively returns int and has an implicit channel for a continuation. Both refer to the same logical method and metadata token, but they have different calling conventions and different jobs. If regular managed code invokes CallerAsync, the VM-generated outer thunk preserves the public contract and returns a Task<int>. If another runtime async method directly awaits it, the JIT can instead call the AsyncCall variant and receive the result directly when the call completes synchronously, or a continuation when it suspends. In other words, it can hand back the T directly and avoid allocating a Task<T>.

That pairing works in both directions. For a method compiled with runtime async, the AsyncCall variant owns the generated (newly compact) IL while the public Task-returning entry point is an adapter thunk; for a traditionally compiled method, the public method owns its usual IL while the VM can create an AsyncCall adapter around it. That means runtime async code remains able to await existing libraries and code compiled by older compilers, a critical capability for our goal of 100% compat. The largest wins naturally appear as more of an async call chain is compiled with runtime async.

This is where the JIT gets an opportunity that simply didn’t exist when every boundary was already expressed as a task and a generated state machine. Suppose A awaits B, which awaits C:

static async Task<int> A(bool yield) => await B(yield);
static async Task<int> B(bool yield) => await C(yield);
static async Task<int> C(bool yield)
{
    if (yield)
        await Task.Yield();

    return 42;
}

Traditionally, each method has its own compiler-generated state machine and its own task-like result. C suspends and eventually completes its task, which wakes B‘s state machine; B then completes its task, which wakes A‘s state machine; and A completes the root task observed by the caller. There has been an enormous amount of work done over the years to reduce the costs of those objects and transitions.

With runtime async, the importer recognizes the adjacent pattern of “call a Task-returning method, then await that task.” In the simple case it can call the callee’s AsyncCall variant instead. When yield is false and C completes synchronously, the int flows back through B and A as a plain value, and only the outermost boundary needs to turn it into the Task<int> promised to the original caller. When yield is true and C suspends, the runtime links continuation state for the chain and eventually resumes it without requiring an intermediate Task<int> at every directly fused edge. The Task contract hasn’t vanished, it just moved to the place where a Task is actually needed.

Runtime async doesn’t make every asynchronous operation allocation-free, though. Rather, it gives the JIT enough information to avoid materializing some task objects that existed only to carry a result from one async method directly into the next. If a consumer stores the task in a collection, manually hooks up a continuation, or otherwise observes the task as an object, that object is still needed. The optimization is about not paying for boundaries that aren’t observably boundaries.

The impact is already visible with just two layers:

// dotnet run -c Release -f net11.0 --filter "*"
// The project also needs the `runtime-async=on` feature switch set.

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Configs;
using BenchmarkDotNet.Running;
using System.Runtime.CompilerServices;

BenchmarkSwitcher.FromAssembly(typeof(Benchmarks).Assembly).Run(args);

[MemoryDiagnoser(false)]
[GroupBenchmarksBy(BenchmarkLogicalGroupRule.ByCategory)]
[HideColumns("Job", "Error", "StdDev", "Median", "RatioSD")]
public class Benchmarks
{
    private static readonly Task<int> s_completed = Task.FromResult(42);

    [Benchmark(Baseline = true), BenchmarkCategory("Completed")]
    public Task<int> ClassicCompleted() => ClassicCompletedOuter();

    [Benchmark, BenchmarkCategory("Completed")]
    public Task<int> RuntimeCompleted() => RuntimeCompletedOuter();

    [Benchmark(Baseline = true), BenchmarkCategory("Yielding")]
    public Task<int> ClassicYielding() => ClassicYieldingOuter();

    [Benchmark, BenchmarkCategory("Yielding")]
    public Task<int> RuntimeYielding() => RuntimeYieldingOuter();

    [RuntimeAsyncMethodGeneration(false)]
    private static async Task<int> ClassicCompletedOuter() => await ClassicCompletedInner();

    [RuntimeAsyncMethodGeneration(false)]
    private static async Task<int> ClassicCompletedInner() => await s_completed;

    private static async Task<int> RuntimeCompletedOuter() => await RuntimeCompletedInner();

    private static async Task<int> RuntimeCompletedInner() => await s_completed;

    [RuntimeAsyncMethodGeneration(false)]
    private static async Task<int> ClassicYieldingOuter() => await ClassicYieldingInner();

    [RuntimeAsyncMethodGeneration(false)]
    private static async Task<int> ClassicYieldingInner()
    {
        await Task.Yield();
        return 42;
    }

    private static async Task<int> RuntimeYieldingOuter() => await RuntimeYieldingInner();

    private static async Task<int> RuntimeYieldingInner()
    {
        await Task.Yield();
        return 42;
    }
}

namespace System.Runtime.CompilerServices
{
    [AttributeUsage(AttributeTargets.Method)]
    internal sealed class RuntimeAsyncMethodGenerationAttribute(bool runtimeAsync) : Attribute
    {
        public bool RuntimeAsync => runtimeAsync;
    }
}

Method Mean Ratio Allocated Alloc Ratio ClassicCompleted 21.221 ns 1.00 144 B 1.00 RuntimeCompleted 6.151 ns 0.29 0 B 0.00 ClassicYielding 254.139 ns 1.00 248 B 1.00 RuntimeYielding 116.927 ns 0.46 168 B 0.68

The synchronously completing chain is more than 3x faster and avoids both intermediate task allocations. Even after a real suspension, the same two-layer chain takes less than half the time and allocates 80 fewer bytes.

Exception handling amplifies the difference. Again consider an async method A calling an async method B calling an async method C. The transformation generated by the C# compiler of each method results in a try/catch block around the whole body of the MoveNext method so that any unhandled exception can be stored into the returned Task. Let’s say code in C throws an unhandled exception. That’s then caught by this manufactured catch block and stored into the Task returned to B. The awaiter in B then retrieves that exception from the Task object and throws it. It’s then caught by B‘s generated catch and stored into its Task. And so on. An exception crossing ten such async helpers can therefore be thrown, caught, and stored ten times even though none of the source methods has an explicit handler. That is super expensive. But runtime async doesn’t need to re-enter a pass-through frame with no handler. On the synchronous path the exception unwinds through the fused calls normally, and after a real suspension, one dispatch-loop catch walks past continuation records that have no handler and faults the observable root task once.

The following benchmark measures both a fully synchronous throw and an exception after one real Task.Yield suspension. It uses a compiler-recognized per-method escape hatch (RuntimeAsyncMethodGeneration) so that the classic and runtime async methods run in the same process on the same .NET 11 runtime and differ only in how the compiler lowers them. (Note that this attribute is experimental and isn’t a public API exposed from the core libraries; as with other attributes known to the C# compiler, it recognizes them by name and signature.)

// dotnet run -c Release -f net11.0 --filter "*"
// The project also needs the `runtime-async=on` feature switch set.

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;
using System.Runtime.CompilerServices;

BenchmarkSwitcher.FromAssembly(typeof(Benchmarks).Assembly).Run(args);

[MemoryDiagnoser(false), HideColumns("Job", "Error", "StdDev", "Median", "RatioSD")]
public class Benchmarks
{
    [Params(1, 10, 30)]
    public int Depth;

    [Params(false, true)]
    public bool Yield;

    [Benchmark(Baseline = true)]
    public int Classic() => Invoke(ClassicThrowAsync(Depth));

    [Benchmark]
    public int Runtime() => Invoke(RuntimeThrowAsync(Depth));

    private static int Invoke(Task<int> task)
    {
        try
        {
            return task.GetAwaiter().GetResult();
        }
        catch (InvalidOperationException)
        {
            return -1;
        }
    }

    [RuntimeAsyncMethodGeneration(false)]
    private async Task<int> ClassicThrowAsync(int depth)
    {
        if (depth == 0)
        {
            if (Yield) await Task.Yield();
            throw new InvalidOperationException("uh oh");
        }

        return await ClassicThrowAsync(depth - 1);
    }

    private async Task<int> RuntimeThrowAsync(int depth)
    {
        if (depth == 0)
        {
            if (Yield) await Task.Yield();
            throw new InvalidOperationException("uh oh");
        }

        return await RuntimeThrowAsync(depth - 1);
    }
}

namespace System.Runtime.CompilerServices
{
    [AttributeUsage(AttributeTargets.Method)]
    internal sealed class RuntimeAsyncMethodGenerationAttribute(bool runtimeAsync) : Attribute
    {
        public bool RuntimeAsync => runtimeAsync;
    }
}

Depth Yield Method Mean Ratio Allocated Alloc Ratio 1 False Classic 4.727 μs 1.00 1.6 KB 1.00 1 False Runtime 3.558 μs 0.75 1.16 KB 0.72 1 True Classic 6.308 μs 1.00 1.68 KB 1.00 1 True Runtime 8.211 μs 1.30 1.42 KB 0.85 10 False Classic 19.469 μs 1.00 15.13 KB 1.00 10 False Runtime 5.885 μs 0.30 2.13 KB 0.14 10 True Classic 24.923 μs 1.00 15.63 KB 1.00 10 True Runtime 6.122 μs 0.25 2.88 KB 0.18 30 False Classic 51.302 μs 1.00 84.2 KB 1.00 30 False Runtime 10.721 μs 0.21 5.71 KB 0.07 30 True Classic 65.974 μs 1.00 85.53 KB 1.00 30 True Runtime 11.254 μs 0.17 7.72 KB 0.09

Runtime async supports Task, Task<T>, ValueTask, and ValueTask<T> as method return types, but as of today it doesn’t support async void, async iterators, or arbitrary custom task-like return types with custom builders; those continue to use the traditional compiler transformation. For ValueTask<T>, the existing reasons to use the type still apply. A ValueTask<T> can carry a result directly, wrap a Task<T>, or refer to an IValueTaskSource<T>. That’s made it useful for APIs where synchronous completion is common enough that avoiding a Task allocation outweighs the larger return value and the more restrictive consumption rules, or where asynchronous completion can have its costs amortized via a reusable backing object. Runtime async then addresses some of the scenarios that would have led developers to use ValueTask<T>. Does that mean everyone should stop using ValueTask<T>? No. Choosing Task versus ValueTask remains an API design decision based on completion patterns, allocation sensitivity, call frequency, and how consumers need to use the result. Write the return type that makes sense for the API, then let the compiler, VM, and JIT optimize it as best they can.

Workloads with many layers of small async methods can benefit the most from runtime async, because those layers are exactly where intermediate tasks and state machines often accumulate. Shared framework code, for example, is full of this pattern: a public method validates arguments and awaits a private helper, which awaits a transport helper, which awaits an operating-system operation. Application services similarly compose authentication, retry, logging, serialization, and I/O helpers. Runtime async can make the source-level decomposition cheaper without asking the developer to flatten the code into one giant method in order to avoid “implementation detail” costs.

The work required to reach this point has been extensive. A GitHub search of the runtime async tracking label on September 14, 2026 returned 235 pull requests, far too many for me to enumerate one by one. So I won’t try; you can peruse that label in your spare time. The work is also not only about direct performance improvements but also about improvements to diagnostics and performance tooling that help you to make better use of async in your own code. When an async method suspends, its physical thread stack unwinds. That method’s continuation might later run on a different thread whose physical stack begins in the thread pool, with the methods that led to the original await nowhere to be found. A sampling CPU profiler can see where the processor is spending time, but without additional information, it can’t reliably connect those traces back through the logical async call chain, making it hard to answer questions about what async call paths were actually costing. Profiling tools like the async profiler in Visual Studio have traditionally reconstructed those chains from events emitted by Task‘s infrastructure, but async-heavy applications can generate enormous volumes of those very chatty events. The resulting overhead easily perturbs the workload being measured, making it all but unusable in production. dotnet/runtime#127238 added a new lightweight async-profiler event stream for .NET 11 and runtime async. Rather than sending every small transition through the eventing system as its own full event, the runtime writes compact records into per-thread buffers, delta-encoding timestamps and instruction pointers and flushing the data in batches. It also puts a small identifiable wrapper frame into the physical stack when invoking a continuation. A profiler can use that frame as an anchor, joining ordinary CPU samples to the logical async call stack represented by the event stream. In some measurements, this new approach added less than 1% overhead and shrank the traced data by an order of magnitude. dotnet/runtime#129043 and a few follow-up PRs extended the same approach to the compiler-generated state machines used by existing async code. Thus this isn’t useful only to applications that opt into runtime async; tooling gets one consistent representation across both implementations.

What should you as a developer do differently with runtime async in the picture? Mostly nothing. Keep writing asynchronous code the way you want it to read, and break a large operation into helpers when that makes the code clearer. Use Task by default and choose ValueTask where its API and usage tradeoffs genuinely fit. And don’t contort source code to remove a clean await just because today’s implementation might allocate an intermediate Task. The lowering strategy should “just work” as an implementation detail, preserve behavior, and make existing source get better as the runtime improves.

Bounds Checks

C# is a memory-safe language. Accesses to arrays, strings, and spans are guaranteed by the runtime to be in-bounds; if you try to access someArray[i], someString[i], or someSpan[i] with an index less than 0 or greater than or equal to the length of the array/string/span, you’ll get an exception, not silently corrupted memory or a process crash. The runtime guarantees that all permitted accesses are within bounds, and that means it needs to be able to prove the access is in bounds. The main method the JIT has for achieving that is by injecting code that performs a bounds check, as if instead of:

int[] array = ...;
int value = array[i];

you’d written:

int[] array = ...;
if ((uint)i >= array.Length) throw new IndexOutOfRangeException();
int value = array[i];

At the assembly level, a bounds check looks something like:

; x64
cmp ecx, dword ptr [rax+8]        ; compare index with array length
jae THROW                         ; unsigned index >= length
mov edx, dword ptr [rax+rcx*4+16] ; load the element

The JIT could just inject such code on every access and call it a day, but such code adds overhead, so the JIT works to elide those checks and that overhead wherever it can prove the index is valid. Proving an index is valid means the JIT needs to be able to see from other evidence that it couldn’t possibly be out of bounds.

The quintessential example of that is a for loop over the full contents of an array or span:

for (int i = 0; i < array.Length; i++)
{
    Use(array[i]);
}

The JIT recognizes from this idiom that, within the loop body, i is guaranteed to be in the range [0, array.Length), and avoids emitting the bounds check for the array[i] access. The JIT has long handled this particular case. Other cases, not so much. Bounds-check elimination has improved in virtually every .NET release; more recent releases added range propagation for derived expressions (.NET 7 and .NET 8 saw significant improvements here), SSA-based reasoning (.NET 9), and better handling of Span<T>, whose length sits in a field rather than an object header, complicating tracking. Each year, the developers contributing to the JIT find new patterns that were being missed, that show up in the wild, and that are fixable. .NET 11 improves several such patterns.

Range analysis in the JIT tracks intervals for each variable, an upper bound and a lower bound. For example, taking the true branch of x < 5 gives the range for x in that branch an upper bound of 4 while taking the true branch of x > 2 makes the lower bound 3. What about x != 5? On the true edge, we know x isn’t 5, and if the current range for x is [5, 10], then we know the range must actually be [6, 10]… the lower bound can be tightened because the only value at the lower end is excluded. Similarly, if the range is [0, 5], an x != 5 assertion tells us the range is actually the narrower [0, 4]. Or, at least, that’s what you’d hope it would do. The JIT had this relevant comment:

// We have a != assertion, but it doesn't tell us much about the interval. So just skip it.
continue;

In .NET 11, dotnet/runtime#121273 replaces that logic with productive reasoning. It checks whether the excluded constant is at either edge of the currently tracked range, adding in the new insights if so. C# list patterns, introduced in C# 11, generate just such comparison sequences. For example, the pattern name is [] or [':'] or [':', not ':', ..] lowers to something like this:

if (name != null)
{
    int num = name.Length;

    if (num == 0) return true;

    if (num == 1)
    {
        if (name[0] == ':') return true;
    }
    else if (name[0] == ':' && name[1] != ':')
    {
        return true;
    }

    return false;
}

Range analysis then proceeds with something like this:

1. We know that Array.Length is never negative, so it has a range of [0, Array.MaxLength].
2. On the false edge of num == 0, we know that num != 0, so the range is narrowed now to [1, Array.MaxLength].
3. Similarly, on the false edge of num == 1, we know that num != 1, so the range is narrowed now to [2, Array.MaxLength].
4. We then access name[0] and name[1], both of which are guaranteed in bounds based on the lower bound of 2 that was established.

Without the != constant tightening, that narrowing wouldn’t happen, and the bounds checks in step 4 couldn’t be elided. Thankfully, they now can be in .NET 11. Consider this example:

// dotnet run -c Release -f net10.0 --filter "*" --runtimes net10.0 net11.0

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;
using System.Runtime.CompilerServices;

BenchmarkSwitcher.FromAssembly(typeof(Benchmarks).Assembly).Run(args);

[DisassemblyDiagnoser, HideColumns("Job", "Error", "StdDev", "Median", "RatioSD")]
public class Benchmarks
{
    private string[] _inputs = ["", ":", ":x", "abc", ":ab", "x", "ab:cd"];

    [Benchmark]
    public int ClassifyAll()
    {
        int total = 0;
        foreach (string s in _inputs) total += Classify(s);
        return total;
    }

    [MethodImpl(MethodImplOptions.NoInlining)]
    private static int Classify(ReadOnlySpan<char> name) =>
        name switch
        {
            [] => 0,
            [':'] => 1,
            [':', not ':', ..] => 10 + name[0] + name[1],
            _ => 3
        };
}

In .NET 10, we can see the call to CORINFO_HELP_RNGCHKFAIL at the bottom of the method. That’s the tell-tale sign there was at least one bounds check in the method. With .NET 11, that sign is removed.

; Arm64
--- .NET 10
+++ .NET 11
@@ -10,17 +10,15 @@
             beq     G_M000_IG08

 G_M000_IG04:
-            ldrh    w2, [x0]
-            cmp     w2, #58
+            ldrh    w1, [x0]
+            cmp     w1, #58
             bne     G_M000_IG06

 G_M000_IG05:
-            cmp     w1, #1
-            bls     G_M000_IG11
             ldrh    w0, [x0, #0x02]
             cmp     w0, #58
             beq     G_M000_IG06
-            add     w0, w2, w0
+            add     w0, w1, w0
             add     w0, w0, #10
             b       G_M000_IG07

@@ -44,8 +42,4 @@
             mov     w0, wzr
             b       G_M000_IG07

-G_M000_IG11:
-            bl      CORINFO_HELP_RNGCHKFAIL
-            brk     #0
-
-; Total bytes of code 112
+; Total bytes of code 96

“Assertion” machinery in the JIT propagates learned facts (like the aforementioned range information) between “basic blocks” (a sequence of instructions with one entry point, one exit point, and no branches into or out of the middle of it), so information established in block A flows to block B if A “dominates” B (meaning the only way to get to B is through A). But what about facts established earlier within the same block? That’s the gap that dotnet/runtime#121527 addresses. Consider this code:

// dotnet run -c Release -f net10.0 --filter "*" --runtimes net10.0 net11.0

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;
using System.Runtime.CompilerServices;

BenchmarkSwitcher.FromAssembly(typeof(Benchmarks).Assembly).Run(args);

[DisassemblyDiagnoser, HideColumns("Job", "Error", "StdDev", "Median", "RatioSD")]
public class Benchmarks
{
    private int[] _arr = new int[512];

    [Benchmark]
    public int RunMany()
    {
        int touched = 0;
        for (int i = 0; i < _arr.Length - 2; i++)
        {
            Test(_arr, i);
            touched++;
        }
        return touched;
    }

    [MethodImpl(MethodImplOptions.NoInlining)]
    private static void Test(int[] arr, int i)
    {
        arr[i] = 0;  // 1: establishes 'i >= 0 && i < arr.Length'
        i++;         // 2: same block
        if (i < arr.Length) arr[i] = 0;  // 3: proven safe from 1's assertion
    }
}

Statements 1, 2, and 3 are all in the same basic block, up to the conditional; after statement 1 executes, if we reach statement 2, the bounds check on statement 1 passed, we know i >= 0 and i < arr.Length, and after statement 2, i becomes i + 1. After the if guard i < arr.Length we know the incremented i is still within bounds. But when the range check pass in the .NET 10 JIT examined statement 3’s bounds check, it saw the assertions propagated from predecessor blocks. Since the assertion from statement 1 is generated within the current block, the range check couldn’t see it. The PR fixed it to walk the current block’s tree in execution order, accumulating assertions as it went. When we reach statement 3’s bounds check, we’ve already walked past statement 1 and picked up its i >= 0 && i < arr.Length assertion.

; Arm64
--- .NET 10
+++ .NET 11
@@ -13,8 +13,6 @@
             ble     G_M000_IG04

 G_M000_IG03:
-            cmp     w1, w2
-            bhs     G_M000_IG05
             str     wzr, [x0, w1, UXTW #2]

 G_M000_IG04:
@@ -25,4 +23,4 @@
             bl      CORINFO_HELP_RNGCHKFAIL
             brk     #0

-; Total bytes of code 68
+; Total bytes of code 60

There are almost an infinite number of things the JIT could look for and special-case. But every special case requires code, maintenance, and, most importantly, compilation time. A “just-in-time” compiler typically runs while the application is running, so the JIT itself must be optimized and spend its limited budget only where there’s a likely payoff. That pushes the developers building it toward patterns that occur in real workloads. One such pattern, often seen in libraries like format decoders, builds a table index with bitwise operations on a byte, for example ((b & 0x03) << 4) | ((b & 0xf0) >> 4). Each masked piece has a tiny upper bound, so the OR of those pieces is always in [0..63], safely in range for e.g. a Base64 alphabet table. Until dotnet/runtime#122263, the JIT often failed to prove that combined bound and left a bounds check on the index. Existing range-check code understood the upper bounds produced by bitwise AND and shifts, but not OR; the change lets the JIT combine the known bounds of both OR operands and remove the remaining array check.

// dotnet run -c Release -f net10.0 --filter "*" --runtimes net10.0 net11.0

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;
using System.Runtime.CompilerServices;

BenchmarkSwitcher.FromAssembly(typeof(Benchmarks).Assembly).Run(args);

[DisassemblyDiagnoser, HideColumns("Job", "Error", "StdDev", "Median", "RatioSD")]
public class Benchmarks
{
    private readonly byte[] _input = new byte[4096];

    [GlobalSetup]
    public void Setup() => new Random(42).NextBytes(_input);

    [Benchmark]
    public int Base64LikeIndex() => Sum(_input);

    [MethodImpl(MethodImplOptions.NoInlining)]
    private static int Sum(ReadOnlySpan<byte> input)
    {
        int sum = 0;
        foreach (byte b in input)
        {
            int index = ((b & 0x03) << 4) | ((b & 0xF0) >> 4);
            sum += "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/="u8[index];
        }

        return sum;
    }
}

The .NET 10 assembly checks the computed index against the 65-byte lookup table on every iteration. In .NET 11, range analysis proves the index is at most 63, so both the comparison and the branch to the range-check failure helper disappear:

; x64
 M01_L00:
        movzx    r9d, byte ptr [rdx+r8]
        mov      r11d, r9d
        and      r11d, 3
        shl      r11d, 4
        and      r9d, 0F0
        sar      r9d, 4
        or       r9d, r11d
-       cmp      r9d, 41
-       jae      short M01_L02
        movzx    r9d, byte ptr [r10+r9]
        add      eax, r9d
        inc      r8d
        cmp      r8d, ecx
        jl       short M01_L00

-M01_L02:
-       call     CORINFO_HELP_RNGCHKFAIL
-       int      3
-
-; Total bytes of code 95
+; Total bytes of code 79

As another example, dotnet/runtime#125056 improves the handling of guards like (uint)i < span.Length that are pervasive in performance-sensitive code. Consider this benchmark:

// dotnet run -c Release -f net10.0 --filter "*" --runtimes net10.0 net11.0

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;
using System.Runtime.CompilerServices;

BenchmarkSwitcher.FromAssembly(typeof(Benchmarks).Assembly).Run(args);

[DisassemblyDiagnoser, HideColumns("Job", "Error", "StdDev", "Median", "RatioSD")]
public class Benchmarks
{
    private int[] _data = Enumerable.Range(0, 512).ToArray();

    [Benchmark]
    public int RunMany()
    {
        int sum = 0;
        for (int i = 0; i < _data.Length; i++)
            sum += Test(_data, i);
        return sum;
    }

    [MethodImpl(MethodImplOptions.NoInlining)]
    private static int Test(Span<int> span, int i)
    {
        if ((uint)i < (uint)span.Length)
        {
            if (i != 0)
                return span[i - 1] + span[i];

            return span[i];
        }

        return 0;
    }
}

Because the comparison is unsigned, (uint)i would be a large positive number if i were negative, making it impossible for (uint)i < (uint)span.Length to be true (since a span’s length is never negative, (uint)span.Length is at most int.MaxValue). Inside the true branch, i is therefore in [0, span.Length - 1]. Previously, the JIT wasn’t always recording the lower bound i >= 0 when it processed the (uint)i < span.Length assertion, and that could leave bounds checks on expressions like i - 1 in place. The fix adds the [0, int.MaxValue - 1] lower bound deduction for the index variable upon entering the true arm of a (uint)i < span.Length check. Combined with the existing range tracking for the upper bound, this gives the JIT a complete picture of i‘s range inside the guarded block.

; Arm64
--- .NET 10
+++ .NET 11
@@ -8,10 +8,8 @@
             cbz     w2, G_M000_IG05

 G_M000_IG03:
-            sub     w3, w2, #1
-            cmp     w3, w1
-            bhs     G_M000_IG09
-            ldr     w1, [x0, w3, UXTW #2]
+            sub     w1, w2, #1
+            ldr     w1, [x0, w1, UXTW #2]
             ldr     w0, [x0, w2, UXTW #2]
             add     w0, w1, w0

@@ -33,8 +31,4 @@
             ldp     fp, lr, [sp], #0x10
             ret     lr

-G_M000_IG09:
-            bl      CORINFO_HELP_RNGCHKFAIL
-            brk     #0
-
-; Total bytes of code 84
+; Total bytes of code 68

Bounds check elision is generally based on forms of range analysis, where the JIT needs to prove that a given index is guaranteed to be within the range of the data structure. But the same range analysis-based facts can prove that other checks are unnecessary. For example, once the JIT knows that an integer is in [0..100], it can prove both that converting it to byte can’t lose data and that multiplying it by 10 can’t overflow. dotnet/runtime#124147 enables the JIT to use such facts to avoid unnecessary branches as part of checked operations. When range analysis proves that the operands are in ranges whose result can’t overflow, making checked a nop, the backend can now emit plain add/multiply/subtract instructions, without the jump to failure, as in the following example:

// dotnet run -c Release -f net10.0 --filter "*" --runtimes net10.0 net11.0

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;
using System.Runtime.CompilerServices;

BenchmarkSwitcher.FromAssembly(typeof(Benchmarks).Assembly).Run(args);

[DisassemblyDiagnoser, HideColumns("Job", "Error", "StdDev", "Median", "RatioSD")]
public class Benchmarks
{
    private readonly int[] _array = new int[99];

    [Benchmark]
    public int ArrayLengthPlusConstant() => AddToLength(_array);

    [MethodImpl(MethodImplOptions.NoInlining)]
    private static int AddToLength(int[] array) => checked(array.Length + 10);

    [Benchmark]
    public int GuardedLengthTimesConstant() => Multiply(_array);

    [MethodImpl(MethodImplOptions.NoInlining)]
    private static int Multiply(Span<int> span)
    {
        if (span.Length >= 100) return 0;
        return checked(span.Length * 10);
    }
}

; Arm64
--- .NET 10
+++ .NET 11
 G_M000_IG02:
             cmp     w1, #100
             bge     G_M000_IG05

 G_M000_IG03:
             mov     w0, #10
-            smull   x0, w1, w0
-            lsr     x2, x0, #32
-            cmp     w2, w0, ASR #31
+            mul     w0, w1, w0
-            bne     G_M000_IG07

 G_M000_IG04:
             ldp     fp, lr, [sp], #0x10
             ret     lr

-G_M000_IG07:
-            bl      CORINFO_HELP_OVERFLOW
-            brk     #0
-
-; Total bytes of code 64
+; Total bytes of code 44

That makes the change broadly applicable: any time you write checked arithmetic on quantities that are inherently bounded, such as collection counts, lengths, or indices constrained by prior comparisons, the JIT now has a chance to prove at compile time that the overflow can’t happen and thus eliminate the run-time check entirely. Building on that range-check work, dotnet/runtime#124184 teaches the JIT to eliminate “narrowing casts” under the same kinds of guards:

// dotnet run -c Release -f net10.0 --filter "*" --runtimes net10.0 net11.0

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;
using System.Runtime.CompilerServices;

BenchmarkSwitcher.FromAssembly(typeof(Benchmarks).Assembly).Run(args);

[DisassemblyDiagnoser, HideColumns("Job", "Error", "StdDev", "Median", "RatioSD")]
public class Benchmarks
{
    private uint _value = 100;

    [Benchmark]
    public byte GuardedNarrowingCast() => Narrow(_value);

    [MethodImpl(Metho

I spent a great chunk of last week at SmashingConf Freiburg and it was lovely as usual. It was great to meet some people I've only ever seen online and see old friends. And I am not going to lie - it was pretty flattering to have people come over to me and challenge me to admit I was the mystery speaker (SmashingConf always has a mystery speaker they only announce on the day). I wasn't, as you all know now, but thank you for thinking I am of that calibre!

Anyway, at the after-party I was chatting to a fun group of people and one of them mentioned how someone else in the group really wanted to ask something to one of the speakers from this year but they were feeling shy and intimidated. I immediately encouraged the situation to happen (and it did)! It helps that I am a little bit shameless after a couple of drinks but facilitating speaker and attendee conversations is something I've been trying really hard to make happen whenever I can. So I brought the speaker in question and a few people to join our group pac-man style and people got to chatting.

This reminded me of when I MCed CSS Day earlier in June. I tried really hard to encourage people to come and chat with the speakers throughout the event. At the speakers' dinner, I asked every single person I was going to present on stage if it was okay if I did so. Of course they said yes - they are all a wonderful bunch of folks.

But this also comes from my experience as a speaker. I arrived at what people call the "speaker circuit" quite recently and I never experienced the wave of feedback that I know speakers used to experience on Twitter before Twitter was ruined. I know it is super intimidating to approach someone who has just been on a stage as you may think you have nothing to add to the conversation. I've been on that side and I still am on that side too. Trust me - even as a speaker or MC I am in awe of many people I met this year and I was so afraid of approaching them and chatting to them only for them to realise I am... just a very simple woman.

I've had events I spoke at that I didn't have a single piece of feedback online which made me wonder how flat it landed. It isn't necessarily an ego thing - but sometimes you wonder if you did okay, if people hated it, etc. I'm only human, gang. But I know I am not the only speaker who has been a bit rattled by silence. I've had a few conversations with other speakers about what to do when there isn't a lot of feedback on a talk. It was good to chat this through as I thought it was a me problem but turns out, after all, no thought is original.

And I need to be better at this myself. Most times, I forget to compliment my own friends who are speaking because, bizarrely, I assume they can read my mind and know that I love their work and I am on the first row cheering them on. I need to live-post more and I need to blog more because I really want people to see the amazing conferences I attend, with fantastic organisers and incredible speakers.

These days, especially if I have an active role in an event, I feel a little bit shy and probably stick around close to people I know more than I should. After I speak on a stage, my brain is rambling on about all the things I know I said incorrectly and all the details I know weren't perfect.

So, if you always wanted to but think it isn't worth telling a speaker you liked their talk, this is your encouragement post to do so. It is always worth it even if it is a small tap and small chat. People put a lot of work into it and they want people to enjoy it and, at least for me, I kinda do want to know if we did alright.