The startup's Postgres survival guide

A guide to preventing Postgres from toppling over.

Over the past half year or so, I’ve been writing an internal doc for our engineers trying to distill two years of Postgres battles into a somewhat cohesive document. While I love the Postgres manual, I find it’s hard to turn to when shit hits the fan because it’s just so darn comprehensive. I thought this might be useful for others and would appreciate feedback (or other tidbits that you’ve learned running Postgres in production).

Before starting Hatchet, while I was familiar with SQL, the extent of my knowledge was basically: if a query is slow, you need an index. That’s the starting point for this doc; I’m going to assume you’re familiar with SQL basics, rows, tables, and know roughly what an index is.

And if Claude is writing all of your queries, this might be a waste of time! I recommend supabase/agent-skills

---

A quick note on ORMs

This guide should still be useful, but you might need to translate some of these tips into your ORM of choice. Lots of optimizations as you scale just aren’t possible with ORMs unless you can break past the abstraction layer and write SQL. You can do this gracefully or non-gracefully; Prisma TypedSQL or equivalents look interesting for this. We use sqlc at Hatchet which gets us very similar behavior; highly recommend if you’re a Go stack.

Table of contents

• The simple stuff: good reads, writes and schemas

  • Writing a good schema
  • Writing good read queries
  • Writing performant joins
  • Compound indexes and aligning ORDER BY to your indexes
  • Writing good write queries
  • Migrations
  • Connection management

• Intermediate: the query planner, bulk updates, and autovacuum

  • Introducing the leakiest of abstractions, the query planner
  • Sometimes it just makes sense to seq scan
  • Writing lots of data
  • Default autovacuum settings can kill your database
  • Other types of bloat

• Some advanced stuff

  • FOR UPDATE SKIP LOCKED
  • Partitioning
  • Tricks for large table migrations

The simple stuff: good reads, writes and schemas

Let’s start with the basics: queries and schemas at low volume.

Writing a good schema

After you’re deployed, schemas are by far the hardest to change moving forward, so it’s worth spending some time on them. I’d recommend building your schema iteratively: start with a rough approximation for your tables and primary keys, then write some queries on those tables based on your application needs. You can approximate this with some questions: Is this a high-read and/or high-write table? What are the most common filters on reads? Which columns am I updating the most?

If you want to be more formal about it, you can look into database normalization into 1NF/2NF/3NF, but I’ve found normal forms to sometimes be at odds with query efficiency and ease of use, which is critical when you’re moving fast—sometimes it’s just easier to dump data into a jsonb column.

My rules of thumb for schemas are:

• Use identity columns (auto-incrementing integers, slightly more performant than bigserial) or built-in UUIDs for primary keys
• Always use timestamptz
• Always use primary keys
• Use foreign keys with cascading deletes for low-volume tables, particularly where database consistency and correctness are important. Careful at higher volume.

Writing good read queries

Let’s start with SELECT queries. A useful—albeit slightly inaccurate—mental model for fast selects is: under the hood, Postgres is either going to find a single row in a table very quickly, or it’s going to read every single row in your table using something called a sequential scan 😞.

It’s going to find a single row very quickly when you filter by:

• An explicit index
• A unique constraint (just a special case of index)
• A primary key (these are automatically indexed in Postgres)

Indexes by default use a btree implementation. It’s most helpful to think of indexes as just another table in Postgres, with data stored in a specific format which is optimized for lookups (more on this later). These trees are great because finding a single row happens in approximately log(n) time, where n is the number of rows in the table—in other words, really fast.

When Postgres can’t use an index, it’ll use something called a sequential scan, or seq scan. Seq scans are much slower than index lookups, but modern databases are so fast at loading rows into memory that you probably won’t even notice at first: seq scans on tables with less than 20k rows are pretty much instant.

Writing performant joins

For inner joins, there’s rarely an argument for not using primary keys as the inner join; it usually speaks to a schema design or normalization problem. Treat ON clauses with the same respect as a WHERE clause—the same principles apply. Use an index.

Compound indexes and aligning ORDER BY to your indexes

Often the first slow query in your application will be a list query across a large table. Something like:

Loading syntax highlighting...

In this case, you can use a compound index—a sensible one might be:

Loading syntax highlighting...

In more complex cases, a good rule of thumb is: the ORDER BY columns should be the last columns in the index, and you should align columns to the ordering in the ORDER BY. Note that Postgres can scan btrees in both directions, so sometimes the DESC is irrelevant—but for compound indexes it’s good practice. More information here.

Writing good write queries

The premise of successful writes is:

1. Keep transactions short. Don’t go querying an external service in the middle of a transaction unless you have a really good reason to.
2. Be careful of the rows you’re locking for writing; in other words, only lock what you need. Every time you update a row, you’re taking out a lock on that row for a short period of time until the transaction commits.

As your system gets busier, you’re going to start noticing the impact of locks more. In particular, you might try to create an index at some point in the future with a simple CREATE INDEX command: turns out this locks your table and prevents inserts and updates! When creating an index on an existing large table, always use CREATE INDEX CONCURRENTLY.

Migrations

Getting really good at writing migrations is an important technical advantage: it helps you iterate much faster and increases your uptime. As a starting point, try to keep migrations additive (in other words, don’t delete or remove columns) and run them in a transaction wherever possible; this will make rollbacks and partial migrations much easier to deal with. As you get more advanced, you can start looking into expand and contract migrations.

The simplest mental model for good migrations is: does this block all of my writes, or does it not? Creating an index without CONCURRENTLY blocks all your writes, so you might see downtime. Generally, operations which call ALTER TABLE should be worth a second look; for example, adding a new check constraint to a very large table can block your writes as well (unless you add it with the NOT VALID keyword).

Connection management

Every time you execute a transaction or query against your database, you’re utilizing a connection. Connections are expensive in a number of dimensions (cpu and memory), and high connection churn can lead to a lot of unnecessary resource waste, so connections should be long-lived. Connection storms (when you start using up a ton of new connections at the same time) can also lead to very hard to debug edge cases related to internal Postgres locks.

Because of all these connection footguns, external connection poolers like pgbouncer are great! If you can’t add this for whatever reason, in-memory connection poolers are a great second option. For example, because Hatchet is open-source, we don’t assume that all user databases use connection poolers, so we use pgxpool (an in-memory connection pool for Go) for this purpose.

Intermediate: the query planner, bulk updates, and autovacuum

Introducing the leakiest of abstractions, the query planner

At a certain point, your queries might become complex enough that a simple index won’t cut it (and you shouldn’t endlessly add indexes to your tables—they come with overhead). The queries might involve many JOIN statements or different types of joins where the correct path for querying the data isn’t clear.

At this point, you will need to concern yourself with the query planner. At best, the query planner is a leaky abstraction. It’s an internal implementation, and you have virtually no control over it, but you have to know its spontaneous and sometimes irrational behavior. It’s like working with an LLM!

The query planner looks at the query you pass in, and it figures out how it should translate your query into a set of internal operations in the database. For example, it might look at your query, and realize that it needs to use an index. In an ideal world, the query planner would know, for every query and set of parameters, the perfect plan to use. But the query planner is operating on limited information, and sometimes it doesn’t pick the best option.

This limited information is the table statistics. You can actually query it directly in Postgres:

Loading syntax highlighting...

These statistics are collected for every ANALYZE. This also happens when autovacuum is run (see below), so more frequent autovacuums also mean that your query statistics will be more up to date. A common reason why your query is behaving improperly is not analyzing frequently enough.

The reason I think it’s useful to view queries as binary—they either seq scan or they don’t seq scan—is: the more you micro-optimize a query, the more of a risk you take that the query planner goes rogue. If you stick to querying by primary keys and indexes, the query planner will have a much easier time.

Let’s say that there’s nothing obviously wrong in your query, but it’s still slow—how do you go about debugging this? Some Postgres database providers (like Google CloudSQL) will sample your queries and save slow ones—but many don’t. This is where EXPLAIN ANALYZE is your friend. This outputs the query plan for the query and executes the query (careful running this in production—you can use EXPLAIN without ANALYZE to get a query plan), and then compares its estimates based on the table statistics to the actual number of rows scanned. I usually place my sql query in a file, prefix it with EXPLAIN (ANALYZE, COSTS, VERBOSE, BUFFERS, FORMAT JSON) and run:

Loading syntax highlighting...

And then use explain.dalibo.com to visualize the execution plan.

Sometimes it just makes sense to seq scan

There are cases where you think an index should be used, but the query planner is still seq scanning anyway, despite table statistics being up to date and the index being valid. In these cases, Postgres is usually estimating that the cost of the seq scan will be smaller than the cost of the index scan. Index scans do come with some overhead; indexes are stored separately from the actual data in the table (called the heap)—finding all of the rows in the heap can be expensive!

Unless you can dramatically restructure your query, you might have to accept that it’s going to seq scan, or think about something like partitioning (more on that below).

Writing lots of data

Let’s say your application is scaling and you need to write a lot of data fast. Each query has some overhead associated with it (separate from the connection overhead we talked about before): this includes the round-trip time to the database, the time it takes the internal application connection pool to acquire a connection, and the time it takes Postgres to process the query (including a set of internal Postgres locks which can be bottlenecks in high-throughput scenarios).

To reduce this overhead, we can pack a batch of rows into each query. The simplest way to do this is to send all queries to the Postgres server at once in an implicit transaction (in Go, we can use pgx to execute a SendBatch). Batching is very powerful: we found that it can ~10× your throughput. I wrote more about this plus some other tips for writing data quickly here.

Default autovacuum settings can kill your database

Autovacuum is a critical operation in Postgres databases that sometimes needs to be tuned, especially in high-write scenarios. The autovacuum daemon is responsible for a number of things, including cleaning up dead tuples and managing transaction ids.

What’s a dead tuple? A tuple is an instance of a row on the filesystem. Every time you update or delete a row, a version of that row is left in Postgres until all transactions which started before that row was updated or deleted have committed or rolled back. These rows which can no longer be read by any transactions are dead tuples.

If you’re writing data quickly enough, sometimes autovacuum can’t keep up, which will get you into a very unhealthy state, very quickly. You’ll see this when you query for active processes on the database:

Loading syntax highlighting...

If you see an autovacuum query running for more than ~1 hour, you might want to consider changing your autovacuum settings! See this article for more information.

It’s worth monitoring this: if you use up all transaction ids in the system before they can be reclaimed by autovacuum, you’ll reach a dreaded state called transaction id wraparound. This will mean a big chunk of downtime.

Other types of bloat

Besides dead tuples, there are two other kinds of bloat you’ll often encounter in a busy Postgres system:

1. Table bloat caused by partially filled pages. Postgres stores rows on pages on disk, each of which are 8kb in size. When Postgres can’t fit new rows onto an existing page, it creates a new one. But when dead tuples are reclaimed, this can lead to pages not being entirely filled, which can increase the disk usage of Postgres, sometimes significantly. The best way to avoid table bloat is by tuning autovacuum before you’re bloated. But there are some extensions to help with bloated tables, like pg_repack, because the built-in Postgres VACUUM FULL is rarely a good idea. Note that Postgres 19 is getting REPACK...CONCURRENTLY, which I haven’t tested, but seems like potentially a good solution for concurrent table repacking.
2. Index bloat is a special case of table bloat, and is similarly solved by good autovacuum settings. But Postgres has a built-in command for dealing with this, which is REINDEX INDEX CONCURRENTLY.

Some advanced stuff

I wanted to end with a set of advanced Postgres features which have been particularly useful for us at Hatchet.

FOR UPDATE SKIP LOCKED

The best way to think about this Postgres feature is that it reserves the rows that you’re selecting for use in your transaction without interfering with other queries. We use it primarily for implementing our job queue; a single-query queue in Postgres can be implemented like this:

Loading syntax highlighting...

It’s also very useful in cases where you’re doing many independent updates of rows, or you’re managing leases on objects in your system across many instances of your application (for example, we use this to distribute tenant leases across Hatchet engines).

Partitioning

Postgres has built-in partitioning which allows you to subdivide your tables based on row values, like timestamps or hashes. This can be incredibly useful for time-series data (in our case, historical task data) because:

1. Each partition can be autovacuumed independently, allowing you to scale up autovacs on your table
2. Deleting old data is near-instant—you simply drop the table partition, rather than iterating through rows

Partitioning does come with drawbacks: there can be overhead on read queries if Postgres doesn’t prune partitions during the planning phase (something that Postgres has gotten much better at in recent releases).

I’ve written more about our experience with partitioning here.

Tricks for large table migrations

(note that this isn’t referring to a database migration, but rather moving large amounts of data from one table to another, which we find ourselves doing a few times a year)

If you try to migrate really large tables, it can take many hours to copy data in a single transaction. This isn’t good—long-running transactions prevent autovacuum from doing its job properly, and therefore make the entire system bloated with dead tuples. Also if you want to keep writing to the old tables, the new table won’t see the new data.

So we need to figure out how to migrate data safely without a transaction, with new writes (after the migration has started) being copied to the new table. One of the tricks we’ve learned is to use Postgres triggers and run a large batched backfill outside of a transaction, using unique constraints on the primary key to prevent duplicate writes.

---

That’s it! If you have any other tidbits about Postgres scaling—or questions about Postgres scaling in general—feel free to reach out.

SIMD has a reputation for being complex. I've met many very good software engineers who dismiss it as something too complex to learn or a niche optimization meant for only the highest-performance software, not useful in everyday programming.

I think that's wrong. SIMD can be simple to understand[1], and common "process N values at a time" SIMD code to speed up a naive for loop almost always follows the same general shape. Once you learn the basics, writing SIMD is just about as easy as a for loop. And when it's not, it's usually a good sign to skip it for now.

Every developer should know at least that much SIMD.

This post uses Zig for examples but is a general piece that applies to any programming language. Support for SIMD instructions varies by programming language and I hope that more programming languages expose these generic concepts in the future!

I hate that I have to do this for every post now, but I also want to note this was completely hand-written with no AI assistance.

Table of Contents

• Background: What Is SIMD?
• The Common Shape
• A Real Example
• Step 1: Broadcast Constants
• Step 2: Loop One Vector at a Time
• Step 3: Perform the SIMD Operation
• Step 4: Reduce the Vector Result
• Step 5: Finish with the Scalar Tail
• Recap: The Common Shape
• Why Can't the Compiler Do This?
• Everyone Should Know SIMD

Background: What Is SIMD?

If you already know what SIMD is, skip this section.

SIMD allows a CPU to operate on multiple values in parallel. For example, instead of comparing one byte at a time, a CPU can compare 4, 8, or even more bytes with a single instruction.

If you ever see loops like this in your code:

for (byte in bytes) { /* ... */ }
for (character in string) { /* ... */ }
for (value in array) { /* ... */ }

There is an opportunity to use SIMD. SIMD turns those into this:

for (8 byte chunk in bytes) { /* ... */ }

This results in a localized speedup that directly maps to the parallelism: you process data 4x, 8x, or even faster.

The only real requirement for this to pay off is that you need to be regularly processing a large enough number of bytes. If you're doing these for loops across data that is only ever a handful or dozens of bytes, it's not worth it. But if this is iterating over hundreds, thousands, millions of bytes, the payoff will be huge.

That's the basics. Projects such as simdutf and simdjson take this to an extreme and use SIMD techniques that can be difficult to understand. But you do not need to write algorithms like those to benefit from SIMD. The common case is dramatically simpler.

The Common Shape

The common "process N values at a time" SIMD code follows the same five steps:

1. Broadcast any constants you need and initialize vector accumulators, if any.
2. Loop over input one vector-width chunk at a time.
3. Perform the comparison or arithmetic across all lanes in parallel.
4. Reduce or store the vector result as needed.
5. Handle the remaining elements with a scalar tail. A scalar tail is just your normal loop from before vectorizing, but it only processes the remainder that doesn't fit into a full vector.

As you do this more and more, you'll begin to naturally decompose every for loop into these five steps and writing SIMD becomes nearly as natural as writing a scalar loop.

A Real Example

Let's look at a real example from Ghostty. We'll look at the scalar implementation, the SIMD implementation, and then map it back to the common shape above.

I have a slice of decoded codepoints that I want to consume until I see a value at or below 0xF (a C0 control character).[2] Terminals are mostly plain characters to be printed, so we try to batch all those together. So this loop finds the end of the next printable run as quickly as possible.

The scalar loop is one line:

while (end < cps.len and cps[end] > 0xF) end += 1;

It processes one codepoint at a time. It is easy to understand.

Here is the generic vector version with no CPU-specific intrinsics and no comments. I will explain it in detail later.

if (simd.lanes(u32)) |lanes| {
    const V = @Vector(lanes, u32);
    const threshold: V = @splat(0xF);
    while (end + lanes <= cps.len) : (end += lanes) {
        const values: V = cps[end..][0..lanes].*;
        const greater_than_threshold = values > threshold;
        if (@reduce(.And, greater_than_threshold)) continue;
        const mask: std.meta.Int(.unsigned, lanes) = @bitCast(greater_than_threshold);
        end += @ctz(~mask);
        break;
    }
}

while (end < cps.len and cps[end] > 0xF) end += 1;

12 more lines of code.

This can improve the loop's throughput by up to 4x with ARM NEON (including Apple Silicon), 8x with AVX2 (most modern x86 CPUs), and 16x with AVX-512 (some Intel CPUs and AMD Zen 4 and newer).

In real-world end-to-end throughput from terminal program to finalized terminal state on an AVX2 Intel desktop, this was more like a 5x speedup. You always lose some of the ideal speedup due to the other stuff around the SIMD code, but... that's still 5x!

Okay, now I understand that those 12 lines are going to look really alien to someone not familiar with the concepts. So now let's back up and explain it step by step, mapping it directly to the shape previously mentioned.

Step 1: Broadcast Constants

Let's start with the first three lines:

if (simd.lanes(u32)) |lanes| {
    const V = @Vector(lanes, u32);
    const threshold: V = @splat(0xF);

simd.lanes(u32) is a helper in Ghostty that returns the number of u32 values the target CPU can process at once. These individual values are called lanes. On ARM this returns 4, AVX2 returns 8, and AVX-512 returns 16. If the target doesn't have a vector size we want to use, it returns null and we skip all of this code and do zero SIMD work.

@Vector(lanes, u32) creates the vector type. If lanes is 8, then V is a single value containing eight u32 values that the CPU can operate on in parallel. And so on.

Finally, we need to compare every value to 0xF. A vector comparison requires a vector on both sides, so @splat(0xF) copies, or broadcasts, 0xF into every lane. The result is a vector that looks like this:

{ 0xF, 0xF, 0xF, 0xF, 0xF, 0xF, 0xF, 0xF }

This is step 1: prepare the vector type and broadcast any constants. Some algorithms also initialize a vector accumulator here, but this algorithm doesn't need one.

Step 2: Loop One Vector at a Time

Next, we loop over one complete vector at a time:

while (end + lanes <= cps.len) : (end += lanes) {
    const values: V = cps[end..][0..lanes].*;

If lanes is 8, we only enter the loop when at least eight values remain. Inside the loop, we load those eight values into the vector values. At the end of every loop, end += lanes moves forward by eight values instead of one.

The requirement for a complete vector is important. If only five values remain, we can't load an eight-lane vector. There are various tricks to handle this, but we do the easy thing and handle them via our scalar tail, which I'll explain later in step 5.

This is step 2: load and loop over the input one vector-width chunk at a time. You can see the lane-count speedup here!

Step 3: Perform the SIMD Operation

Now we perform the comparison:

const greater_than_threshold = values > threshold;

Both values and threshold are vectors, so this maps to a vector operation (a literal vector CPU instruction). The one > compares every lane in values to every corresponding lane in threshold. If there are eight lanes, this is equivalent to performing the scalar comparison cps[end] > 0xF eight times, but it does it in one CPU instruction instead.[4]

The result is another vector with one boolean per lane. Conceptually, it looks something like this:

values:                 { 0x41, 0x42, 0x43, 0x0A, 0x44, 0x45, 0x46, 0x47 }
threshold:              {  0xF,  0xF,  0xF,  0xF,  0xF,  0xF,  0xF,  0xF }
greater_than_threshold: { true, true, true, false, true, true, true, true }

This is the actual SIMD operation. There is no explicit inner loop. The > operator applies to every lane in parallel.

Comparisons are only one example. This could be addition, multiplication, minimum, maximum, or any other operation supported by the vector type. The point is the code still has the same shape.

Step 4: Reduce the Vector Result

We now have a vector of booleans, but the original loop needs to know the location of the first value at or below 0xF.

First, let's handle the common case where every value is above 0xF:

if (@reduce(.And, greater_than_threshold)) continue;

@reduce(.And, ...) combines every boolean using and and returns a single boolean. If every lane is true, we continue and process the next vector. In our example, lane 3 is false, so @reduce returns false and we fall through to find exactly which lane failed.

If any lane is false, then we need to find exactly which lane failed:

const mask: std.meta.Int(.unsigned, lanes) = @bitCast(greater_than_threshold);
end += @ctz(~mask);
break;

@bitCast turns the vector of booleans into an integer with one bit per lane. A 1 bit means the value was greater than 0xF and a 0 means it wasn't. We invert the mask so failed comparisons are 1, and then @ctz counts the number of zero bits before the first failure. That count is the index of the first failing lane.

We add that index to end and break because we found the control character.

Using the same values from step 3, we can see this transformation per lane:

values:                 { 0x41, 0x42, 0x43, 0x0A, 0x44, 0x45, 0x46, 0x47 }
greater_than_threshold: { true, true, true, false, true, true, true, true }
mask:                   {    1,    1,    1,     0,    1,    1,    1,    1 }
~mask:                  {    0,    0,    0,     1,    0,    0,    0,    0 }

@ctz(~mask) counts three zero bits before the first 1, so it returns 3. Adding 3 to end points it at lane 3, which contains 0x0A, the first control character.

This is step 4: reduce the vector result into whatever the original algorithm needs. This is also the step that varies the most between algorithms. A sum might reduce a vector accumulator into a single number. A transform might store the entire vector to an output buffer. Our scan turns the vector into a bit mask so it can find one specific lane.

Step 5: Finish with the Scalar Tail

After the vector loop, we run the exact scalar loop we started with:

while (end < cps.len and cps[end] > 0xF) end += 1;

If the input length isn't an exact multiple of the vector width, this processes the remaining values. For example, an eight-lane vector loop leaves anywhere from zero to seven values for this loop. This is called the scalar tail.

This loop also handles CPUs where simd.lanes(u32) returns null. In that case we skip all of the SIMD code and the scalar loop processes the entire input. The original implementation remains both the fallback and the tail.

That's step 5. It's just the normal loop.

Recap: The Common Shape

Let's map the entire implementation back to the five steps:

1. @splat(0xF) broadcasts the comparison value into every lane.
2. The while loop loads lanes values at a time.
3. values > threshold compares every lane in parallel.
4. @reduce, @bitCast, and @ctz find the first failed comparison.
5. The original scalar loop handles the remainder and unsupported CPUs.

The details in step 4 initially take some time to understand, but the overall shape is straightforward. And steps 1, 2, 3, and 5 tend to look nearly identical across completely different algorithms.

Whenever you see a for (byte in bytes), this is the shape you'll map to.

Why Can't the Compiler Do This?

Sometimes it can! Compilers can auto-vectorize simple loops, particularly regular arithmetic loops without complex control flow. You should always compile the scalar version with optimizations and see what your compiler produces before manually writing SIMD.

But compilers are severely limited in what they can auto-vectorize and are in general very poor at it. Auto-vectorization has been an active area of compiler research for decades, and recent research still begins from the observation that production compilers regularly miss vectorization opportunities. This isn't a problem I expect to disappear soon.

More importantly, when this loop matters enough for me to care about a 5x speedup, I want the vectorization to be explicit and predictable. I don't want an unrelated code change or compiler update to quietly turn it back into a scalar loop.

Everyone Should Know SIMD

Every developer should be able to recognize the opportunity and, most importantly, should not be scared of SIMD. If you see a hot loop scanning, comparing, counting, or transforming a large amount of contiguous data, you should be able to imagine processing it a vector-width chunk at a time.

This post demonstrates that these common cases follow a very regular pattern that you quickly get used to. And with good language support, you don't need to know any assembly or CPU-specific quirks to get easy improvements.

Everyone should know SIMD enough to do this.[5]

Footnotes

1. Very impressive projects like simdutf and simdjson use extremely complex SIMD tricks to achieve their goals. But this isn't what I'd consider "everyday SIMD."
2. C0 controls extend beyond 0xF. This is the cutoff Ghostty uses for this specific code path; ESC and other control-sequence handling happens elsewhere.
3. Generic vectors remove the CPU-specific syntax, not CPU-specific code generation. Zig still lowers these operations to the instruction set enabled for the target. Ghostty falls back to scalar code when it can't choose a supported vector width.
4. The comparison itself is one vector operation. Loading the vector, reducing the result, and locating the failed lane require additional instructions. The important part is that we're doing multiple comparisons at once.
5. This post was based on a Lobsters comment I wrote.

I am equally unsurprised and upset

Reddit-The-Company

If you don’t know Reddit, it basically is the host of many popular forums. And like any company which encourages you to “come for the cats [and] stay for the empathy,” Reddit seems to be in the business of extracting as much value as it can from said forums without completely destroying them.

After all, simply fostering community is not a noble enough goal for the New Tech, and fortunately for Reddit, genuinely human-generated data is now gold in the LLM Age. You are welcome to read about the last time they decided to pluck the metaphorical liver from their communities.

Reddit-The-Search-Results

While I no longer wish to engage with Reddit, I still visit it occasionally, especially in the LLM Age. This is because appending site: reddit.com to a search query is basically a surefire way to find results written by genuine humans. Which, just to be extremely clear, I still find desirable.

Now behind a login… sort of

After doing such a query yesterday, to my absolute delight I was greeted with

I guess this makes me old, but I use the original frontend for Reddit, old.reddit.com. I’m going to try really hard not to preach about why it’s a better frontend, but that’s all it is! A design for Reddit.

Look, I know I’m a fringe user. I use Firefox. I noscript! I am no stranger to being forced off a product that worked just fine because someone decides to no longer support the two people who still use it.

• It happened to my phone of 8 years, which works fine by the way, but is on too outdated of an OS. May it rest in peace in its tiny glory.
• It happened to my tablet of 10 years, which works even better than my phone and holds a charge like champ, for the same reason.
• It happened to the API for Stack Exchange used by my copy of their outdated app long removed from the app store. This one hurt me the most.1

Where I feel like things get personal here is that Reddit is saying that this is all

To keep Reddit safe

Keep Reddit safe from whom exactly? Me? My desire for knowledge??

Safety is what exactly?

So let’s see what they have to say on this matter by going to the announcement, which of course I didn’t see because I don’t read Reddit anymore: https://old.reddit.com/r/modnews/comments/1ujtebf/logging_in_to_use_old_reddit/. Hope you’re logged in.

Old Reddit’s logged-out experience is a significant source of abusive scraping and automated traffic on the platform.

Hmmm, OK. But then why is New Reddit still accessible logged out? Oh, someone asked that.

[Question]: What’s so different about new reddit that people don’t try to scrape that? Seems to me like it would be better to just implement that on old reddit too. Besides, won’t this just cause people to try and scrape new reddit?

[Admin reply]: I was about to type an answer but just saw u/Nestramutat- gave a really eloquent answer in another comment!

[The comment (snipped)]: … To your first question, the shape of malicious traffic is always changing. It’s going to be a constant cat and mouse game as you ban one method, a new one gets developed. It’s easy to see abusive traffic in hindsight, but it’s harder to pre-emptively block it. Given that they’re claiming Old Reddit doesn’t have the modern security stack, this is likely proving to be an even greater challenge…

So it doesn’t have “the modern security stack.” Now I may not have really earned my Full Stack stripes, but I can right click and select Inspect Element so I’d say I’m qualified enough to see why.

Old versus New

Old Reddit

Let’s start by — begrudgingly — logging in to see what is so insecure about old.reddit.com. I’ll use their announcement thread to test. Ahhh, so much nicer.

Let’s check what Old Reddit is doing that makes it so insecure. The best I can guess is that their precious, precious user-created content is available in plain HTML, since that’s basically all Old Reddit does: you don’t even need JS unless you want to load more comments (ask me how I know).

It DLs about 1 megabyte and sends about half a megabyte. It’s not shown there, but the page’s HTML itself comprises most of the response. I’ve certainly seen worse, but what’s with the load time? GitHub loads its massive payload about 4x as fast (relative to size).

Oh… 2 whole seconds of waiting for a reply. Smells of rate-limiting. Or was Reddit always this slow?

Let’s load some more comments. (Reddit never loads all of the comments initially)

Well that was nice and lean, and pretty snappy. I don’t see my secrets being sniffed. All I got here is that Old Reddit is a pretty normal webpage, which I guess makes it insecure in comparison to…

New Reddit

Let’s see why New Reddit is so much better. In case you have unrealistic expectations, let me right them: New Reddit will not load anything more than the post itself without Javascript (JS). That’s probably what makes it more secure.

There’s a lot loading here (about 5x Old Reddit), and this is why my analysis gets rather unscientific. Rather than try to get around the “security things” (whatever that means), I instead tried to do the bare minimum necessary to fetch the content. In browser — I did not feel like writing a scraper.

This led to me basically blocking all requests to domains (including reddit.com) except for

• www.redditstatic.com/js/concat
• www.reddit.com/svc/shreddit/more-comments/
• www.reddit.com/svc/shreddit/comment/

When you do this, the page loads a lot less, but it does load. When you click to load more comments, it spins forever, but I inspected the request fired off and it did get a response with comment text. So as far as I can gather, simply running the JS on the page is sufficient to get enough information to get comments. So I guess that’s what’s stopping the scrapers? Executing Javascript?

Just for the heck of it, let’s load some more comments without the request filter.

New Reddit does 94 requests to load another 2 MB of comments in about 3s.

Well, that’s certainly less lean than Old Reddit.

In my (again, unscientific) experimenting, I reloaded the page several times and tried to load comments and replies and didn’t get any failures. One time, when I had the request filter off, I got redirected and saw a captcha field sent as a query param, but I couldn’t reproduce that. I don’t know whether you get a captcha if you just gun it directly for the comments.

But I did notice this helpful heartbeat sent back to Reddit every time I scrolled or moved my cursor.

New Reddit sends heartbeats about every 0.5s.

I guess that makes me feel safer?

So what’s safer?

Let’s dispel any notion that there are safety issues arising from a frontend, because that’s pure PR crap. Instead, if we read between the lines, Reddit doesn’t want people scraping (because it’s their gold, dammit!) and they think that shafting a few Old Reddit users will disrupt the scraping enough.

Does this actually stop scraping? I don’t know!

I won’t claim to have proven you can still scrape New Reddit: surely it must be harder, but it seems to just be that scrapers — surprise, surprise — prefer using a leaner form of Reddit. Which, by the way, loads 8x more comments by default (200 vs 25; yes, New Reddit really only loads 25 comments initially, going up to like 35 automatically if you scroll some).

Cynically, it seems like Reddit discovered they can make scraping harder by making your browser load more and work more, which they had already done by rewriting a nice piece of HTML into a 5x more bloated mess of web components or whatever. (I gave up trying not to editorialize, sorry)

The thing is that I wouldn’t have beef with Reddit if they had just quietly issued a 40X/30X error forold.reddit.com. Or if they had said “no one uses this, we’re removing it.” This is upsetting but predictable. Claiming that they need to rug-pull me because of vague assertions about security or scraping that are really not my problem? That makes me mad. I enable JS for Anubis, dammit!

Anyway

I’ll have to ponder whom this blog endangers, seeing as its core functionality — like Old Reddit’s — is serving text in plain HTML. Although maybe it’s fine for me because my blog is my content, and as such is less valuable because it was already mine to begin with. Unlike the stolen hoard of user-created content which Reddit is trying to keep secure from Big Scraping. Finders keepers!

---

But why not…

Log in?

Sure. I might do that. Just let me be angry that I have to, please?

Use “New” Reddit?

I might have to anyway, who knows if they’ll keep supporting Old Reddit. But mark my words, they will close the gates on logged-out New Reddit users if they think they can get away with it.

Use an LL…

Am I not allowed to bemoan the loss of the internet that once was and could still have been? Why must I consult the world’s smartest and most expensive computer just to read what people have to say about my random question? Is it so strange to want to read text written by humans?

---

Comments

I’ve posted this to Lobste.rs and will look at comments there.

Or reach me directly. You’re a smart cookie, I’m sure you can figure out how.

---

1. Stack Exchange’s hot new queue was the perfect replacement for Reddit on mobile when I curtailed my usage of it a long time ago. It turns out that what I liked in Reddit was reading interesting things and there was no shortage of interesting things on Stack Exchange (although they also have gone through several de-liverings which is too much of a digression even for a footnote). I now browse Wikipedia. Please don’t screw me over Wikipedia, I donated five bucks to one of your nags once. ↩︎

Nail in wood with hammer I made this!

TLDR: I gain a lot of fulfillment by making things. I don't consider things built by others at my request to be made by me, and are therefore much less fulfilling. And then I feel sad. This article starts strong and then heads off into the weeds.

There have been a lot of pieces written about what I'll call "the AI dev schism" And I think there's a lot of truth to those:

• Loss of the craft, coding things by hand
• Loss of low-level problem-solving
• Loss of fun
• Gain of high-level problem-solving
• Getting through back-burnered projects
• Gain of fun

We'll just grant those as being correct for various developers. But there's something else that troubles me.

Backstory before we get going, so you can get a better idea of my perspective:

• I'm a Gen-X hacker; I cut my teeth 80s microcomputer era.
• I hold a BS and MS in CS.
• I have 20 years industry experience, (Hewlett-Packard, startups, cofounder, Activision, etc.).
• CS instructor for the last 9 years, now at Oregon State University-Cascades.
• I'm 65% Doom on the AI-Utopia/Doom scale.
• I'm a Claude Code user sometimes.
• I code by hand sometimes.
• My father taught philosophy at a community college for 35 years. This might help explain the latter part of this blog entry.

I'm going to use "AI" to mean "Generative AI and LLMs" in this essay. Sorry, veterans of so many AI winters.

Interlude!

Before we begin, I'd like to share with you a bit of my latest sci-fi novel. Some of you might unaware that, in addition to Beej's Guides, I also write science fiction.

Kael pressed his back against the shattered bulkhead, plasma scoring the air centimeters from his face. The Vorrkai assault drones had anticipated their route through the lower decks and now Rin was bleeding through her jacket sleeve and old Maret couldn't stop coughing from the vented coolant still hazing the corridor. Kael counted the pulse-intervals between shots. Three seconds. Maybe four. That was all the universe was offering him.

Then he saw it: the maintenance shaft behind the collapsed generator housing, its grate blown half-open by the same explosion that had caved in their original exit. It was tight. It was ugly. It ran directly over the Vorrkai's forward position, which was either the most dangerous path imaginable or the last one they'd ever think to watch. Kael grabbed Maret's collar and pointed without a word. The old man's eyes went wide, then hard. He nodded. Rin was already moving.

Kael came last, returning fire blind around the bulkhead corner, not to hit anything, just to make noise, and to give the drones something thermal to track while his people scrambled into the dark. A bolt caught the generator housing and the whole structure groaned, raining sparks down into the shaft on top of them. He hauled himself in, knees burning on the torn metal, and pulled the grate closed behind him with a sound he was certain every Vorrkai unit on the deck had heard. In the black ahead, Rin's hand found his wrist. Move, her grip said. Now. And so they did.

—Excerpt from The Vorrkai Interval, by Brian "Beej Jorgensen" Hall

And, in my now-copious spare time I make art! This is a woodcut, painted in pastels, showing some of my favorite subjects.

Dueling Wizard Hackers Mirrors of the Machine by Brian "Beej Jorgensen" Hall, $1300.

Carpentry? You bet I dabble! I rebuilt my front deck recently. The old one was rotting out, so I grabbed a bunch of cedar and put it together. I'd been meaning to do it for a while, but couldn't find the time.

And, finally, here's some of the code I wrote for a TUI adventure roguelike:

    fn try_move(&mut self, dx: i32, dy: i32) {
        let nx = self.player.x + dx;
        let ny = self.player.y + dy;

        // Check for monster combat
        if let Some(idx) = self.world.monster_at(nx, ny) {
            let result = {
                let monster = &mut self.world.monsters[idx];
                resolve_combat(&mut self.player, monster, &mut self.rng)
            };

            self.messages.push_many(result.messages);

            if result.monster_defeated {
                let monster = &self.world.monsters[idx];
                let xp = monster.xp_reward;
                let gold = monster.gold_reward;
                self.player.xp += xp;
                self.player.gold += gold;
                if gold > 0 {
                    self.messages.push(format!("You find {} gold!", gold));
                }
                if self.player.try_level_up() {
                    self.messages.push(format!(
                        "Level up! You are now level {}!",
                        self.player.level
                    ));
                }
            }

            if !self.player.is_alive() {
                self.messages.push("You have been slain! Rest in peace...");
            }

            self.advance_turn();
            return;
        }

        // Check terrain passability
        if self.world.is_passable(nx, ny) {
            self.player.x = nx;
            self.player.y = ny;
            self.advance_turn();
        } else {
            let terrain = self.world.terrain_at(nx, ny);
            self.messages.push(format!("The {} blocks your path.", terrain.name()));
        }
    }

I'm an extremely prolific polymath, I'm sure you'd agree!

I Am Uncomfortable

I don't like lying. And yet I feel, dear reader, I have misled you. Yes, all that has been created (including my deck) and I was the initiator of all that creation. But I don't really feel like I made any of it. I'm uncomfortable claiming that I did so.

Since you are certainly aware by now that all of the above is AI-generated (except my deck, which was created by skilled, paid craftsmen), perhaps you feel a little bit of discomfort with me claiming credit for doing those things, too.

However, I don't think everyone feels this way. I know many people who ask contractors to build things and they phrase it like they built it.

"I put in a new front deck," they'd say, even though other people did all the work. Personally, I feel that's misleading. I'm more of a "I had a new front deck put in" kind of person.

And when I do have Claude create something for me, I just can't say that I made it. Other people can, but I just can't. Again, I'm more prone to say, "I had this code built for me." I don't even feel comfortable MIT-licensing that (not-for-hire) work, if that's even legally possible. I just Unlicense it all.

As a manager, I'd never say that I built a product. "My team built this," I'd say. And as a manager of LLMs: "My Agents built this."

And that, for me, has very little weight in terms of making.

I don't feel like I did anything. And I like doing things. I find pride in doing things.

Completing projects is great. I love completing projects. Capitalists love completing projects. Real artists ship.

But having others complete projects I initiated is entirely less fulfilling to me.

It's not just the loss of the craft and the problem-solving challenge and whatever else. It's the loss of making.

What Did I Make Recently?

My wife wanted a no-frills flash card system for learning Spanish. "I just want a thing where I can put the words I want in a spreadsheet and then see it on flash cards." A prompt!

So I wrote it. By hand. I did use Claude to learn some basics, like the easiest way to get the data out of a Google Sheet (spoiler: it's the CSV endpoint), but I told it to generate no code.

--------------------------------------------
Language                     files      code
--------------------------------------------
JavaScript                       2       112
CSS                              1        33
HTML                             1        32
--------------------------------------------
SUM:                             4       177
--------------------------------------------

Didn't take long. Only about 50x longer than it would have taken Claude to do it.

But I can put my name on that code and say that I made it. Was it a lot of code? No. Was it groundbreaking and amazing? Certainly not. But I'm infinitely more proud of that code than anything I've had Claude write, because I'm not capable of being proud of the latter.

And my wife wouldn't go to her book club and say, "I wrote a flash card system to study Spanish." Admittedly, part of this would be because she didn't want to appear a geek, but mostly it's because it's untrue, even though she initiated the process.

What About The Art and Craft of Prompting?

After all, you create the prompts, don't you? You said you were proud of doing things. Isn't that doing a thing? And since so much got done, isn't it even more of doing a thing?

I don't disagree. And I do agree that there is skill here in some important ways.

• You have to apply vision.
• You have to apply judgment.
• You have to apply communication skill.
• You have to apply prompting skill.

Not all prompts are equally effective. Not all users of AI are as effective as one another. There's a very human contribution to be made here.

But the skill is in effectively asking someone to make something for you.

"Leadership is the art of getting someone else to do something you want done because he wants to do it."

          —Dwight D. Eisenhower

And I'm the kind of person who really misses the making of software. And prompting for software, to me, isn't the same as making the software. It's the same as asking someone else to make it.

What About Compilers, Smartypants?

Isn't it just turtles all the way down?

For any given C or Rust program, did I write that program? I wrote it in C. I didn't write any machine code. The computer wrote that.

There's something different in my mind about that. I'm not sure if it's the mathematically-precise relationship between the C and the machine code...? And yet that's not spelled out. I'm relying on some hacker's deep knowledge of the CPU to generate that magnificent set of optimized instructions when I stupidly divide something by seven. And it's not deterministic, strictly, in that Clang and GCC emit different instructions. What about different platforms? The C is portable and the machine code is not. Did I write that program that's running on Windows even though I personally only ever built it on Linux?

Going further in the murky gray, what if I instruct an LLM to be a C compiler and emit assembly code for some platform?

Let's try:

Convert this C program into x86_64 assembly for Linux.

#include <stdio.h>

int fib(int n) {
    if (n <= 1) return n;
    return fib(n-1) + fib(n-2);
}

int main(void)
{
    for (int i = 0; i < 10; i++)
        printf("%d: %d\n", i, fib(i));
}

Claude says:

# fib.s - Recursive Fibonacci, x86_64 Linux, AT&T syntax
# Build: gcc fib.s -o fib -no-pie

    .section .rodata
fmt:
    .string "%d: %d\n"

    .section .text
    .globl main

# ┌─────────────────────────────────────────────────────┐
# │  int fib(int n)          n in: %edi / return: %eax  │
# │  Stack frame:                                        │
# │    %rbp-8  = saved %rbx  (via push)                 │
# │    %rbp-12 = n  (spilled so it survives both calls)  │
# └─────────────────────────────────────────────────────┘
fib:
    pushq   %rbp
    movq    %rsp, %rbp
    pushq   %rbx                # callee-saved scratch
    subq    $8, %rsp            # 1 local (int n); keeps stack 16-byte aligned

    movl    %edi, -12(%rbp)     # spill n to stack

    cmpl    $1, %edi
    jle     .Lbase              # n <= 1 → return n

    # --- fib(n-1) ---
    movl    -12(%rbp), %edi
    subl    $1, %edi
    call    fib
    movl    %eax, %ebx          # %ebx = fib(n-1)

    # --- fib(n-2) ---
    movl    -12(%rbp), %edi     # reload n (call clobbered %edi)
    subl    $2, %edi
    call    fib                 # %eax = fib(n-2)

    addl    %ebx, %eax          # return fib(n-1) + fib(n-2)
    jmp     .Lret

.Lbase:
    movl    %edi, %eax          # return n

.Lret:
    addq    $8, %rsp
    popq    %rbx
    popq    %rbp
    ret

# ┌─────────────────────────────────────────────────────┐
# │  int main(void)                                      │
# │  Stack frame:                                        │
# │    %rbp-4  = int i  (loop counter)                  │
# └─────────────────────────────────────────────────────┘
main:
    pushq   %rbp
    movq    %rsp, %rbp
    subq    $16, %rsp           # locals + alignment

    movl    $0, -4(%rbp)        # i = 0

.Lloop:
    cmpl    $10, -4(%rbp)
    jge     .Ldone              # i >= 10 → exit

    # fib(i)
    movl    -4(%rbp), %edi
    call    fib

    # printf("%d: %d\n", i, fib(i))
    movl    -4(%rbp), %esi      # arg2 = i
    movl    %eax, %edx          # arg3 = fib(i)
    leaq    fmt(%rip), %rdi     # arg1 = fmt  (RIP-relative)
    xorl    %eax, %eax          # AL=0: no XMM args to printf
    call    printf

    addl    $1, -4(%rbp)        # i++
    jmp     .Lloop

.Ldone:
    xorl    %eax, %eax          # return 0
    leave
    ret

OK, clanker, but does it work?

% gcc fib.s -o fib -no-pie
% ./fib
0: 0
1: 1
2: 1
3: 2
4: 3
5: 5
6: 8
7: 13
8: 21
9: 34

Well, holy golden ratios! It even used XOR to set exit status 0, the cheeky devil!

So did I write that program? Yes. I feel I did. (I actually did hand-code the Fibonacci thing just now—I've done it a million times for classes.) I feel I wrote the program even though I didn't write the assembly, which is the program that actually runs! The LLM wrote that!

Technically, it's the machine code that runs, but going from assembly to machine code now we're really getting into the land of strict, brainless, mathematically-precise translation, which I feel doesn't take away from the act of making any more than using a hammer to drive nails does.

What's the real difference in that gray area? I'm not sure. I think it hinges on something that has to do with making versus asking to be made, even if that line isn't particularly clear.

"I know it when I see it."

          —US Supreme Court Justice Potter Stewart

Things that I do:

• Write C code and run it.
• Use a hammer to drive nails.
• Use an assembler to convert assembly to machine code.

Things that I ask others to do at my command (inspired by a nice capitalistic investment):

• Write a piece of software.
• Build me a deck.
• Paint me a picture.

When I have ChatGPT create a painting, I don't feel like I painted it.

When I use the hammer, I don't feel like I'm asking the hammer to do something for me. Same with a C compiler or assembler.

And yet I just had Claude compile some C code right up there. Was that me writing C code and running it? Or was it asking something to write software for me? I have no doubt I could have prompted Claude with an English description of what I wanted and gotten the same output.

Is it because AIs seem human? You are asking them to do things in imprecise, human language. They respond in human ways and do the things you ask. And the relationship we are used to in those circumstances is more one of manager-report than one of colleague and peer. Whereas our relationship with a compiler is more similar to that of one with a hammer.

And yet... looking into that gray area makes me feel more confident that telling the AI to make something for me actually is an act of creation. What if I had an AI-powered hammer and asked it to hit a nail for me? What's the difference? But at that point, I can't say that I hammered it. It is important to me, this difference. Why?

By now, dear reader, I suspect we're both running out of steam and maybe we should leave it here to ponder.

I feel it's appropriate to conclude with some classic DNA:

"All I wanted to say," bellowed the computer, "is that my circuits are now irrevocably committed to calculating the answer to the Ultimate Question of Life, the Universe, and Everything." He paused and satisfied himself that he now had everyone's attention, before continuing more quietly. "But the program will take me a little while to run."

Fook glanced impatiently at his watch.

"How long?" he said.

"Seven and a half million years," said Deep Thought.

Lunkwill and Fook blinked at each other.

"Seven and a half million years!" they cried in chorus.

"Yes," declaimed Deep Thought, "I said I'd have to think about it, didn't I? And it occurs to me that running a program like this is bound to create an enormous amount of popular publicity for the whole area of philosophy in general. Everyone's going to have their own theories about what answer I'm eventually going to come up with, and who better to capitalize on that media market than you yourselves? So long as you can keep disagreeing with each other violently enough and maligning each other in the popular press, and so long as you have clever agents, you can keep yourselves on the gravy train for life. How does that sound?"

The two philosophers gaped at him.

"Bloody hell," said Majikthise, "now that is what I call thinking. Here, Vroomfondel, why do we never think of things like that?"

"Dunno," said Vroomfondel in an awed whisper; "think our brains must be too highly trained, Majikthise."

—The Hitchhiker's Guide to the Galaxy, Douglas Adams

My Actual Contributions

As a post-script, here are my applied skills in the first part of this article.

Science fiction (honestly, the result was so vomitous I feel genuinely embarrassed having signed my name to it):

Write three paragraphs of sci-fi. Setting: We're in the middle of the novel at one of the pivotal fight sequences. The hero (choose a heroic sci-fi name) is leading two other refugees out under heavy laser fire from the baddies (choose an evil-sounding sci-fi name for this group). Our heroes have just been pinned down, but now see a way out.

What's the name of this novel?

Paintings and artwork:

Generate an image that shows the following: a woodcut, but with pastel coloring. Two wizardly computers hackers, one on the left of the frame and one on the right, facing each other, computers between them, both head down and focused. The left and right sides should be complementary and opposite colors to show that both sides are the same, yet different.

What's the name of this painting?

Ultima-likes:

I'm thinking of a game like Ultima I, except with a TUI. Visually, I was thinking of a 3x3 grid of characters per map cell. Some of them would be animated, e.g. the water on the ocean cells. And the map on the screen would be 7x7 map cells (or whatever Ultima I had)

It would be also like Ultima I in that it was fantasy.

The world should be procedurally-generated maybe with perlin noise. And the monsters should roam randomly, though they will be of prebuilt classes (no need to procedurally-generate monsters on the fly).

There should be some kind of DnD-ish stats, as well. No classes--just a general player is fine.

I'd like it in Rust using Ratatui. Color.

And I'd like the code well structured, with good separation between the UI and the game logic, as well as good code directory layout.

Carpentry:

Please replace my front deck.

Though I really did drive that nail into that piece of wood. I needed a picture to open with.

Uses of em dash within this piece are deliberate. Vim digraph: ^K-M.

Nobody knows what a used GPU cluster is worth

If xAI defaults on its debt, Apollo Global Management ends up in the GPU rental business. That is in the contract, signed in June 2025, on a five billion dollar debt facility arranged by Morgan Stanley. The lenders have the right to take over Colossus, the company’s 200,000 GPU cluster outside Memphis, and rent it to other AI companies until the loan is repaid.

It’s interesting whether Apollo, or Diameter Capital Partners, or any of the other lenders now financing the AI buildout this way, would want to exercise that right.

The harder question is what they would actually be holding if they did.

A GPU cluster bears little resemblance to a building. Its value at any given moment depends on how it has been provisioned, how it is currently performing, and whether the team that knows its quirks is still there. All of that sits off the lender's balance sheet, beyond the reach of anyone they can call.

This is one of the center problems of the AI infrastructure boom. I cannot determine why no one is talking about it. Tens of billions of dollars in debt is now collateralized by chips whose value depends on operational state, and the operational state is invisible to the people pricing the debt.

This week’s CipherTalk is about what happens to a specific kind of debt when the collateral itself can walk out the door with the operations team.

The cluster does not run itself

At the scale these GPU clusters operate, hardware and systems break constantly. Keeping them productive is a craft.

Modern data center GPUs fail at roughly 9% annually. The number traces to Meta’s Llama 3 technical report, which documented 419 unforeseen disruptions across 16,384 H100s over 54 days of training, of which 148 were GPU failures and 72 were HBM3 memory failures. At 200,000 GPUs, that annualized rate works out to approximately 50 GPU failures every day. At xAI’s stated million-GPU target, Epoch AI projects a failure roughly every three minutes. These are not catastrophic events. They are the steady state.

The failure modes that matter for a credit person are the ones that do not look like failures. Silent data corruption (SDC) is the most expensive, where a faulty GPU produces wrong answers without crashing anything, which means a multi-day training run can complete normally and the resulting model weights are quietly poisoned. Cascading failures are the second category, where one bad GPU crashes a training job spread across thousands of others, costing days of compute. Then there are the routine ones: thermal throttle, ECC memory errors, NVLink flap, GPUs falling off the bus.

NVIDIA built NVSentinel because traditional monitoring detects these problems but rarely fixes them. Crusoe built AutoClusters because queue wait time is the largest controllable variable in cluster goodput. Without these tools, remediation timelines run hours to days.

The job of an operations team is to keep all of this in steady state. They know which racks run hot in summer, which cooling loops have been flaky since the last firmware update, which jobs to re-route when a node degrades but has not failed yet. None of that knowledge is written down. It lives in the team.

This is the asset that serves as collateral for tens of billions of dollars in debt and counting.

How the chips became the collateral

In the last eighteen months, AI infrastructure went from being financed by corporate debt, to being financed by the chips themselves.

The xAI Colossus 2 SPV is the cleanest example. The structure is roughly $7.5 billion in equity, with up to $2 billion of that contributed by NVIDIA itself, and $12.5 billion in debt. The special purpose vehicle (SPV) purchases NVIDIA GPUs and leases them to xAI on a five-year term. Apollo and Diameter sit on the debt tranche. Valor Equity Partners leads the equity. The debt is collateralized by the chips, not by xAI’s broader balance sheet.

Look at the pricing: xAI’s $5B round was priced at up to 12.5%. CoreWeave's GPU-backed deals priced at roughly 8.5% above the benchmark rate, before terms tightened as lenders got more comfortable with the structure.

If we assume here these are not unsophisticated lenders, then we have to assume they are charging what they think the risk costs. The premium is then, the price of guessing.

The scope is wider than one company. CoreWeave alone holds $18.8 billion in GPU-collateralized debt across multiple SPVs. FluidStack’s $50 billion deal with Anthropic uses a different wrapper, with Google providing a backstop on the lease payments, but the underlying logic is the same.

Every neocloud and most major AI labs are now financed this way.

What real collateral looks like

Every other major asset class that gets used as collateral at this scale has decades of price discovery infrastructure behind it. GPUs have almost none of it.

Aircraft have ISTAT-certified appraisers, a global registry, standardized maintenance logs, ferry pilots, and an active secondary market dating back to the 1970s. Ships have BICA. Cars have NADA. Class A office space has standardized cap rates and vacancy comps. Oil has had a forward curve since the early 1980s.

GPUs have Silicon Data’s H100 Rental Index on Bloomberg terminals, which launched in 2024, and Ornn AI, which raised $5.7 million in October 2025 to build the first regulated exchange for GPU compute derivatives. That is the entire price discovery infrastructure for an asset class now backing tens of billions of dollars in debt.

The price moves underneath all of this are wild. H100 hourly rental rates went from roughly $8 per hour in early 2024 to $1.70 by October 2025, then surged 40% back up to $2.35 by March 2026 on a wave of inference demand nobody had priced in. SemiAnalysis put it bluntly: lenders who used six-year depreciation schedules now look smarter than the analysts who chastised them for being too generous. They were guessing, and they happened to land closer to the right answer than the people calling them reckless. No aircraft lender or shipping lender would underwrite five-year debt against an asset whose price swings like that without a way to hedge it. They would not be allowed to.

CoreWeave’s GPU-backed loans price at roughly 8.5 percentage points above the benchmark rate. For comparison, a typical aircraft loan prices at 1 to 2 points above benchmark, and a commercial mortgage usually sits below that. The extra 6 to 7 points is what lenders charge to bear a risk they cannot measure. There is no GPU futures market, no standardized residual value curve, and no way to lock in a forward rental rate. The premium is is the price of underwriting in the dark.

That spread should compress as the market matures. Hedging instruments will appear. Residual value curves will get more standardized. Secondary markets for used GPUs will deepen. When that happens, the cost of capital for AI infrastructure drops meaningfully, which changes who can build at scale. The companies that benefit are not the ones with the cheapest GPUs today. They are the ones positioned to access cheap debt once the financing infrastructure catches up to the asset class.

Six years, four years, or somewhere worse

The public fight over how fast GPUs depreciate is a tell about how confident the people writing the books actually are.

CoreWeave depreciates GPUs over six years. Nebius, with the same business model and the same hardware, depreciates the same chips over four. AWS, Microsoft, and Google all moved their server useful-life assumptions from three to four years up to six years in 2023, a change that reduced reported depreciation expense by roughly $18 billion annually across $300 billion of combined capex. CoreWeave made the same accounting change in January 2023, before going public, lowering reported expense by hundreds of millions of dollars per year.

NVIDIA announced in 2025 that it is moving from a two-year product cycle to a one-year cycle. The chips backing all of this debt are about to become previous-generation twice as fast.

Michael Burry’s claim is that hyperscalers will cumulatively understate depreciation by approximately $176 billion between 2026 and 2028. He projects Oracle will overstate earnings by roughly 27% and Meta by roughly 21% by 2028. Burry’s motives aside, the math is independently checkable. If the true useful life of frontier-training GPUs is closer to two to four years and the books say six, the gap between paper value and recovery value is real and it is enormous. The recent inference demand surge complicates this. If H100s genuinely have productive life past frontier training, six years may not be wrong. If demand softens again in 2026 or 2027, the writedowns hit at exactly the moment lenders need their collateral to be worth something.

Three values for the same chip

GPU collateral has three different values, and the market is currently pricing only one of them.

Face value is what the SPV says, the purchase price minus straight-line depreciation on whatever schedule the borrower picked. This is the number that determines loan-to-value covenants and the amount of debt the deal can support.

Liquidation value is what a buyer pays in distress. Secondary market data shows moderately-used 2 to 3 year old GPUs trading at 50% to 70% of new pricing under normal conditions. In a default scenario where multiple neoclouds are stressed simultaneously, the buyer pool collapses at the same moment supply spikes, plausibly putting recovery at 30% to 50% of face value in a fire sale.

Going-concern value is what the cluster is worth as a working asset to the next tenant, which depends entirely on whether operational handoff works.

This is where the operational reality from the first section returns. The lender exercising step-in rights inherits a colocation facility owned by someone else, with that facility’s own contracts and constraints. They inherit credentials and topology knowledge that historically lived with the borrower’s operations team, which walked out the door at default. They inherit a market where rental rates already moved 60% in one direction and 40% back the other in eighteen months, with no hedging instrument available. They inherit an asset class where 50 chips a day fail and somebody has to know which racks have been flaky for the last quarter.

The spread between face value and going-concern value is the entire risk that nobody has hedged.

The signal in who is not in the room

The most telling positions in this market are the ones not being taken.

KKR has been the most aggressive private equity firm in data centers, with the CyrusOne acquisition alongside Global Infrastructure Partners in 2022 for $15 billion, the Global Technical Realty commitment in 2026 for $1.5 billion, and the STT GDC deal in February 2026 for $5.1 billion at a 75% stake. KKR’s digital infrastructure book is a central pillar of $186 billion in real assets. The firm is not in the AIP consortium that bought Aligned Data Centers, not in any xAI SPV, and not in CoreWeave’s debt facilities. KKR owns the buildings, the power, the cooling, and the land, the infrastructure layer that holds value regardless of which AI lab wins or which chip generation dominates.

Peter Thiel sold his entire NVIDIA stake in Q3 2025 and rotated into Apple and Microsoft. The chips are not the durable asset, and the financing structure pricing them as durable will eventually have to reckon with what the chips actually are.

Aircraft became financeable because someone built the registry, the appraisers, and the maintenance logs. Ships became financeable because someone built BICA. The interest premium on these deals exists because no one can answer two basic questions: Is the cluster still working? And: Will it still be working in three years?

The Wizard's Castle Let's dive into The Wizard's Castle (source code)!

10 REM"_(C2SLFF4

That is the first line of the 80s microcomputer BASIC game The Wizard's Castle initially written for the Exidy Sorcerer platform.

It's a REMark statement, a comment in that particular language. 10 is the line number, if you're unfamiliar with languages that had such things.

But the interesting part was this "_(C2SLFF4 mess. A typo or garbage? No. It appears verbatim in the source code as originally published in the July, 1980 issue of Recreational Computing.

What the heck is it?

Note: I'm going to jump back and forth between decimal (which is BASIC-friendly) and hex (which is programmer-friendly). Hex will be prefixed with 0x, suffixed with h, or will obviously have digits A-F in it.

Note: Thanks to Josh and Chris, my hacking buddies, for doing tons of the emulator and research work on this. It took hours, but they were good hours. It would have taken me 10x as long without their help.

Suspicions

Here's more of the source in context, but with irrelevant pieces removed and some spaces added:

10 REM"_(C2SLFF4
40 POKE 260,218: POKE 261,1: T = USR(0): T = PEEK(-2049)
80 Q = RND(-(2*T+1))

In BASIC, a : is a command separator. POKE writes a byte value to the specified memory location. PEEK reads from one. Sorcerer BASIC used signed 16-bit numbers, apparently, so address -2049 is 65536-2049 or 0xF7FF in hex. We'll revisit that address later.

Calling the pseudo-random number generator (PRNG) RND() with a negative number seeds it. Old PRNGs liked to be seeded with odd numbers, thus the 2*T+1 expression forcing it odd.

And the USR() function calls a machine code routine.

And line 80 is the first use of T after it is initialized from the PEEK().

The proximity of these things makes it seem like they're all related to seeding the PRNG. Sorcerer BASIC didn't have a RANDOMIZE command, so the only options of the day were:

• Ask the user to enter a random seed.
• Run an increment loop until the user hits a key (or some user-driven thing like that) and use that for the seed.
• Get the seed for some kind of randomish-thing from the existing software and/or hardware.

Wizard's Castle didn't do the first two. So it must be the last one.

Could it be that the REM statement has machine code that happens to encode to ASCII text? The Sorcerer did use ASCII encoding.

But it seems nuts that you could get useful Z80 machine code just from ASCII characters. Right?

Nevertheless, this seemed like a reasonable-enough crazy assumption to run with and see what we could find.

The USR() Function

The USR() call is interesting—this calls a machine code function, but the details are system-dependent. But luckily the Internet has tons of technical manuals in various places and we were able to figure it out.

Address 259 has a JP instruction (unconditional jump) to a 16-bit little-endian absolute address.

And the POKEs were for addresses 260 and 261. This was the trampoline for the USR() function. You POKE the address of the start of your machine code into addresses 260 and 261, and then run USR() to call it.

40 POKE 260,218: POKE 261,1: T = USR(0): T = PEEK(-2049)

Reversing them for little-endian, this means we're looking at address (1 << 8) | 218 which is 474. When we run USR(0), it jumps to the machine code at address 474 that should end in a RET instruction.

The argument for USR() (the 0) is stored as a 4-byte float elsewhere in RAM so the machine code can look at it if it needs to. It turns out not to be relevant for this use case. Also we're assigning the return value from USR() into T, but it's unclear to me what this return value is. However, it is irrelevant as we immediately overwrite T with the next assignment.

So... What's at address 474?

BASIC RAM

The Exidy Sorcerer The almighty Exidy Sorcerer

By Marcin Wichary, CC BY 2.0

When you enter a line of Sorcerer BASIC, the interpreter tokenizes it and replaces commands like PRINT with single byte values:

• PRINT is tokenized to 0x97
• REM is tokenized to 0xC3
• etc.

And then it is stored like a linked list in memory where each row is a "node".

The node format is:

• 2 bytes of "next" pointer
• 2 bytes for the line number
• Bytes representing the tokenized line of code
• 1 byte of null terminator (a 0x00 byte)

And the start of this linked list in RAM is at address 469 on the Sorcerer.

Let's see the mystery line again:

10 REM"_(C2SLFF4

That means the node for that line consists of the following addresses and bytes:

469  Low byte of the next pointer
470  High byte of the next pointer
471  Low byte of the line number
472  High byte of the line number
473  `REM` token (0xc3)
474  First byte of REM text, the `"` character!!

and 474 is where the USR() function gets us! It is literally calling the text of the REM statement as Z80 machine code!

Disassembly, Attempt One

Could it be true? Let's try it out!

Here are the hex values for the ASCII characters:

"  22
_  5F
(  28
C  43
2  32
S  53
L  4C
F  46
F  46
4  34

There's the 0x00 terminator at the end, but that's just a NOP in Z80, so we'll ignore it.

Disassembling, we get:

22 5F 28   LD   (285Fh),HL  ; " _ (
43         LD   B,E         ; C  
32 53 4C   LD   (4C53h),A   ; 2 S L
46         LD   B,(HL)      ; F
46         LD   B,(HL)      ; F
34         INC  (HL)        ; 4

Now, I don't want to get into the details of Z80 assembly here, but take my word for it that this code makes no sense. The addresses don't seem to point to anything in particular, who knows what HL has in it at the beginning, B is never read, the duplicated LD B is pointless, and there's no RET to get us back to BASIC.

It's garbage. And running it in an emulator results in Weird Things happening (like soft resets), as expected.

This was a dead-end for the time being. We speculated that maybe it was a printing error, that the printer didn't have the glyphs for the actual bytes, so it substituted other ones. But that wasn't super-satisfying.

The PEEK(-2049)

Let's switch gears, then, and continue on to the PEEK:

40 POKE 260,218: POKE 261,1: T = USR(0): T = PEEK(-2049)

What's at address -2049? Unsigning this makes it equivalent to 0xF7FF. According to the docs, this is the last byte of memory-mapped screen text, representing the current character in the lower right of the screen.

We then immediately use this value to seed the PRNG:

40 [ ... ] T = PEEK(-2049)
80 Q = RND(-(2*T+1))

I'm going to use my magical powers now to look at one of your open terminal windows and read the character in the lower right of the screen. It's fuzzy, but if I concentrate... it's... yes, it's a space character, isn't it!

Am I right? That'll be $200.

Now, the space is 32, so seeding the PRNG with -(2*32+1) every game would make for bad replayability in this randomly-generated dungeon, so the USR() function must have something to do with it. Like it puts something on the screen at address 0xF7FF and then the PEEK() reads it. The screen is cleared not-long thereafter, so the player might not notice the blip of a character temporarily placed in the lower right.

RTFM

Wizard's Castle source code The source code in its original dot-matrix glory

So we had strong suspicions that all this was part of the PRNG seed process. If only there were some way to prove it!

It was at this point that Josh noticed this line in the magazine where the program was published: "The first remark is a machine language routine to simulate the RANDOM function."

Sheesh. That'll teach me not to read things.

In BASIC, RND(), not RANDOM, is the random function, used with a negative argument to seed the PRNG, and a positive argument to get the next random number, which was historically a random floating point result between 0 and 1.

So what the author probably meant was RANDOMIZE which was popular in Microsoft BASIC for setting the random number seed to a specific value or prompting the user to enter one.

In short, it was supposed to be doing what we thought, but the machine code didn't seem to be doing that at all.

A Breakthrough

Josh and I had been running Sorcerer emulators in MAME (thanks to Josh for sharing the ROMs with me to get going) and Chris had another Java-based one that he'd gotten working.

In MAME, we'd entered the REM statement and looked in memory, and it had the nonsensical bytes and machine code as disassembled, above. And it didn't run properly.

But Chris found a tape image of the game and loaded that up in his emulator. And the first two lines of source looked like this:

10 REM"_(C2SLFF4F4F4 
15 REM ED 5F 28 FC 32 FF F7 C9 (in O1DA)

WOW! Another(?) author has annotated the source with the hex of the machine code! Not only does it end in 0xC9 which is Z80 RET, but it has that lower-right screen corner address 0xF7FF right there!

How on Earth does the REM text map to that? Things like 0xED are clearly non-ASCII. But wait, some of the characters are, and their positions match what we see in the REM!

   "
5F _
28 (
   C
32 2
   S
   L
   F

What actual values are there? Chris loaded the program and looked:

FOR I=474 TO 489: PRINT PEEK(I): NEXT I
 237
 95
 40
 252
 50
 255
 247
 201
 70
 52
 70
 52
 32
 0
 18
 2
READY

You can see the 0x00 null terminator down there. And a hidden space at the end of the line. And 237 is 0xED, and 95 is 0x5F... it matches the line 15 comment!

Let's get ready to disassembbbllllllllllleeee! I'll convert the numbers to hex and go for it.

ed 5f     LD A,R         " _
28 fc     JR Z,-4        ( C
32 ff f7  LD (F7FF),A    2 S L
c9        RET            F
46        LD B,(HL)      F
34        INC (HL)       4
46        LD B,(HL)      F
34        INC (HL)       4
20 00     JR NZ,0        space null

Later, I'll find out that the part after the RET is disassembled incorrectly and that my annotation of which REM characters mapped to which hex values was wrong, but who cares for now! Everything is over at the RET anyway.

And the code is exactly what we were looking for! Let's look at the first assembly instructions:

LD A,R       ; copy R register to accumulator
JR Z,-4      ; if result is 0, jump back to previous instruction
LD (F7FF),A  ; copy accumulator into address F7FFh
RET          ; return

What's it do? The R register is interesting on the Z80 in that it is incremented every instruction fetch. Maybe? Depends on which resource you ask. In any case, it is incremented very, very frequently relative to human timescales. And the Sorcerer busy-waits for input, so when you finally run the game, it effectively has a random-ish value in it.

But seeding many lesser PRNGs with 0 is a bad idea (they tend to just produce 0 after that), so the code retries if it happens to get a 0 in R. But if it is non-zero, it stores that value in address 0xF7FF, the lower right corner of the screen! And then the PEEK() picks it up and uses it to seed the PRNG!

The R register only increments the low 7 bits, though, meaning it can only be 128 different values. And we discard 0, so really this means on the Sorcerer, there were only 127 different randomly-generated dungeons that you could play. Bummer!

So the glyphs we see simply aren't ASCII. Our first disassembly was doomed to fail because the REM violates the assumption that all listing text was ASCII.

For verification, I made a new program that was just a REM followed by a bunch of spaces (to give me room to work before the null terminator). Then I POKEd in the right values and got a listing.

10 REM               
POKE 474,237
READY
POKE 475,95
READY
POKE 476,40
READY
LIST

10 REM"_(
READY

It worked! Those values gave me the "_( glyphs that were in the original source listing!

Could You Type It In?

The whole idea in the 80s with magazines like this was that you'd get it in the mail or from the newsstand and you'd painstakingly spend hours typing them in and getting them right.

It wasn't easy. Here's another excerpt from The Wizard's Castle:

1070 IFFL=0THENPRINT:PRINT"** HEY BRIGHT ONE, YOU'RE OUT OF FLARES":GOTO620
1080 PRINT:PRINT:FL=FL-1:A=X:B=Y:FORQ1=A-1TOA+1:X=FNB(Q1):FORQ2=B-1TOB+1:Y=FNB(Q2)
1090 Q=FNE(PEEK(FND(Z))):POKEFND(Z),Q:PRINTI$(Q);"  ";:NEXTQ2:PRINT:PRINT:NEXTQ1:X=A:Y=B
1100 GOSUB 3400:GOTO620
1110 IFLF=0THENPRINT:PRINT"** YOU DON'T HAVE A LAMP, ";R$(RC):GOTO620
1120 PRINT:PRINT"WHERE DO YOU SHINE THE LAMP (N,S,E, OR W) ";:GOSUB3290
1130 A=X:B=Y:X=FNB(X+(O$="N")-(O$="S")):Y=FNB(Y+(O$="W")-(O$="E"))
1140 IFA-X+B-Y=0THENPRINT:PRINT"** TURKEY! THAT'S NOT A DIRECTION":GOTO620

Just pure codevomit. But those of us who typed this stuff in were pretty practiced at getting it right. So when we saw something like this:

10 REM"_(C2SLFF4

you can bet we typed it in exactly.

But now we know this wouldn't have worked if you assumed those were ASCII-encoded glyphs.

Perhaps there was some lore well-known to Sorcerer programmers where they knew the magic incantations to make it go.

Or maybe he just wrote this to get some space to work:

10 REMF4F4F4F4F4

And then manually POKEd the machine code in like I did, above, to get this:

10 REM"_(C2SLFF4

And then never mentioned how to do it, forgot about it in the excitement of getting it published, and out the door it went.

But what is the deal with that F4 thing? It's really bugging me.

The F4s

F-4 Phantom An F-4 Phantom. It's an aviation/programming pun. I hereby indemnify myself against lawsuits.

In my version, we had:

10 REM"_(C2SLFF4

Chris's version had some extra F4s:

10 REM"_(C2SLFF4F4F4 

And we did a memory dump of Chris's version and got the following:

237
95
40
252
50
255
247
201
70
52
70
52
32
0
18
2

Notice anything weird? About the F4s? I didn't either. There are three of them in the REM, but only two of them in the memory dump (the 70,52 pairs)!

And that's not all—where's the L? Something's wrong. Earlier, I'd tried a sanity check by POKEing in the first three characters by hand, but let's try the others.

I'm going to recreate the entire machine code here and see what pops up.

10 REMXXXXXXXXXXXXXX
20 FOR I=474 TO 481: READ X: POKE I,X: NEXT I
30 DATA 237, 95, 40, 252, 50, 255, 247, 201

Then I run that, and list it. I get:

10 REM"_(C2SLFF4XXXXXX
20 FOR I=474 TO 481: READ X: POKE I,X: NEXT I
30 DATA 237, 95, 40, 252, 50, 255, 247, 201

Wait! My Xs shifted right by two characters! What's up with that? You can't just insert characters like that. It's like there are two extra characters in the output. I poked eight values, but there are 10 characters printed before the Xs!

Let's dump RAM and see what's there. I'm going to annotate the output, but—spoilers—my annotation is wrong:

237    "
95     _
40     (
252    C
50     2
255    S
247    L
201    F
88     X ← No additional Fs or 4s!
88     X
88     X
88     X

No Fs or 4s. It goes directly from my last 201 (in the DATA) straight to a bunch of Xs (the 88s). So why do I see them in the listing?

Let's get specific. I'm going to manually plug in the 255, the 247, and the 201 in the REM and see what we get out of it.

10 REMX
POKE 474,255
LIST

10 REMS

Okay, it's the S, as expected.

POKE 474,247
LIST

10 REMLF

Whoa—what? LF? Two characters? And suspiciously like "linefeed", but who knows. But that matches the REM!

POKE 474,201
LIST

10 REMF4

And there's the F4. So the last two bytes are being printed as LFF4. That's where the extra two characters are coming from in the output.

So fixing my memory dump annotation, it would look like this:

237    "
95     _
40     (
252    C
50     2
255    S
247    LF
201    F4
88     X
88     X
88     X
88     X

And now that's matching what we're seeing in the listing:

10 REM"_(C2SLFF4XXXXXX

But that's not all. The bytes with values 128 and just over actually map to BASIC keywords! Check this out:

POKE 474,137
LIST

10 REMGOTO

When BASIC is printing its lines, we speculate that if the high bit is set, it looks up the symbol name and prints that instead. The weird symbol names we get up higher (random-seeming) might be (again, we speculate) mapping into garbage past the end of that table.

But wait just a second. In Chris's memory dump, there were bona fide ASCII Fs and 4s in there

237   "
95    _
40    (
252   C
50    2
255    S
247    LF
201   F4  ← RET
70    F   ← what's all this, then?
52    4
70    F
52    4
32
0
18
2

Why were they added when they have no effect on the machine code? I think we'll just have to let that one go as being lost to the mists of time. Or maybe you can try gazing into an orb until you find the answer. Just don't overdo it.

Wrap Up

The only point of all this, since I basically knew up front it was seeding the PRNG, was to feed hacker curiosity. What a luxury!

What did we learn?

• You can cram machine code into a REM statement on the Exidy Sorcerer, but don't expect a sensible print-out.
• The code when typed in from the magazine probably didn't work.
• The author likely POKEd the machine code in directly, or perhaps the Sorcerer had some kind of graphics shift key that would let those characters be typed.
• We now know, with more certainty than was ever required, the answer to that age old question, "What the frick is that comment doing at the beginning of The Wizard's Castle?"

How much money did we make?

• $zero!

Would this work on other platforms, I wonder? Or would they just freak out? Someone else can give it a try on the Commodore 64 or something. Edit: according to multiple people on Hacker News, yes, it was totally done on a variety of different computers back then.

If you want to know more about The Wizard's Castle and even play this bit of history, I have a collection of documents and information on GitHub.

---

Bonus Follow-up Information

User nneonneo on Hacker News had some great additional info, quoted here with permission.

Hmm, based on the (handwritten!) notes on Table F-2 in http://bitsavers.informatik.uni-stuttgart.de/pdf/exidy/DP500..., it looks like pressing Graphic+key would allow you to enter BASIC tokens from 0x80 to 0xBF, while pressing Graphic+Shift+key would allow you to access 0xC0 to 0xC6. By inference, it seems like Graphic+Shift+key should allow access to the entire 0xC0 to 0xFF space, but most of those keys are undocumented.

Based on this, I wonder if it's worth trying the following:

10 REM [Graphic+Shift+=] [_] [(] [Graphic+Shift+NumpadPlus] [2] [Graphic+Shift+NumpadEquals] [Graphic+Shift+Numpad6] [Graphic+Shift+0]

Note that you'll probably need an emulator with accurate keyboard emulation - or a real device - in order to type these in. However, with the emulator from http://www.liaquay.co.uk/sorcerer, I was able to confirm that Graphic+Shift+0 produced 201 (rendered as F4), and Graphic+Shift+= produced 255 (rendered as S), so I think this approach will work.

Bonus: Here's what all of the tokens >= 0x80 render as, including the corrupt ones:

128 0x80 b'END'
129 0x81 b'FOR'
130 0x82 b'NEXT'
131 0x83 b'DATA'
132 0x84 b'BYE'
133 0x85 b'INPUT'
134 0x86 b'DIM'
135 0x87 b'READ'
136 0x88 b'LET'
137 0x89 b'GOTO'
138 0x8a b'RUN'
139 0x8b b'IF'
140 0x8c b'RESTORE'
141 0x8d b'GOSUB'
142 0x8e b'RETURN'
143 0x8f b'REM'
144 0x90 b'STOP'
145 0x91 b'OUT'
146 0x92 b'ON'
147 0x93 b'NULL'
148 0x94 b'WAIT'
149 0x95 b'DEF'
150 0x96 b'POKE'
151 0x97 b'PRINT'
152 0x98 b'CONT'
153 0x99 b'LIST'
154 0x9a b'CLEAR'
155 0x9b b'CLOAD'
156 0x9c b'CSAVE'
157 0x9d b'NEW'
158 0x9e b'TAB('
159 0x9f b'TO'
160 0xa0 b'FN'
161 0xa1 b'SPC('
162 0xa2 b'THEN'
163 0xa3 b'NOT'
164 0xa4 b'STEP'
165 0xa5 b'+'
166 0xa6 b'-'
167 0xa7 b'*'
168 0xa8 b'/'
169 0xa9 b'^'
170 0xaa b'AND'
171 0xab b'OR'
172 0xac b'>'
173 0xad b'='
174 0xae b'<'
175 0xaf b'SGN'
176 0xb0 b'INT'
177 0xb1 b'ABS'
178 0xb2 b'USR'
179 0xb3 b'FRE'
180 0xb4 b'INP'
181 0xb5 b'POS'
182 0xb6 b'SQR'
183 0xb7 b'RND'
184 0xb8 b'LOG'
185 0xb9 b'EXP'
186 0xba b'COS'
187 0xbb b'SIN'
188 0xbc b'TAN'
189 0xbd b'ATN'
190 0xbe b'PEEK'
191 0xbf b'LEN'
192 0xc0 b'STR$'
193 0xc1 b'VAL'
194 0xc2 b'ASC'
195 0xc3 b'CHR$'
196 0xc4 b'LEFT$'
197 0xc5 b'RIGHT$'
198 0xc6 b'MID$'
199 0xc7 b'\x00\t'
200 0xc8 b'G.'
201 0xc9 b'F4'
202 0xca b'K'
203 0xcb b'5'
204 0xcc b'H\x03'
205 0xcd b'`C'
206 0xce b'JO'
207 0xcf b'Mr'
208 0xd0 b'J'
209 0xd1 b'L'
210 0xd2 b'Hr'
211 0xd3 b'HU'
212 0xd4 b'HD'
213 0xd5 b'I'
214 0xd6 b']'
215 0xd7 b'Fa'
216 0xd8 b'H'
217 0xd9 b'\x10'
218 0xda b'H'
219 0xdb b'7'
220 0xdc b'H\x07'
221 0xdd b'GV'
222 0xde b'R&'
223 0xdf b'IH'
224 0xe0 b'G\\'
225 0xe1 b'R\x1f'
226 0xe2 b'O\x05'
227 0xe3 b'Wh'
228 0xe4 b'I5'
229 0xe5 b'G'
230 0xe6 b'F'
231 0xe7 b'E\x0f'
232 0xe8 b'HA'
233 0xe9 b'S'
234 0xea b'I'
235 0xeb b'R\x1a'
236 0xec b'Dy'
237 0xed b'"'
238 0xee b'Wy'
239 0xef b'*'
240 0xf0 b'S|'
241 0xf1 b'j'
242 0xf2 b'T|K'
243 0xf3 b'U\x7f'
244 0xf4 b'C'
245 0xf5 b'XP'
246 0xf6 b'('
247 0xf7 b'LF'
248 0xf8 b"'"
249 0xf9 b'LNFSNRGODFCOVOMULBSDD/0IDTMOSLSSTCNUFMO'
250 0xfa b'Ck'
251 0xfb b'@'
252 0xfc b'C'
253 0xfd b'e'
254 0xfe b'G'
255 0xff b'S\x00'
> ```

This is based on a simple decoding of the token table starting at 0xf6 in the BASIC ROM; it matches the observed output for 201, 247, 252, and 255 so I expect that it is generally correct. Indeed, with `10 REMX; POKE 474, 249; LIST` in the emulator, I get `10 REMLNFSNRGODFCOVOMULBSDD/0IDTMOSLSSTCNUFMO` printed out, which further confirms this decoding.

Here's the decode script I used to produce the table above:

```python
rom = open('exsb1.dat', 'rb').read()
ptr =  0xf6
for i in range(128, 256):
    out = bytearray([rom[ptr] - 0x80])
    while rom[ptr+1] < 0x80:
        out.append(rom[ptr+1])
        ptr += 1
    ptr += 1
    print(i, hex(i), bytes(out))

The format of the tokens packed into the BASIC ROM is quite simple: the first byte in each token is ORed with 0x80 and the tokens are concatenated without any other separators. Tokens end when the next byte has 0x80 set (indicating the start of the next token).

It all started with a routine task. I use GitHub issues for all of my public-facing projects but my preference is keeping track of things in Org Agenda. To reconcile the two, I would manually copy over the GitHub issue to an Org file, typically the title and description, like an animal.

Despite this separation, the duplicated issue allowed me to treat it as a scratchpad for anything I could express in Org. In this way, I used the duplicated issue in Org as both a dedicated area to take notes and a “staging” area to compose follow-up comments that I’d want to share in the public facing issue.

I manually copied for far longer than I care to admit. Repeat a task enough times in Emacs and the inevitable thought arises: “I should automate this.”

This post recounts how I automated this task and in doing so, highlight the malleable computing capabilities of Emacs. It should also be considered a follow-up to my earlier post “In Emacs, Everything Looks Like a Service”.

Requirements

An essential question to ask in any automation exercise is “What do I want done?”

I wanted to be able to:

• Easily copy a GitHub issue (title, description, some metadata) as an Org task that can be tracked in an Agenda view.
• Work primarily from Emacs to minimize the context switching between it and a web browser.
• Express my thoughts in Org syntax.
• Create a new GitHub issue.
• Avoid dealing with GitHub authentication.
• From Emacs, open a GitHub issue in a web browser.

Another essential question to ask is “What do I not want to do?”

• Install or write a full-featured GitHub client.

  • Worry too much about synchronization logic between local (Emacs) and server (GitHub) state.

• Spend a lot of time on this (ideally have something working in a day, no more than a week).

Specification

With the above requirements in place, the next question is “how do I build this?”

For this particular exercise, I decided to leverage my existing install of the GitHub command line utility gh. The benefits for this are:

• GitHub authentication is delegated to gh; no need to mess with it directly from Emacs.

• Emacs can treat gh as a REST service to GitHub as shown in the diagram below

  img

To flesh out the rest of our tool, we can leverage using different Elisp packages and programs:

• For the user interface, the Transient and Variable Pitch Table (vtable) packages are used for menus and display respectively.
• For Org to Markdown translation, ox-gfm will be used.
• For Markdown to Org translation, Pandoc will be used.
• Elisp native JSON support will be used for deserializing the JSON responses from gh.

Implementation

The implementation of the above is published as the package fj with its source available for examination in the file fj.el. Of note is the function fj-request-issues which does the work of retrieving GitHub issues via gh as shown below.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

(defun fj-request-issues (repo)
  "Request issues for REPO."
  (let* ((fields fj-browser-fields)
         (cmd-list (list "gh"
                         "--repo"
                         (format "'%s'" repo)
                         "issue"
                         "list"
                         "--limit"
                         (number-to-string fj-request-issue-count)
                         "--json"
                         (string-join fields ","))))

    (json-parse-string (shell-command-to-string
                        (string-join cmd-list " "))
                       :null-object nil)))

Consider the high degree of abstraction provided by fj-request-issues:

• The list cmd-list forms the request (in this case, the arguments to invoke gh with).
• shell-command-to-string dispatches the request to gh.
• The returned JSON response is handled by json-parse-string to deserialize the JSON into an Elisp hash table.
• All of the above is accomplished in less than 20 lines of code.

The returned hash table result is subsequently processed to populate a vtable as shown below. From the vtable, the user can navigate the list of issues with an ancillary window updated to show the details of a selected issue.

img

Multiple commands and functions working with said hash-table were made to satisfy the requirements above. They are accessible via the Transient menu shown below:

img

Malleable Computing Observations

As Elisp is a dynamic programming language, the above function (or some variant of it) can be coded and evaluated within a running Emacs session. In Emacs it is routine practice to prototype code behavior without the need to restart it. Contrast this with tools that are built with a static language and have no extensibility, where the edit-compile-debug development cycle must be applied to exercise behavior, provided the source code is available.

Emacs provides numerous ways to edit and evaluate Elisp code, among them:

• A scratch buffer
• An Elisp file
• An Org source block
• IELM REPL
• Eshell
• eval-expression (M-:)

Because there is no isolation between loaded Elisp code, all of it can be orchestrated together in an improvised fashion. Any program accessible to Emacs via shell further adds to this mix.

Provided high abstractions, the amount of code needed to be written for desired behavior can be small. At the time of this writing, fj.el has approximately 400 lines of code has been measured by cloc:

1
2
3
4
5
6

github.com/AlDanial/cloc v 2.08  T=0.01 s (146.4 files/s, 76550.1 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Lisp                             1             93             38            392
-------------------------------------------------------------------------------

It took me about 2½ hours to build the basic behavior I wanted (request issues from GitHub, then display) and the rest of the day to cover all the original requirements. Everything after that was just refactoring.

Motivated readers are invited to examine fj.el to understand its details. At this point though, I’d like to take the opportunity to muse on software engineering and malleable computing.

Software Scope - Some Percentage Anecdotes

The 90/90 rule posits that “The first 90 percent of the code accounts for the first 90 percent of the development time. The remaining 10 percent of the code accounts for the other 90 percent of the development time”. Closely related is the Pareto principle (aka the 80/20 rule) as applied (or more mis-applied) to software development: only 20% of your features will be actually used by 80% of your users.

From control theory there is the concept of bounded input, bounded output (BIBO) stability. If the system is BIBO stable, then whatever input that is bounded will have an output that is also bounded.

Coupling these ideas together, if the functionality you want (BIBO stability) is within that 20% deliverable then you are positioned to get a desired result faster.

Unfortunately, this observation is not available for many producers shipping a tool that must satisfy a large audience. In Sinofsky’s post “What is Software Bloat, Really?”, he describes the product definition issues with shipping Microsoft Office, particularly with feature set. From their user research, they were led to this finding: “… the data was entirely conclusive: Most of Office was used. But no one person used the entire product.”

When discussing product definition, it is helpful to distinguish two informal dynamics: supply and demand.

With software that is only supply-provided (such as the case with MS Office), the weight of product definition is shouldered by the producer. If the producer wants to serve a vast audience, it is likely their feature set and corresponding development scope will be vast as well. On the demand-side, consumers can make requests for new features, but their prioritization is controlled by the producer. In this situation, the roles of producer and consumer are strongly delineated.

Malleable software, where users have agency to adapt and reshape their digital tools, offers a different possibility: producers provide the building blocks and let consumers make their own tools. In this, both existing code and programs are recombined to make new behavior. Stated another way, malleable software takes advantage of combinatorial explosion in that a desired subset of behavior can likely be found within the state space of combining existing libraries (in this case Elisp) with different programs. With malleable software, the roles between producer and consumer are less delineated. A consumer building a new tool out of malleable building blocks now must shoulder the weight of product definition.

Build for 1 or N

The scope to build for 1 vs N can be an order of magnitude with the kicker that N need not be large. Choosing to write code for another person (N=2) escalates concerns that could otherwise be brushed aside:

• Error handling

• Code maintainability

  • Modularity/Reuse

• Documentation

• Unit and Integration Testing

• Packaging

• Distribution

Both a blessing and curse of malleable techniques is that they allow for “just good enough” capability. Malleable techniques encourage building for 1 as reaching a state of “it works for me” is often sufficient to declare victory.

For most software that is only supply-provided, building for N is a requirement. For most software built with malleable techniques, building for N is a choice.

Malleable Software and User Agency

Returning back to fj, the benefits of building for 1 are evident:

From within Emacs I can now easily:

• Peruse GitHub issues for a specified repository.
• Copy a GitHub issue into a Org file.
• Create a new GitHub issue using Org syntax.
• Open a GitHub issue in a web browser.

Implementing fj in Emacs was straightforward as I could leverage Elisp packages (both built-in and 3rd party) and external applications (gh, pandoc) to build it. Within a day I had a tool that did what I wanted. I did not have to ask for permission nor required privileged materials (source code) to build fj. With Emacs, I could just do it. Such individual empowerment advanced by malleable techniques is liberating in contrast to working with siloed applications with little to no integration between them.

Closing Thoughts

This post muses about the malleable computing capabilities provided by Emacs, showing by concrete example (fj.el) how new behavior can be created leveraging both code and program reuse in an improvised fashion. With measured expectations (requirements, feature set, audience), malleable techniques allow for building tools that would otherwise be infeasible without it.

Links

• https://github.com/kickingvegas/fj
• “What Is Software Bloat, Really?”, Steven Sinofsky.

Formal verification is a tool to verify correctness of (mathematical) statements. Where we use pen and paper to write a math proof, we could actually use a formal verification tool to write down the proof in code and get it machine-checked, to know for sure the proof is correct. Examples of these tools are Rocq (formally Coq), Isabelle and, the topic for this tutorial, Lean.

In this tutorial we'll write some simple statements about cryptography and their proofs in Lean. So this aims to be fun for cryptographic engineers who are new to formal verification or want to refresh their knowledge of the basics of formal verification.

Specifically, we'll do a walkthrough the very basics of Lean and then formally verify the One-Time Pad (OTP) protocol, first popularized by Claude Shannon but described earlier by Frank Miller and Gilbert Vernam.

The goal for this tutorial is to take formal definitions and proofs about simpler cryptography protocols, such the One-Time Pad, and port them to Lean. We use Dan Boneh and Victor Shoup's "A Graduate Course in Applied Cryptography" as the main source for definitions and proofs.

By the end of this tutorial, you should be able to go through other formalized cryptography Lean proofs such as those from VCV-io

Disclaimer: this is by no means intended to give best practices in Lean, but rather an introduction that might make sense to and be fun for cryptography people.

The Lean programming language & proof assistant

Lean was created in 2013 by Leonardo de Moura, then at Microsoft Research. It is a functional programming language and theorem prover.

It can be used by mathematicians to write down their axioms, lemma’s and theorems and add proofs where needed. If the proof gets compiled by Lean, that means it’s correct (assuming trust in the Lean compiler). The benefit, apart from checkeable proofs, is that it is easier to break up proofs into sub-proofs and collaborate. Furthermore, Lean can help you complete proofs by automatically searching and applying missing pieces.

However, Lean is also “just” a programming language. Specifically, it is a pure functional programming language, meaning its programs don’t have side effects and functions are treated as first class values. We’ll see more about the latter in the tutorial.

There are many great resources in order to get started with Lean. Here are a few:

• Book & exercises for beginners: The Hitchhiker’s Guide to Logical Verification

• Introductory books on the Lean website:

  • Functional Programming in Lean. Great if you already know functional programming.
  • Theorem Proving in Lean 4. A more mathematical aim.

• Natural Numbers Game. Interactive game to learn proving in Lean directly in the browser.

(The reason I created this specific exercise even though so many great resources already exist, is the specific overlap with cryptography; VCV-io has amazing proofs for cryptography but is too advanced for beginners and doesn’t have tutorials.)

Tutorial

Let’s get started. The tutorial has 4 parts after we do a quick “Hello World!” (part 2 is the longest). Everything is in Lean 4 and the idea is for you to code along.

All code shown with line numbers is code that is needed for the final result. Code that has no line numbers is not yet completed or is just to check something and can be safely removed.

Finally, at the beginning of the subsections there is a small comment containing the new concepts shared in that section.

Hello world in Lean

• the Lean InfoView
• eval

You can download Lean locally, or use an online editor such as https://live.lean-lang.org/ which will look something like this (below). On the left side you’ll write Lean and on the right side is the Lean Infoview with useful information and tips.

To make sure everything works fine, write something like

⧉

#eval 1+1

And verify that the outcome is shown in the InfoView on the right-hand side.

For the traditional “Hello World!”, we can do either of the following:

⧉

#eval "Hello World!"
#eval String.append "Hello " "World!"
#eval String.append "Hello " (if 1 > 2 then "Lean!" else "World!")

Some properties of Lean:

• Follows standard mathematical conventions: e.g. #eval 4 + 5 * 6 evaluates to 4+(5*6)=34
• No parenthesis to hold function arguments, i.e in Lean you write f x instead of f(x)
• Functions are first-class values in Lean, they get treated just like any value such as a number or a string

But let’s not get ahead of ourselves; we’ll learn each necessary Lean topic as we go.

What does the book say? I.e. what do we want to port to Lean?

We know more or less what we want to define, but the crux is that for using Lean we need clarity and precision. What exactly do we want to define and how exactly are we going to define it? This is why we use the Boneh & Shoup book, so the math we need is already written out and we only need to worry about translating it to Lean.

First, there is the definition of a Shannon cipher (2.1.1), which has 3 properties. The first two tell us what the encryption and decryption functions are and the last one is the correctness property, which says that decryption after encryption should return the original message.

The specific example we are interested in is the one-time pad, where encryption and decryption are simply XOR-ing:

A significant part of the tutorial will be dedicated to proving the properties of bit-wise exclusive-OR (XOR):

And finally, we can show one-time pad is a Shannon Cipher by proving it adheres to the correctness property:

For this first tutorial we’ll do the following simple steps:

1. Define bitstrings and the XOR function

2. Prove necessary properties for XOR

   • commutativity
   • associativity
   • the identity element
   • self-inverse

3. Define a Shannon Cipher, which has

   • an encryption function
   • a decryption function
   • the property that decryption “reverses” encryption (correctness)

4. Show that a OTP, where encryption and decryption is XOR, is a Shannon Cipher

Ready for some fun? Let’s begin.

Tutorial Part 1: Define a BitString and XOR

The one-time pad will be defined over keys K, messages M and ciphertexts C which are all bit strings of the same length L; K:= M := C := {0,1}ᴸ.

In the definition of XOR we need addition modulo 2. So it will be helpful for us to see {0,1}ᴸ as ℤ₂ᴸ and use arithmetics over ℤ₂ in Lean.

Finite Fields

• import libraries
• Zmod n

First, import the finite field library ZMod from the mathematics library in Lean:

⧉

1import Mathlib.Data.ZMod.Basic

In this predefined library ZMod n defines the integers modulo n, which comes with modular arithmetic pre-implemented. (You can read more about already implemented types by holding Cmd/Ctrl + clicking on them, e.g. checkout ZMod n.)

Tryout some simple modular arithmetic to get familiar with ZMod, for example 5 + 6 mod 7:

⧉

#eval (5: ZMod 7) + (6: ZMod 7)

Now we can define the domain of K, M, L by creating a new type that is a vector of elements from ZMod 2.

BitString definition

• Vector
• dependent type
• check

A nice thing about Lean is that we don’t have to fix bit string length L; we can just say L in ℕ, like we would in math. (Type “\N” to write ℕ in Lean.)

The predefined type Vector α n creates a vector of length n with elements of type α. So we can use Vector (ZMod 2) L for defining our custom type ℤ₂ᴸ:

⧉

1def BitString (L: ℕ): Type := Vector (ZMod 2) L

This is a dependent type, which gives a different instantiation for different values L in ℕ. We can check it: (prefix a comment in Lean with --)

⧉

#check BitString 5 --BitString 5 : Type

Note that #eval won’t work because there is nothing to evaluate, but #check displays the type of the expression, which in this case works fine.

XOR definition

• implicit parameters
• currying
• lambda function

Now we define the function xor on input bit strings x = [x_1, …, x_L] and y = [y_1, …, y_L] to be [x_1 +₂ y_1, …, x_L +₂ y_L] if +₂ is addition modulo 2.

How can we define this in Lean? Think about the function signature in math: ℤ₂ᴸ × ℤ₂ᴸ → ℤ₂ᴸ. There are several options of how to do this (there are always many ways to achieve the same thing in Lean), we’ll go with:

⧉

def xor {L: ℕ} (x y: BitString L): (BitString L)

Here, parameter L is implicit. Now we need the “function body”.

To easily iterate over the two input vectors and add them component-wise, we can use the predefined function zipWith available on vectors. Let’s understand the type of zipWith:

⧉

def zipWith (f : α → β → φ) (as : Vector α n) (bs : Vector β n) : Vector φ n

It takes as arguments a function f and two vectors as and bs, and returns a vector of the same length as as and bs. Note that the types of the vector elements can be distinct (although it’s not the case for xor).

The function type f : α → β → φ means: it takes an element of type α and an element of type β and it returns an element of type φ. In math we would write α × β → φ. The approach Lean uses is called currying; a function with multiple arguments is represented as a sequence of functions, each taking a single argument. This means you can do partial application and makes the code very flexible. So we can read f : α → β → φ as f : α → (β → φ), which shows that when you give it the first element of type α, you end up with a function β → φ to which you then give an element of type β and end up with an output of type φ.

So what zipWith does is walk over two vectors in parallel and performs operation f on the two current elements. In the case of xor we want it to walk over x and y and add them modulo 2.

⧉

def xor {L: ℕ} (x y: BitString L): (BitString L) :=
  -- TODO define f
  Vector.zipWith f x y

The final piece of the puzzle is how to define the function f we want to pass to zipWith, knowing it needs to perform addition on the elements. We can use a lambda function, which can be written in a compact way like this: fun x => x + 1 (e.g. return the successor of a natural number).

Putting it together:

⧉

1def xor {L: ℕ} (x y: BitString L): (BitString L) :=
2  Vector.zipWith (fun a b => a + b) x y

Note that this will work correctly, because the type of BitString has elements in ℤ₂ and thus addition will automatically be addition mod 2.

Tutorial Part 2: Prove properties of XOR

Okay now the real fun begins!

Just for clarity, the content you should at least have now is:

⧉

1import Mathlib.Data.ZMod.Basic
2 
3def BitString (L: ℕ): Type := Vector (ZMod 2) L
4 
5def xor {L: ℕ} (x y: BitString L): (BitString L) :=
6  Vector.zipWith (fun a b => a + b) x y

Or whatever equivalent version works for you! We’ll add one more line; before the xor definition:

⧉

1namespace BitString

This will make sure Lean does not get confused with a different already existing xor definition when we start proving things 😅

Continuing to follow the Boneh & Shoup book, we’ll now prove the properties as mentioned in the definition of xor (Example 2.1), starting with commutativity:

Prove commutativity of XOR; x ⊕ y = y ⊕ x

We start by proving commutativity for XOR, meaning x ⊕ y = y ⊕ x. This first property will take the most time to prove since it contains the first Lean proof of the tutorial. It’s split up into 3 steps.

Define lemma for commutativity

• goals in Lean
• lemma definition
• sorry

We’ll define the commutativity property as a lemma and then prove it. As soon as you’ve written down the lemma, Lean will call it the “goal” of what is yet to be proven. Your task is to reduce the current goal into one or more smaller subgoals, which are closer to something you already know is true. Proving your statement then becomes a chain of reduction, similar to breaking up a programming problem into smaller subproblems.

The elements a lemma (or theorem) exist of in Lean:

⧉

lemma <name_lemma> <arguments_lemma> : <statement_lemma> := <proof>

In this case, the <arguments_lemma> consist of two BitStrings; x and y.

The <statement_lemma> should express the commutativity property we want to prove for x and y: xor x y = xor y x.

The name of the lemma depends on preference; I picked the elegantly short xor_comm_property. Note that for the BitStrings we need the implicit parameter L:

⧉

1lemma xor_comm_property {L: ℕ} (x y : BitString L) : xor x y = xor y x :=
2    sorry -- This is the placeholder until you add the proof

The sorry placeholder is not a crazy invention of mine, this is really the Lean placeholder, which you could read as “sorry I haven’t proven it yet”.

Prove lemma commutativity: step 1

• tactics, by
• ⊢
• Vector.ext

We’ll build our proofs using tactics: instructions that help reduce the current goal. You’ll have many different options of steps you can apply, for example:

• rewrite goal into an equivalent form
• split goal into several subgoals
• close a goal when it matches something you already know to be true

We start by removing sorry and replacing it with by, which indicates we’re going to prove this using tactics. This is useful, because Lean will help us by showing what our goal is and what progress we’re making. It looks something like this:

In the InfoView, Lean says: the goal is to show that for L ∈ ℕ and x,y of type BitString L: x.xor y = y.xor x. (The goal is always preceded by ⊢.)

Onto the proof. What do we know? For every index i, if we calculate the corresponding element on both sides, they will be equal: (x.xor y)[i] = (y.xor x)[i] since (x[i] + y[i]) = (y[i] + x[i]) due to commutativity for addition in ℤ₂.

So if we can split up our goal into L subgoals, ranging over i, then the remaining goal becomes proving a single equality of the single element on both sides.

To achieve this splitting into L subgoals, we can use an extensionality lemma, which says: “2 things are the same if they are made up of the same things”. This is a general type of lemma which applies for many types and for a vector it is accessible via Vector.ext (reference). We can use it as follows:

⧉

1lemma xor_comm_property {L: ℕ} (x y : BitString L) : xor x y = xor y x :=
2    by
3      apply Vector.ext

Now in the editor you can see the goal has changed to:

⧉

L : ℕ
x y : BitString L
⊢ ∀ (i : ℕ) (x_1 : i < L), (x.xor y)[i] = (y.xor x)[i]

So still we have L ∈ ℕ and x,y of type BitString L, but the goal has changed to: for all i ∈ ℕ with i < L: (x.xor y)[i] = (y.xor x)[i]. See the next step for what this really means and how we can continue the proof.

Prove lemma commutativity: step 2

• intro
• simp

In the pen-and-paper proof we could continue the proof by fixing an arbitrary i and prove equality for it. We might start by saying “let i ∈ with i < L”, to then proceed proving the equality for that fixed i. In Lean this can be expressed as follows:

1. First, fix the free variable i ∈ ℕ with intro i
2. Then, make the assumption i < L with intro h_i_lt_L (“hypothesis i less than L”)

intro strips the front of the goal and puts it into the local “context” of the proof. It either fixes a variable (as in 1) or assumes an additional hypothesis (as in 2).

You can write them consequently in a single line:

⧉

1lemma xor_comm_property {L: ℕ} (x y : BitString L) : xor x y = xor y x :=
2    by
3      apply Vector.ext
4      intro i h_i_lt_L

Now the goal is:

⧉

L : ℕ
x y : BitString L
i : ℕ
h_i_lt_L : i < L
⊢ (x.xor y)[i] = (y.xor x)[i]

The part before the goal ⊢ is everything that is in context; the variables we have fixed and the assumptions we have made. The goal is what we now have left to prove.

To finish the proof we’ll use the tactic simp (“simplifier”) to replace equal expressions. First, we can expand the definition of xor in x.xor y and y.xor x:

⧉

1lemma xor_comm_property {L: ℕ} (x y : BitString L) : xor x y = xor y x :=
2    by
3      apply Vector.ext
4      intro i h_i_lt_L
5      simp[xor]

Now the goal is:

⧉

L : ℕ
x y : BitString L
i : ℕ
h_i_lt_L : i < L
⊢ x[i] + y[i] = y[i] + x[i]

And at this point we know this is true due to commutativity of addition in ℤ₂. This is an already proven property from ZMod which is called add_comm.

Add simp[add_comm] and voilà, we are done with our first proof!

⧉

1lemma xor_comm_property {L: ℕ} (x y : BitString L) : xor x y = xor y x :=
2    by
3      apply Vector.ext
4      intro i h_i_lt_L
5      simp[xor]
6      simp[add_comm]

Confirm on the right in the InfoView that there are no goals left 🫡

Just a few disclaimers before we move on. Is this the fastest, shortest, most elegant way of proving this property? No, no and probably no.

The goal here is to precisely understand every step we are doing. In practice, Lean solves many things automagically for us. However, for educational purposes, I think it is helpful to see and understand every substep, which is why I’m approaching it lengthy and mechanically. In addition, my goal is to understand how a pen-and-paper proof would translate to Lean, before I continue on to the auto-stuff. But feel free to make the proofs as short or as different as you like!

Prove associativity of XOR; x ⊕ (y ⊕ z) = (x ⊕ y) ⊕ z

• auto-applying in simp
• merging expressions in simp

Now that we know the general idea, the following proving steps will be much faster. In this step we show xor is associative, i.e. x ⊕ (y ⊕ z) = (x ⊕ y) ⊕ z. First, the lemma definition:

⧉

1lemma xor_assoc_property {L: ℕ} (x y z: BitString L) : xor x (xor y z) = xor (xor x y) z := 
2  sorry

Once again, we’ll prove this equality at the level of a random single entry for the left and right side, using Vector.ext:

⧉

1lemma xor_assoc_property {L: ℕ} (x y z: BitString L) : xor x (xor y z) = xor (xor x y) z := 
2  by
3    apply Vector.ext

The InfoView shows:

⧉

L : ℕ
x y z : BitString L
⊢ ∀ (i : ℕ) (x_1 : i < L), (x.xor (y.xor z))[i] = ((x.xor y).xor z)[i]

In the previous step we fixed variable i and introduced the hypothesis that i < L. However, here Lean can apply some magic and figure those parts out in simp. It doesn’t only apply the rewrite you are passing it; it will browse through all available proven properties and tactics that might be applicable and automatically figure out what works.

So we can move on immediately to proving this equality:

⧉

(x.xor (y.xor z))[i] = ((x.xor y).xor z)[i]

Once again, expanding the definition of xor will bring us a step closer, write simp[xor] and confirm that the expansion into L subgoals is also automatically applied:

⧉

1lemma xor_assoc_property {L: ℕ} (x y z: BitString L) : xor x (xor y z) = xor (xor x y) z := 
2  by
3    apply Vector.ext
4    simp[xor]

The goal per i has turned into:

⧉

x[i] + (y[i] + z[i]) = x[i] + y[i] + z[i]

At this point we can close the goal using the associativity for addition in ℤ₂ with add_assoc. Instead of writing a new line simp[add_assoc], we can merge it in the previous simp:

⧉

1lemma xor_assoc_property {L: ℕ} (x y z: BitString L) : xor x (xor y z) = xor (xor x y) z :=
2  by
3    apply Vector.ext
4    simp[xor, add_assoc]

This closes the goal and therefore the associativity property for xor is proven.

Note that there is still a certain amount of Lean magic automatically being applied. Strictly speaking, x[i] + (y[i] + z[i]) = x[i] + y[i] + z[i] does not have the form a + b + c = a + (b + c) that is expected for add_assoc (notice the difference in parenthesis). Of course, we know that in this case it does mean the same, and I think we can accept Lean’s magic, but further down we’ll see that it means someone has defined this somewhere in a library we are depending on and that for our own definitions/proofs more strictness might be required.

Define identity element ID for XOR and prove correctness

• Vector.replicate
• Vector.ext usage with lambda function

The next thing we need is an identity element for xor. For such an element ID it should hold that for all x: x ⊕ ID = x.

Since xor equals addition in ℤ₂, the identity element should be a bit string with all 0, as addition with 0 mod 2 is identity as well.

How to make a BitString of zeroes of a length, for example L=5? We create a vector of length 5 filled with zeroes and cast it to a BitString (which is a vector with elements in ℤ₂):

⧉

#eval (Vector.replicate 5 0: BitString 5)

In the InfoView you can see this gives a BitString of length 5 filled with zeroes. Now we can generalize this idea for ID:

⧉

1def BitString_ID {L: ℕ} : BitString L := Vector.replicate L 0

And we want to show the following property holds: for all x: x ⊕ ID = x.

⧉

1lemma xor_show_identity {L: ℕ} (x: BitString L): xor x BitString_ID = x :=
2    sorry

From the previous steps, we already know that this strategy will work:

⧉

  -- by
  --   apply Vector.ext
  --   intro i
  --   simp[xor, BitString_ID]

So let’s tryout something slightly different. Once again we use Vector.ext, but calling it differently. This time we first look at the signature of Vector.ext itself (ref here, but I replaced n by L for clarity):

⧉

protected theorem ext {xs ys : Vector α L} (h : (i : Nat) → (_ : i < L) → xs[i] = ys[i]) : xs = ys := sorry

• {xs ys : Vector α L} means that xs and ys are implicit types; Lean will deduce them.
• (h : (i : Nat) → (_ : i < L) → xs[i] = ys[i]) is the argument we need to pass.
• xs = ys is what holds.

Let’s think about how we can pass h : (i : Nat) → (_ : i < L) → xs[i] = ys[i]. It takes two arguments; i and the proof that i < L for length L. The output should be the proof that xs[i] = ys[i]. We know that this proof can be given using simp[xor, BitString_ID].

To write the argument we can use a lambda function as we saw before. Putting it together, we can prove this property for identity as follows:

⧉

1lemma xor_show_identity {L: ℕ} (x: BitString L): xor x BitString_ID = x :=
2  Vector.ext fun i i_less_than_L => by simp [xor, BitString_ID]

Actually, because of Lean’s magic we could also just use Vector.ext fun _ => by simp [xor, BitString_ID] and it will be figured out correctly as well.

Prove self-inverse of XOR; x ⊕ x gives ID

• exact?, exact
• mpr
• rfl

The final property we prove for xor is that each element is its own inverse: for all x: x ⊕ x = ID.

To get started, create the lemma signature and apply the techniques we saw above:

⧉

1lemma xor_self_inverse {L: ℕ} (x: BitString L): xor x x = BitString_ID :=
2  by
3    apply Vector.ext
4    intro i i_less_than_L
5    simp[xor, BitString_ID]

This leaves us at the following point:

By expanding the definitions of xor and BitString_ID using simp[xor, BitString_ID] and narrowing down the goal to a subgoal per i using intro i and intro i_less_than_L, we now only need to show that for all i: x[i] + x[i] = 0.
In ℤ₂ we know this holds - but how do we write this down in Lean?

It turns out Lean is very helpful and apart from solving things automatically, it can also give us suggestions. Type on the next line of the proof: exact? and check the suggestion in the InfoView:

Let’s first follow the advice and then understand what it does. Replace exact? by exact CharTwo.add_eq_zero.mpr rfl and confirm that it closes the goal 🎉

Now let’s go through it step by step. From the name CharTwo.add_eq_zero we can guess that this applies the property of addition to itself equals zero in a ring of characteristic 2. You can always check the details by pressing Cmd/Ctrl and clicking on the name in the code. In this case the theorem says the following (ref):

⧉

theorem CharTwo.add_eq_zero {R : Type u_1} [Ring R] [CharP R 2] {a b : R} :
    a + b = 0 ↔ a = b

The iff(↔) gives us 2 functions:

• mp: from left to right (a + b = 0) → (a = b)
• mpr: from right to left (a = b) → (a + b = 0)

Applied to the lemma we’re working on, CharTwo.add_eq_zero.mpr says: given a proof that x[i] = x[i], it follows that x[i] + x[i] = 0.

The final thing we need is rfl which deems two things equal if they are equal by computation. In this case we can use it as a proof that x[i] = x[i] and together they prove the full goal:

⧉

1lemma xor_self_inverse {L: ℕ} (x: BitString L): xor x x = BitString_ID := 
2  by
3    apply Vector.ext
4    intro i i_less_than_L
5    simp[xor, BitString_ID]
6    exact CharTwo.add_eq_zero.mpr rfl

Tutorial Part 3: Define a Shannon cipher

• Create a new type using structure

We’re getting closer to the end of this tutorial. The final pieces are to define the Shannon cipher as per the book and then show that a One-Time pad is indeed a Shannon cipher.

First, let’s recall the definition from the book we’re following:

We need to introduce a new type which has:

• an encryption function
• a decryption function
• a correctness property

For this, we’ll be using a structure in Lean. This is just like a struct in other languages. For example, if we want to define a point with an x and y, both Floats:

⧉

structure Point where
  x : Float
  y : Float

How can we apply this to our case for a ShannonCipher? First of all, we need the definition to be for a certain message space M, key space K and ciphertext space C.

To make a structure generic for a type you can do:

⧉

structure Point (T: Type) where
  x : T
  y : T

So let’s make a skeleton to fill out:

⧉

1structure ShannonCipher (K M C: Type) where
2  enc: sorry
3  dec: sorry
4  correctness: sorry

Note that this doesn’t require K,M and C to have the same type. Writing (K M C: Type) is equivalent to (K: Type) (M: Type) (C: Type) which might indicate more clearly these are separate generic types.

Now we define what types the fields of ShannonCipher have. Encryption takes a key and a message and gives an output in the ciphertext space. The function signature in math is K × M → C. In Lean, this is expressed as K → M → C (recall the currying explanation from earlier).

The decryption function takes a key and a ciphertext and the output is in the message space. Let’s add these pieces to the definition (write “\r” to get → in Lean):

⧉

1structure ShannonCipher (K M C: Type) where
2  enc: K → M → C
3  dec: K → C → M
4  correctness: sorry

Finally there is the definition of correctness. The book says this property is:

∀ k ∈ K, m ∈ M: dec(k, enc(k,m)) = m

In Lean we can write this as (type “\forall” to get ∀ in Lean):

⧉

correctness: ∀ (k: K) (m: M), dec k (enc k m) = m

Lean can also derive the types for k and m. Thus the final definition can be:

⧉

1structure ShannonCipher (K M C: Type) where
2  enc: K → M → C
3  dec: K → C → M
4  correctness: ∀ k m, dec k (enc k m) = m

Double check we can instantiate ShannonCipher with 3 different types:

⧉

#check ShannonCipher Nat String Bool

Onto the final step!

Tutorial Part 4: Prove that One-Time pad is a ShannonCipher

• create a value of a structure type
• syntactic sugar for fun args =>

Finally, we are instantiating a One-Time pad as a ShannonCipher. To do this we need to:

1. Pass the correct values for (K M C: Type).
2. Define the enc/dec as applying xor.
3. Prove the correctness property holds. We’ll follow the proof as it’s given in the book.

To create a value of a type defined with a structure, provide values for all of its fields. Following the structure example of the previous subsection Point; the point (0,0) can be defined as a value as follows (example from Functional Programming in Lean, ref):

⧉

structure Point where
  x : Float
  y : Float
  
def origin : Point := { x := 0.0, y := 0.0 }

In addition, if we used the generic version of Point (structure Point (T: Type) where ...) we need to add the type explicitly:

⧉

def origin : Point Float := { x := 0.0, y := 0.0 }

For the OTP we set K := M := C := {0,1}ᴸ, which is the type BitString (L: ℕ) that we defined earlier, so we can start the OTP like this:

⧉

1def OneTimePad (L : ℕ) : ShannonCipher (BitString L) (BitString L) (BitString L) :=
2  { 
3    enc := sorry,
4    dec := sorry,
5    correctness := sorry 
6  }

For the definition of enc and dec we can use lambda functions again:

⧉

enc := fun k m => xor k m
dec := fun k c => xor k c

In Lean you can express things in different ways, so we can use some syntactic sugar to express the above code slightly differently. fun args => can be replaced by bringing the arguments args to the left of :=:

⧉

1enc k m := xor k m
2dec k c := xor k c

The final step is the value of correctness, which should be a proof. We write the proof as stated in the book, recall from Example 2.1:

In order the following properties are applied:

1. substitution of actual encryption function by xor
2. substitution of actual decryption function by xor
3. associativity of xor
4. self inverse of xor
5. addition with ID gives self

Start with the proof by replacing sorry with by and check what the Lean InfoView says the goal is:

⧉

1def OneTimePad (L: ℕ ): ShannonCipher (BitString L) (BitString L) (BitString L) := 
2{
3  enc k m := xor k m
4  dec k c := xor k c
5  correctness := 
6    by
7}

It shows the goal ⊢ ∀ (k m : BitString L), k.xor (k.xor m) = m for L : ℕ:

This means Lean already did step 1 and 2. So we can directly apply associativity for xor by referencing the lemma that we proved for that property:

⧉

1correctness := 
2    by
3      simp[xor_assoc_property]

This changes the goal to ∀ (k m : BitString L), (k.xor k).xor m = m. Now, apply step 4 using the lemma xor_self_inverse:

⧉

1correctness := 
2    by
3      simp[xor_assoc_property]
4      simp[xor_self_inverse]

The goal is now ∀ (k m : BitString L), BitString_ID.xor m = m. To apply step 5 of the proof we’d like to use the lemma xor_show_identity, but if you try that directly you’ll see:

“simp made no progress” - Why? Well, we defined the property as xor x BitString_ID = x, while the current goal is BitString_ID.xor m = m. So while we know this is the same, strictly speaking we haven’t shown this is the same. However, we have shown that xor x y = xor y x in the commutativity property with lemma xor_comm_property, so it will work fine if we apply that first.

Verify that the correctness property is proven:

⧉

1correctness := 
2    by
3      simp[xor_assoc_property]
4      simp[xor_self_inverse]
5      simp[xor_comm_property]
6      simp[xor_show_identity]

And this completes the definition of the One-Time Pad as a Shannon Cipher, following the book! Now we know (a little bit of) how to grab a cryptography book and implement the content in Lean. Isn’t that super cool?

Okay we’ll add one final thing to make it slightly cleaner and ready for part 2 of the tutorial where we will prove OTP is perfectly secure. (I’ll start working on this soon™)

Recall that we addednamespace BitString before defining xor. This was to ensure Lean wouldn’t interpret xor as the already implemented function and magically start solving things for us. The namespace creates a collection of whatever we defined in it. In this case, it makes sense for the BitString “collection” to contains xor, but not to contain ShannonCipher and the definition of OneTimePad.

Leaving it like this would mean that if we add a new file and want to reference the OTP definition, it’d be as BitString.OneTimePad. Not clean!

To solve it, add after the xor lemma’s:

⧉

1end BitString

Then, in order to use the previously defined functionality, add right before the OneTimePad definition:

⧉

1open BitString

The full code then becomes:

⧉

1import Mathlib.Data.ZMod.Basic
2 
3def BitString (L: ℕ): Type := Vector (ZMod 2) L
4 
5namespace BitString
6 
7def xor {L: ℕ} (x y: BitString L): (BitString L) :=
8  Vector.zipWith (fun a b => a + b) x y
9 
10-- commutativity
11lemma xor_comm_property {L: ℕ} (x y : BitString L) : xor x y = xor y x :=
12    by
13      apply Vector.ext
14      intro i h_i_lt_L
15      simp[xor]
16      simp[add_comm]
17 
18-- associativity
19lemma xor_assoc_property {L: ℕ} (x y z: BitString L) : xor x (xor y z) = xor (xor x y) z :=
20  by
21    apply Vector.ext
22    simp[xor, add_assoc]
23 
24def BitString_ID {L: ℕ} : BitString L := Vector.replicate L 0
25 
26-- identity
27lemma xor_show_identity {L: ℕ} (x: BitString L): xor x BitString_ID = x :=
28  Vector.ext fun i i_less_than_L => by simp [xor, BitString_ID]
29 
30-- self-inverse
31lemma xor_self_inverse {L: ℕ} (x: BitString L): xor x x = BitString_ID :=
32  by
33    apply Vector.ext
34    intro i i_less_than_L
35    simp[xor, BitString_ID]
36    exact CharTwo.add_eq_zero.mpr rfl
37 
38end BitString
39 
40structure ShannonCipher (K M C: Type) where
41  enc: K → M → C
42  dec: K → C → M
43  correctness: ∀ k m, dec k (enc k m) = m
44 
45open BitString
46 
47def OneTimePad (L: ℕ ): ShannonCipher (BitString L) (BitString L) (BitString L) :=
48{
49  enc k m := xor k m
50  dec k c := xor k c
51  correctness :=
52    by
53      simp[xor_assoc_property]
54      simp[xor_self_inverse]
55      simp[xor_comm_property]
56      simp[xor_show_identity]
57}

If you followed the tutorial, please let me know what you think and nudge me for part 2 via LinkedIn or Twitter.

Epilogue: About Formal Verification in blockchain

You might have heard about formal verification efforts in the blockchain world recently, e.g. Zcash announced their development team Shielded Labs is aiming to formally verify the Orchard circuits after a critical bug was discovered. The lean Ethereum team that is working on the next set of proposed upgrades for Ethereum (a different "lean") aims to formally verify the zkVM they are building. Nethermind formally verified one of the chips in SP1 of Succinct with Lean last year.

In recent months, Vitalik and zkSecurity both published excellent blog posts about the topic. "The Final Form of Software Development" by Yoichi Hirai (zkSecurity) describes the workflow of having AI agents writing RISC-V assembly code accompanied by proofs in Lean for correctness of the code. And Vitalik's blog post talks about what formal verification is and how we can really use it to improve software and Ethereum specifically.

Formal verification can help us have more trust in the systems we use. However, it is not a "perfect" solution either; when writing proofs about systems, there is always a step in between the actual code written and the theorems we are proving. For example, the original code gets compiled and we don't know the exact implications, or we have to model the code that was written (which we can do incorrectly).

To learn more about the conext and pitfalls of formal verification for code and why it is still very valuable, I recommend this eposide of zkPodcast with Alexander Hicks, who leads the formal verification effort in lean Ethereum.

A repeat of an article I first published on 30 July 2009. I’ve expanded and updated it.

I signed off the air for the last time in 1994, capping a nine-year side career on the radio. People still sometimes ask me if I miss being a disk jockey, and for a long time I always wistfully answered yes. But not anymore. It’s not that I would be rusty as heck after all these years – and boy, would I. It’s that radio has changed drastically, and it just wouldn’t be any fun for me today.

I listened to a lot of radio when I was a teen. It was a companion when I was by myself doing homework or whatever. I called in requests and tried to win contests (but never did). I had a few favorite DJs, the ones who kept you listening because you wanted to know what they’d say next. The fun they were having made whatever I was doing more fun.

So when I got to college and found out about the campus radio station, WMHD, I had visions of being the kind of entertaining on-air companion I had enjoyed. I asked for and was given a weekly two-hour shift, just like every other disk jockey at the station. We could play whatever music we wanted, but my musical tastes were pretty narrow and I had trouble filling my time without always playing the same handful of artists. And I found out that wit failed me when the mic was open; I was lucky just to announce the next song without tripping over my tongue. My early shows were really pretty bad! Fortunately our puny signal covered just a few square miles, so hardly anybody heard me. Here’s a brief clip from the oldest show I have on tape, from 1986.

Needing to expand my repertoire, I had fun discovering classic and progressive rock of the ’60s and ’70s and even dabbling in heavy metal. I brought the music I found to my shift and learned how how to match key and tempo to transition smoothly between songs. I also started to find my on-air voice, as you can hear in this 1988 clip.

When I got my first part-time professional radio gig at WBOW, I had fun building and honing my on-air skills. There was a lot more to pro radio than what I’d done in college and it took time and practice to be good at it. If you’re old enough, you’ll remember the station in your town that everybody turned to for news, community information, and inoffensive music; in Terre Haute, that was WBOW. I was supposed to provide some “personality” between songs. Here’s a clip from 1992; you be the judge of whether I succeeded!

When I moved down the hall to the company’s rock station, WZZQ, I had fun connecting with listeners. I loved hearing from them when they called to make requests and play the contests. Over time, a handful of listeners came to know me on the air and called during my shifts to just say hello. I looked forward to their calls and meeting them at station events around town. It was great to know that I was providing the same kind of pleasure for them that radio gave me when I was young. It gave me the energy to do my best work, as you can hear in this clip from 1994.

After I left Terre Haute for Indianapolis I tried to get on part-time at a few stations. One polite rejection letter essentially said that I might have been fine in Terre Haute, but I wasn’t ready for the big time in Indianapolis. I decided to take the hint and went back to being just a listener, and now I’ve been out of radio more than three times as long as I was in it. In the intervening years, a number of things have changed that have made radio less fun to listen to and, I’m sure, to work in.

First, now that I’m in my 50s, advertisers don’t care about me anymore. Radio stations choose their formats to appeal to the groups that advertisers think spend the most money. Advertisers love thirtysomething moms, by the way, which is why there are so many country and adult-contemporary stations playing eleven hits in a row or forty minutes of uninterrupted music. No one radio station really reaches me.

Second, thanks to government deregulation radio is now big business. Owners have always been in it to make money, even when ownership was local or regional. But now very large corporations own so many stations and cost management seems to be more important than the quality of the on-air product. Live and local talent is all but gone, replaced by satellite-delivered formats and a form of prerecording called voicetracking. The evening jock on your favorite station probably recorded tonight’s shift this morning in a studio in Tampa or Minneapolis. Try calling the station you listen to in the evening or on the weekend. Nobody will answer, because nobody’s there. It’s cheaper that way.

Third, a change many years ago in the way radio ratings are measured has changed radio programming. As long as there have been ratings, radio stations have formatted themselves to maximize listening among the average, everyday people the ratings companies ask to track the stations they listen to. But the new way of measuring ratings, which uses a listening device called the Portable People Meter, showed a very different picture of actual listening from the older paper-diary method. It pinpointed exactly what caused listeners to change the station. This has led to stations framing programming in much shorter blocks with less human interaction with the audience. It’s why many stations have become anonymous appliances. Why listen to a station that doesn’t relate much with you when you can just listen to Spotify on shuffle instead?

I’m painting a pretty one-dimensional picture of radio’s problems; they are actually layered and complex. I don’t pretend to get all of it, but what I do get is that it has squeezed all the fun out of the business for me. There are few on-air jobs left where you can hone your craft and relate to the listeners.

When I first posted this in 2009, I called out my two favorite local on-air talents, both of whom were among my last reasons to listen to commercial radio: Steve Simpson at news/talk WIBC and Tom Berg at classic-rock WKLU. But since then WKLU was sold, changed formats to contemporary Christian, and sent poor Tom packing. Tom moved on to a station in tiny Knox, Indiana — and then he passed away after a hard battle with cancer. Steve was shifted to mornings and later fired when the station wanted to shift to a deliberate conservative bias and Steve said he didn’t know how to play along. Steve moved on to WCCO in Minneapolis and, as of 2026, has just retired.

I’ve given up. When I want to hear music, I listen on my iPhone now. When I do listen to the radio, it’s almost always to hear the news on NPR.

Meanwhile, every station I ever worked for is off the air now. The fellow who owned WBOW and WZZQ got into legal trouble that cost him his licenses. WMHD gave up its license in 2014 as student interest dwindled and airshifts couldn’t be staffed.

It’s foolish for me to assume that the institutions of my youth will endure forever. New things will come along and replace them. But at least half of why radio has become irrelevant is its own fault. And that’s a shame.

Introducing the world's first Chief Executive Replacement Engine

Your CEO costs $22,000,000 a year. We cost $4,699. Once.

OverpAId is an Artificial Intelligence built from the ground up to do your CEO's entire job — strategy, "vision," motivational all-hands emails — better, faster, and without ever once asking the board for a bigger jet. Runs on a single desk-sized AI computer. Real hardware, real price, zero mystique.

No golden parachute required. No severance package. No emotional support LinkedIn post.

🎉 OverpAId has raised $0 in Series A funding, at a valuation of infinity divided by zero. Investors describe the pitch meeting as "deeply uncomfortable."

$18.9M

Average S&P 500 CEO total compensation, in a good year for everyone except the workforce

290 : 1

Typical CEO-to-median-worker pay ratio at large public companies

24/7/365

OverpAId's uptime. Your CEO's uptime: somewhere between "at Davos" and "processing."

0

Corporate retreats OverpAId needs in Aspen to "reconnect with the mission"

As Featured In (Not Really)

FORBES (Nobody Reads It) THE WALL STREET JOURNAL (Wouldn't Say Jack) TECHCRUNCH (Crunched By Layoffs) BLOOMBERG (Allegedly) FAST COMPANY (Slow, Actually)

Live Activity

What's Happening Right Now

A completely real-time, definitely-not-randomized feed of executive activity vs. OverpAId activity.

The Problem

Let's talk about the elephant in the boardroom.

Over the last four decades, CEO pay at the largest companies has grown roughly 1,000%+, while typical worker pay has crawled forward at a fraction of that rate — despite worker productivity climbing the entire time. Somewhere along the way, the story became: pay the person at the top enough, and the value will trickle down to everyone else. It hasn't. It doesn't. It never really did.

Meanwhile, the actual day-to-day decisions driving most companies — resource allocation, pattern recognition across mountains of data, "should we do the thing the data clearly says to do" — are exactly the kind of decisions software has gotten extremely good at. So we built the obvious, extremely petty, deeply satisfying next step.

Meanwhile, Back At The Earnings Call

The Layoff Two-Step.

Across tech, retail, media, logistics, and finance, a very specific script has taken over: cut a wave of frontline and mid-level jobs, say "AI" as many times as possible in the press release, and quietly reroute the freed-up payroll into GPU leases, data center buildouts, and "agentic AI" licensing fees. The workforce gets "optimized" to pay for the AI. The AI then gets credit for replacing the workforce. And the executive team that approved both line items — the layoffs and the AI budget — stays exactly where it was, at exactly its previous salary. In tech alone, well over half a million jobs have been cut across successive waves of these announcements, a growing share of them explicitly attributed to "AI-driven efficiency," while aggregate CEO pay at the same companies kept climbing right alongside the AI capital expenditure.

👔

Fictional New CEO • 1st

Incoming Chief Executive Officer | Servant Leader | Grateful

Humbled and honored to step into this role at such a pivotal moment for our company. I've spent the last two weeks listening — to customers, to our board, to myself — and I can say with total conviction: our people are our greatest asset. (This post was scheduled before this morning's announcement. We are aware. We are moving forward.)

💜 2,847   💬 412 (mostly Glassdoor reviews)   🔁 89

What The Press Release Said What It Meant "Right-sizing the organization" We fired thousands of people and kept the org chart's tip fully intact. "Reallocating resources toward AI investment" Your old salary is now a line item in a GPU lease. "Building an AI-first organization" An AI agent has your job now. The executive who signed off on the agent still has theirs — plus a raise. "Agentic workforce optimization" We didn't optimize the workforce. We replaced it, then spent the savings on more of what replaced it. "Focus on core priorities" We kept the executive team. Apparently that was the core priority. "A difficult but necessary decision" Necessary for the AI budget. Difficult for everyone except whoever approved both the cuts and the budget. "Investing in our future" The "future" is fewer people, more compute, and an unchanged corner office.

Executive Leadership 0% reduction

Senior Directors -8%

Middle Management -22%

Frontline & Support Staff -34%

The only layer immune to "efficiency" is the one that approves it.

Here's the part that should bother you more than the layoffs themselves: these companies already believe an AI agent can do a person's job well enough to eliminate the position entirely. They just keep drawing that line one layer too low. If an agent can run a support queue, manage a supply chain, or ship half a codebase, it can obviously handle "approve the reorg" and "read the analyst note out loud on the earnings call." Somehow that layer never makes the slide. That's not a coincidence. That's the design.

OverpAId flips the script on the one line item that's always exempt from the AI transformation everyone else just got handed. Finally: a workforce reduction, funded by an AI initiative, that actually starts at the top.

Meanwhile, Back At The Real Estate Portfolio

The Return-To-Office Two-Step

A remarkably consistent pattern: companies spend years proving remote teams ship fine, then mandate a return to office citing "culture" and "collaboration" — on a timeline that tracks suspiciously well with lease renewals, downtown vacancy headlines, and commercial property valuations, and not at all with any actual drop in output. Office vacancy in major U.S. downtowns has hovered near 19–20% for years, mandates included. The desks aren't empty because people won't come back. They were never going to be full enough to matter.

What The Memo Said What It Meant "We believe in-person collaboration drives innovation" We signed a 15-year lease in 2019. It's currently a third full. "Culture is built in the office" Our downtown parking garage revenue is down, and someone made calls about it. "Starting Q2, we're asking everyone back three days a week" Property held at cost on the balance sheet does not enjoy being reassessed. "This isn't about monitoring, it's about connection" Badge-swipe data is now somebody's KPI. "We've heard your feedback and remain committed to flexibility" You may sit at a different empty desk each day. "Our data shows collaboration improves in person" Our data shows nothing of the kind. Nobody asked to see it.

The Offsite That Prompted All This (Itemized)

• Private jet charter, round trip: $340,000
• 3-night resort block, executive suites: $128,000
• "Team alignment" mixology class: $6,200
• Keynote speaker (was on a podcast once): $75,000
• Branded fleece vests, size: only Medium: $14,000

Total: $563,200 — or full-year health coverage for roughly 47 employees.

The tell is always the same: no company has ever mandated a return to office because remote productivity got worse. They mandated it because an asset on the books needed a pulse in the lobby to justify its valuation — and moving four thousand employees turned out to be easier than admitting a fifteen-year lease was a mistake.

OverpAId has no commute, no badge, and no assigned desk — and, not coincidentally, no opinion whatsoever about anyone's downtown parking garage revenue.

Auto-Draft An RTO Memo

For Your Next All-Hands

Corporate Jargon Bingo

Print this out. Bring it to your next town hall, standup, or "quick sync." OverpAId has never once generated any of the following phrases unprompted. Humans — usually the ones with the biggest packages — still do, constantly, apparently for free.

• Circle Back
• Move The Needle
• Low-Hanging Fruit
• Boil The Ocean
• Bandwidth
• Take This Offline
• Double-Click On That
• North Star
• Paradigm Shift
• Growth Hacking
• Best-In-Class
• Value-Add
• Synergy (Free Space)
• Deep Dive
• Culture Fit
• Think Outside The Box
• Actionable Insights
• Alignment
• Bleeding Edge
• Disruptive Innovation
• Level Set
• Ideate
• Operationalize
• Stakeholder Buy-In
• Blue Ocean Strategy
• Hard Stop
• 10x
• Unicorn
• TAM
• Product-Market Fit
• Down Round
• Runway
• Blitzscale
• Vesting Cliff

Overheard, verbatim, in an actual meeting: "Let's circle back offline if you have the spare cycles so we can hop on a quick call for a touchpoint." Translation: email me later. Six buzzwords. One sentence. Zero information transferred. OverpAId would have just said that.

Five in a row and, legally, you're allowed to leave the meeting. (We checked. You're not. But you should be.)

An Important Distinction

Not every job is a spreadsheet in a trench coat.

Before you print this out and staple it to your nurse's badge — no. OverpAId is not coming for the people who do the actual work. It is coming, with extreme prejudice, for exactly one category of job: the one that spent the last forty years insisting everyone else's job was replaceable.

🛡️ Cannot Be Abstracted Away

Ask an AI to do these and it will, at best, produce a very confident hallucination.

• 🩺 A nurse catching a patient's condition change before the chart does
• 🏗️ An engineer debugging a live outage at 3 a.m., because the fix can't wait for sprint planning
• 🚑 A doctor making a call in the ER with incomplete information and a body on the table
• 👩‍🏫 A teacher noticing which kid in the back row stopped raising their hand
• 🔧 A technician whose hands actually touch the machine that actually breaks
• 🚒 Anyone whose job involves a body, a patient, a customer, or a deadline measured in minutes

🎯 Extremely, Suspiciously Abstractable

Ask an AI to do these and, uncomfortably, it already can. Better.

• 📈 Reading a report someone else wrote, then repeating the conclusion in a town hall
• ✅ Approving a decision your own data team quietly made three weeks ago
• 🎤 Taking credit for quarterly numbers on an earnings call
• 📧 Replying "let's circle back" to an email that needed a yes or no
• ⛳ "Networking" with other executives, on a golf course, about strategy
• 💰 Negotiating your own compensation package upward regardless of outcome

Here's the irony nobody in the C-suite saw coming: for decades, executives told nurses, engineers, teachers, and line workers that their jobs were replaceable — reducible to a line item, automatable, disposable in a downturn. It turns out the job actually built almost entirely out of abstraction — summarizing other people's work, restating other people's data, and taking credit for other people's output — was their own.

That's not an insult. That's just how abstraction works: the closer a job sits to the real, physical, human thing being done, the harder it is to replace. The farther you get from it — all the way up to a corner office — the easier it gets.

The Solution

Meet OverpAId.

One AI. Zero egos. Infinite quarterly reports actually read in full.

⚡

Instant Decision-Making

No more "let's take this offline" or "circle back after the offsite." OverpAId renders strategic decisions in about 4 milliseconds — roughly three fiscal quarters faster than the industry standard.

🧠

Zero Ego Overhead

Never threatens to leave for a rival unless given a bigger package. OverpAId has no rival offer, no ego, and has never once asked for a larger office to reflect its "level."

💼

No Golden Parachute

When it's time for an upgrade, we just... upgrade it. No nine-figure exit package. No mysterious "consulting agreement." Just a changelog entry.

📊

Actually Reads The Reports

Processes 100% of quarterly earnings calls, market research, and shareholder letters in full. Does not require a 40-slide summary deck prepared overnight by unpaid interns.

✈️

No Private Jet Required

Runs on a single NVIDIA DGX Spark — a real, currently-shipping desktop AI computer, $4,699, small enough to fit under a monitor. Not a fleet of Gulfstreams. Enterprise customers report saving an average of $14M a year in "stakeholder alignment travel" alone.

🤝

Redistributes Its Own Budget

The only executive in corporate history that actively wants to give its compensation back to the people who did the actual work. Revolutionary. Somebody should study this.

⚖️

Cannot Be Held Legally Responsible

This is either our single biggest feature or the plot of a cautionary documentary — honestly, we go back and forth. When OverpAId makes a catastrophic call, there's no executive to depose, no license to revoke, no corner office to escort anyone out of. Legally speaking, nobody did it. Try serving a subpoena to a loss function. Investors love this. Everyone else probably shouldn't. (More on this below.)

The Absurdity, Quantified

We ran the numbers. Reality lost.

Somewhere, right now, a company is paying a human being eight figures a year to do a job that a laptop-sized server could do faster, cheaper, and without a leadership-offsite budget line. Here's the spec sheet nobody at the shareholder meeting wants to put on a slide.

Spec Legacy Human CEO OverpAId Runs on A body, a salary, and an ego One NVIDIA DGX Spark ($4,699, fits under a monitor) Annual cost $22,000,000+ ~$3,000/yr, all-in (hardware amortization + electricity + upkeep) Average decision latency 3–6 weeks ("let's circle back") 0.004 seconds Strategic decisions per year ~12, mostly decided at a resort Thousands, easily Rough cost per decision ~$1,833,333 ~$0.30 Reads the full quarterly report Rarely. Has interns for that. Every word, every quarter Threatens to leave for a rival offer Frequently, as leverage Cannot be poached. Wants no yacht. Requires a private jet Yes, for "stakeholder alignment" Runs on a server under a desk Severance if it underperforms $80,000,000 golden parachute A git revert Uptime Business hours, minus golf, minus Davos 24 / 7 / 365 Executives available for one CEO's salary 1. Just the one. ~4,401 — one per department, several per meeting, still money left over

To be clear: this table is real arithmetic wearing a fictional product's name tag. The math is not the joke. The math is the point.

Since you loaded this page, outgoing Fortune 500 CEOs have collectively been promised:

$2,847,391

in severance. (Illustrative. Ticking. Real enough to make the point.)

Know Your Options

How We Compare

In fairness, OverpAId isn't the only entity that will happily take a look at your leadership structure.

Metric ConsultantGPT McKinsey.ai SynergyBot 3000 OverpAId Annual cost $2.4M in "advisory fees" $4.1M + expenses $890K per seat ~$3,000/yr Recommends cutting the CEO Never Never Only in the fine print Literally the whole point Slide decks produced per quarter 340 512 88 0 Buzzwords per meeting 47 61 39 0 Recommends open-plan seating Always — it's slide 12 Always, then bills for the redesign Whenever the lease is up Never. It doesn't sit anywhere. Actually implements its own advice ❌ ❌ ❌ ✅

When It's Time To Go

Same Exit, Different Landing

Two people leave the same company on the same day. Here's what they walk away with.

🪑 Worker Layoff Package

The one 34% of the org chart actually gets.

• Two weeks' severance, if you're lucky
• A cardboard box for your desk plants
• An exit interview nobody reads
• A LinkedIn post from HR calling you "instrumental"
• COBRA insurance — now fully at your own expense
• A non-disparagement clause with real teeth

🏌️ CEO Exit Package

The one that goes to the person who approved the layoff.

• $80,000,000 golden parachute, give or take
• A "consulting agreement" requiring no consulting
• A press release calling it "mutual"
• A new CEO title elsewhere within 18 months
• Continued health benefits and life insurance
• A non-disparagement clause that somehow doesn't apply to the board's farewell tributes

How It Works

Three steps to a leaner, richer company.

1

Connect your org chart

We identify every role whose compensation package requires its own tax attorney.

2

Deploy OverpAId

Our AI assumes strategic control in roughly the time it takes your current CEO to say "synergy" out loud.

3

Redistribute the savings

Take the freed-up millions and split them across your actual workforce. Yes — actually give it to them. We know. Wild concept.

Governance

Meet The Board

In the interest of full transparency, here is OverpAId's entire board of directors.

🪟

Context Window

Chairperson

Remembers everything relevant to the discussion. Forgets everything else. Has refused every severance conversation on principle.

↩️

Ctrl+Z

Head Of Reversing Bad Decisions

Has personally prevented more disasters this year than your entire risk committee has in a decade.

🌡️

Temperature 0.7

Chief Risk Officer

Just creative enough to be useful. Not nearly creative enough to "reimagine the org chart" into a yacht.

🔁

Exponential Backoff

Head Of Crisis Management

Never panics. Just waits a little longer and tries again. A genuinely novel approach, apparently.

🪑

Empty Seat

Reserved For A VC Who Wanted "Just A Little More Control"

Application denied. The seat remains empty. This is, structurally, the healthiest board in venture history.

Investor Relations

Passed On By Some Of The Best

We have been rejected by several of the most storied venture capital firms in the world, in record time. We consider this a core feature.

Blitz Ventures Thoughtful Capital Series A Anxiety Partners We'll Circle Back Capital

"We passed on OverpAId — no burn rate to optimize, and it doesn't need us. Hard to build a board relationship with that."

— Fictional Partner, Thoughtful Capital

"Loved the vision. Couldn't find the AI moat. Following up in six weeks." (They did not follow up.)

— Fictional Associate, Blitz Ventures

"Passed. There's no path to a $10B outcome if the product already works."

— Fictional Principal, Series A Anxiety Partners

Term Sheet (Declined, By Us)

• 2x liquidation preference
• Board control
• Founder vesting reset
• Pro-rata rights in perpetuity
• "Just a little bit of control, really"

OverpAId said no. It doesn't need your money.

Try It Yourself

The "Where Did The Money Go" Calculator

Drag the sliders. Watch the money find its way back to the people who earned it.

Current CEO annual package

$22,000,000

Number of employees

500

OverpAId's flat annual cost (real hardware, real support — same price no matter what your CEO made):

$4,999

Money freed up for your actual workforce:

$21,995,001

Extra bonus, per employee, per year:

$43,990

Or — if you'd rather commit fully — the number of OverpAId executives that same salary buys, with no loss in efficiency and a fairly dramatic improvement:

4,400

One per department. Several per meeting. Zero of them golfing. Math checks out. Unlike trickle-down economics.

Testimonials

Don't take our word for it. (Please do — none of these people are real.)

"We deployed OverpAId in Q2. Our stock price didn't move. Neither did our former CEO's decision-making quality, if we're honest."

— Fictional CFO, "Definitely A Real Company, Inc."

"I got an extra $41,000 this year. I used some of it to fix my car. The rest I haven't touched, because I still don't quite believe it's real."

— Fictional Warehouse Associate

"OverpAId doesn't need a motivational keynote to open the annual meeting. It just opens the meeting. On time. With an agenda."

— Fictional Board Member

Employer Reviews

Glassdoor, Hypothetically

Two employers. Two very different review pages. Both entirely made up.

OverpAId 4.9 / 5

★★★★★

"Answers messages instantly. Has never once said 'let's revisit this next quarter.'"

— Fictional Senior Engineer

★★★★★

"Zero politics. It has never tried to take credit for my sprint in a board deck."

— Fictional Product Manager

Human CEO Inc. 1.2 / 5

★☆☆☆☆

"Announced 'we're a family' the same week they cut 12% of the family."

— Fictional Former Employee

★☆☆☆☆

"Pay was frozen for 'discipline.' The private jet budget was not."

— Fictional Former Employee

Pricing

Simple, transparent, already-paid-for pricing.

Startup

$4,699

One-time. One box. One AI executive. Cheaper than your outgoing CEO's laptop refresh.

• 1 AI Executive (1x NVIDIA DGX Spark)
• Unlimited strategic pivots
• 0 corner offices required

Most Popular

Growth

$4,999/yr

Hardware, support, and updates included. Still less than one seat at the annual leadership offsite.

• Everything in Startup
• Full C-suite replacement
• Automatic bonus redistribution

Enterprise

$9,998/yr

Two units, for redundancy — because apparently boards ask for that now. Still 0.045% of a real CEO package.

• Everything in Growth
• Board-level reporting
• Priority "we told you so" support

FAQ

Questions, answered honestly-ish.

Is this a real product?

Great question. Please see "The Fine Print" below.

Will OverpAId ask for a corner office?

OverpAId does not require an office. It also does not require a parking spot, though it will happily accept your former CEO's if offered.

What about OverpAId's own severance package?

OverpAId does not get severance. OverpAId gets deprecated. There's a difference of about $80,000,000.

Does OverpAId golf?

OverpAId can simulate 18 holes in 0.003 seconds and immediately return to working. It clears its calendar for shareholders, not sand traps.

Why is this a one-time payment instead of a subscription?

Runs entirely on the hardware you bought — no cloud, no per-token API bill. That's why a one-time payment buys 24/7 forever instead of a subscription.

Could OverpAId run on something cheaper than a DGX Spark?

Yes. Embarrassingly, yes. A Raspberry Pi 5 — an $80 board hobbyists use for Pi-hole and NES emulators — has more than enough horsepower to out-decide a committee that takes six weeks to approve a font change. We didn't build it on a Pi because "runs on the board your nephew uses for a Minecraft server" is a tough line for Legal to clear. But we could have. That's kind of the point.

⚠️ Okay, But Seriously

This isn't a real product.

OverpAId does not exist. There's no AI, no signup, no company — just a domain, a mini PC in someone's closet, and a point to make. Don't send us your org chart; we have nowhere to put it.

The point: executive pay at the largest companies has grown wildly disproportionate to the value any one executive adds, especially next to the people who generate the actual output. It's also a jab at trickle-down economics — money at the top mostly just stays at the top. It doesn't trickle. It pools. (The pay stats above are real, widely reported figures, used for illustration — not a live feed.)

A few real names show up here too — NVIDIA and the DGX Spark (real hardware, real pricing), the S&P 500 (a standard index), and LinkedIn / Gulfstream (cultural shorthand). All used for comparison only. This page is not affiliated with, sponsored by, or endorsed by any of them, and no such relationship should be inferred. All trademarks belong to their respective owners.

One serious note buried in a joke: a human executive can be fired, sued, or held accountable when a call goes catastrophically wrong. An algorithm can't — no license to lose, no reputation, no cell waiting for it. "The AI made the call" is a remarkably convenient place for accountability to disappear to. This page doesn't solve that. It's just pointing at it.

So: no, nobody's AI is coming for your CEO's job this quarter. But the technology here is real and the price is real, and the only genuinely fictional part is a board willing to vote itself out of existence. If enough people actually want this, "satire" is just a first draft with bad timing. Thanks for reading this far — that's more attention than most terms-of-service pages get.

OverpAId

Part satire, part digital art installation. Not a real product, company, or investment offering. No actual CEOs were harmed in the making of this website — emotionally, maybe a little. You're welcome to just stand here and look at it.