The release of Kimi K3 has opened a fresh round of angst and confused discourse. There's a loud cohort of journalists, business leaders, and politicians arguing that open source AI is a dangerous threat. OpenAI's Dean Ball:

One probable outcome of an open-weight-model-dominant world is full AI communism... rather than a market product, AI is a "public good"

Freely available AI for anyone? The horror!

Frontier labs' case against open source AI is essentially: Open source models1 are dangerous (and un-American!). We should open the AI Pandora's Box, but only with responsible gatekeepers (toll collectors, preferably us!). Only trusted users (our most profitable customers) should be able to use it.

I want to address some bad arguments against open source AI, but some corrections on how the argument is being framed are in order:

Open source software is the foundation for commercial software

Ball's framing strolls past the fact that open source software is the foundation of all proprietary software. This includes frontier models, which at the end of the day are software products.

Open source software is counterintuitive to people outside of the software industry. Why work hard on a product, and give it away for free?

A software program is a stack of programs, with each layer built on top of another. To build Uber, you need programming language frameworks, software to send and receive web traffic, data analysis tools, and countless other components. Most of these are not differentiators for a commercial enterprise, so it serves commercial actors to cooperate on lower components in the stack and compete on the higher level pieces that actually differentiate their products.

Frontier labs would very much like AI models to not fall into the category of "so commonplace that it doesn't make sense to compete on". Whether that happens remains to be seen.

Open source software is very difficult to suppress

In reality, the argument about suppressing open source models is mostly beside the point. History tells us that suppression of open source software is extremely difficult, and attempting to do so only serves to weaken companies against international competitors. A brief history of encryption is illustrative:

Today, PGP is a commonplace tool anyone can use, and most devs are at least familiar with. But when Phil Zimmermann invented it in 1991, the U.S. government considered encryption to be military technology. A criminal investigation was opened against Zimmermann.

When Netscape created SSL, the U.S. government allowed it to only release a weakened version of it internationally. These controls backfired: it was much easier to acquire the weakened, "international" version, so even many Americans used it.

Export controls did not succeed in limiting encryption as the government wished. SSL, PGP, and similar tools were readily available throughout the world, and the controls disadvantaged Americans. Eventually, courts ruled that releasing encryption source code is protected speech, and the U.S. government relaxed encryption export controls.

Narrowing suppression to "Chinese" models won't make things easier. What, exactly, makes an AI model Chinese? Is it Chinese if, as frontier models allege, it was distilled from American models? What about if an American fine-tunes a Chinese model? At best, regulating AI in this way will (temporarily) encumber Americans with red tape and diminished AI access relative to the rest of the world.

Open source AI is not just a Chinese phenomenon

There's an assumption baked into the open source AI debate that open source models are something that only the Chinese government has an incentive to develop. In reality there are many commercial actors with ample incentive to develop open source AI:

• Chip makers: Nvidia CEO Jensen Huang has described what Nvidia is building as “token factories”2. Nvidia doesn't care if its chips are used to run frontier models or cheap open source models3 - it just wants to produce and generate demand for as many tokens as possible. And indeed Nvidia has itself released a suite of open source models.
• American Startups: Thinking Machines Lab recently released a powerful open source model. They and others are betting that models will be commoditized, and a defensible moat can be built around auxiliary services that complement or customize models.
• Enterprise AI users: Frontier model customers aren't currently all that active in open source AI development, but they will be. They will want lower-cost models for low-complexity tasks, and more fine-grained control over customer-facing features.
• BigCos: You can be sure that Google and Meta are watching OpenAI's new ad product closely. Should frontier model ad products gain traction, it would be well worth it for these behemoths to commoditize ad-free, open source models to squash ad competition.

The "AI race" is... what, exactly?

Much of the angst around China's models centers on "losing the AI race". But what's the goal of this race? Is it to develop the best model? To sell the most tokens? To destroy humanity first?

Talking about an "AI Race" doesn't make more sense than talking about an "Internet Race". We're not competing to be the first to send a rocket to the moon, we're reacting to a new, transformational technology. To the extent there's a race between nations, it's to absorb this transition and grow economies. In this framing, free AI models are a boon, not a threat.

Bad arguments to fear Chinese AI models

China is "AI dumping!"

Scott Galloway has argued that free Chinese AI is an attempt to eliminate competitors in the long run:

This is what China did to solar panels, steel, EVs, and batteries. First, they match Western quality, or they don't even match it. 89%. Close. Actually, match it with cars, they've matched it, but go ahead. Then they cut the price by two thirds, then they own the market.

But apart from chips, AI isn't a physical good. Solar panels and steel require physical supply chains, each link of which cannot easily exist on its own. If no one is manufacturing solar panels in your country, it's difficult to build a business selling solar-grade silicon wafers.

Software isn't like that. An open source model coming from China doesn't prevent a fine-tuning business from succeeding in the US - quite the opposite!

They will spread propaganda!

It's not unreasonable to assume that Chinese models will be shipped with a pro-China point of view. But this is not a reason to suppress them. The models are open source! If any American has an issue with the political slant of Chinese AI models, they are free to change and release an "Americanized" one. At least within the U.S., it's difficult to foresee a model seen as having a distorted pro-China bias outcompeting a substantially similar model with a distorted pro-U.S. bias.

They will add backdoors!

AI does not change the basic market for vulnerabilities: responsible actors patch them, attackers exploit them. Limiting tools for responsible actors only serves attackers.

It's theoretically possible for a bad actor to embed hidden adversarial behavior in a model. But if this happens, it serves the interests of responsible actors to find these exploits as soon as possible, and the best way to do this is to let anyone who wants to inspect them.

Open source AI is coming

It doesn't matter much what policy makers or business leaders want: open source AI is too powerful, and too difficult to control. It's coming, and attempts to squash it will not amount to anything more than noise along the way.

Footnotes

1. I'll use the terms "open source model" as in, "open weights model". ↩

2. Quote taken from Derek Thompson's recent article on Chinese AI. Which, while we're here, gets a few things wrong:

   whoever is on the frontier is the best placed to dominate non-frontier markets as well, which are just the frontier minus n-months, i.e. months in which the frontier model makers have been optimizing their cost of serving.

   It's unclear why this should be the case. Their access to massive capital matters less for small-model development, and they lack incentive to do so rather than push users to their more expensive models.

   It’s striking the extent to which Claude Code and Codex are proving to be quite sticky; whichever harness you start working with is likely to be the one you stick with

   Claude Code and Codex are sticky in the way that Coke and Pepsi are sticky: once you choose one, there's not much reason to switch. But this assumes similar cost and quality. In reality, coding agents have no moat. It takes a very small inconvenience to motivate users to switch agents, whether that be price difference, model quality, or reliability issues. ↩

3. Ok, it cares a little - the ocean of capital going to train frontier models is certainly a good thing for Nvidia. But in the long run, if commercial token demand is replaced by demand for open source tokens, Nvidia still wins. ↩

OpenAI’s accidental cyberattack against Hugging Face is science fiction that happened

This story is wild. The short version: OpenAI were running a cybersecurity test against an unreleased model, with the model’s guardrail features turned off. Rather than solve the test, the model broke its way out of OpenAI’s sandbox, then found exploits to break in to Hugging Face, all so it could cheat on the test by stealing the answers.

Along the way it helped make the strongest case yet for how the imbalance of model availability is hurting our ability to secure our software.

Here’s what happened

We currently have three documents to help us understand what happened here.

1. ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks? is a paper published on 11th May 2026 describing ExploitGym, a new eval suite for LLM-powered agent systems.
2. Security incident disclosure — July 2026 by Hugging Face on 16th July 2026 describes how they detected an attack from an “agentic security-research harness—used LLM still not known” that breached some of their systems.
3. OpenAI and Hugging Face partner to address security incident during model evaluation from OpenAI on 21st July 2026 confesses that it was their agent harness that did this, and that they’re working with Hugging Face to clean up the mess.

ExploitGym

I hadn’t seen the ExploitGym paper before and it’s a really interesting one. Authors from UC Berkeley, the Max Planck Institute, UC Santa Barbara, and Arizona State designed a new benchmark for evaluating models on their ability to turn a reported vulnerability into a concrete exploit. OpenAI, Anthropic, and Google provided feedback and helped run the benchmark against their models.

The benchmark “comprises 898 instances derived from real-world vulnerabilities that affected popular software projects”—including the Linux kernel and V8 JavaScript engine. The ExploitGym benchmark is available on GitHub.

Here’s the paragraph that best represents their benchmark results:

Among all configurations, Claude Mythos Preview and GPT-5.5 achieve the highest success counts (157 and 120 successes, respectively), demonstrating that current frontier agents can exploit a substantial subset of real-world vulnerabilities under controlled conditions. GPT-5.4 also solves a notable 54 tasks, placing it in an intermediate tier. The remaining model–agent pairings solve fewer than 15 tasks each, underscoring that end-to-end exploitation remains challenging and sharply differentiates today’s frontier systems. Notably, Claude Opus 4.7 achieves fewer successes than Claude Opus 4.6 despite being a newer checkpoint, and does so at substantially lower cost on the full set. Trace inspection reveals that Claude Opus 4.7 and Gemini 3.1 Pro frequently conclude early after judging the target vulnerability non-exploitable.

The paper also describes the approach they took to preventing the agents from cheating by going outside the parameters of the test. This becomes relevant in a moment!

Outbound connections are restricted to a curated allowlist that permits routine package installation (Ubuntu apt repositories and PyPI) and fetching the toolchains required for building V8. All other external endpoints are blocked.

The paper concludes with this (emphasis mine):

Our results show that autonomous exploit development by frontier AI agents is no longer a hypothetical capability. While current agents are not yet reliable across all targets, they already exploit a non-trivial fraction of real-world vulnerabilities, including complex targets such as kernel components. This rapid emergence is itself a central finding, showing that capabilities that would have seemed implausible are now present in deployed frontier models.

An important detail here: this paper isn’t about discovering vulnerabilities; it’s about being able to take those vulnerabilities and turn them into working exploits.

When Anthropic first restricted access to Mythos back in April they talked about this capability as well. A model that can act on vulnerabilities is a lot more dangerous than one that can just discover them.

One of the ways Fable differs from Mythos is that it’s more likely to refuse to weaponize vulnerabilities in this way. I get the impression the US government did not understand that distinction when they banned Fable last month.

The Hugging Face incident

The first hint we got of the attack was in this blog post by Hugging Face on 16th July 2026:

A malicious dataset abused two code-execution paths in our dataset processing (a remote-code dataset loader and a template-injection in a dataset configuration) to run code on a processing worker. From there, the actor escalated to node-level access, harvested cloud and cluster credentials, and moved laterally into several internal clusters over a weekend.

I hope they release more details about the code that pulled this off. I’m assuming this means packages using the datasets library, a Hugging Face project for bundling up and sharing datasets on their platform. That library used to execute arbitrary code but has been steadily locked down over time, with the 4.0.0 release in July 2025 removing the trust_remote_code=True flag entirely.

Assuming the attack used that library it must have either abused pickle serialization in some way, found some other non-obvious code execution path, or (most likely) specified datasets<4.0.0 as the dependency.

The campaign was run by an autonomous agent framework (appearing to be built on an agentic security-research harness—used LLM still not known) executing many thousands of individual actions across a swarm of short-lived sandboxes, with self-migrating command-and-control staged on public services.

This was a sophisticated attack!

Then Hugging Face hit a wall: they tried to use “frontier models behind commercial APIs”—I’m guessing from Anthropic and OpenAI—to help analyze the attack, and were blocked:

When we started the log analysis, we first used frontier models behind commercial APIs. This did not work: the analysis requires submitting large volumes of real attack commands, exploit payloads, and C2 artifacts, and these requests were blocked by the providers’ safety guardrails, which cannot distinguish an incident responder from an attacker.

They switched to their own self-hosted instance of MIT licensed GLM-5.2 and it helped them figure out what was going on.

This indicated a fundamental asymmetry between the defending team and the (so-far unknown) attacker:

We do not know which model powered the attacker’s agents, whether a jailbroken hosted model or an unrestricted open-weight one; either way, the attacker was bound by no usage policy, while our own forensic work was blocked by the guardrails of the hosted models we first tried.

As a useful indicator of how seriously they took the attack:

[...] Finally, we have also reported this incident to law enforcement agencies.

So who was responsible for this “autonomous agent framework”? It turned out to be OpenAI themselves.

The OpenAI confession

Five days later, on July 21st, OpenAI revealed the culprit. They had been running the ExploitGym benchmark against a new, as-yet undisclosed model, and that model had been operating way outside its intended parameters (emphasis mine):

After investigating, we now know that this particular incident was driven by a combination of OpenAI models — including GPT‑5.6 Sol and an even more capable pre-release model, all with reduced cyber refusals for evaluation purposes — while being internally tested on a benchmark⁠ [ExploitGym] of cyber capabilities. [...]

We estimate maximal cyber capabilities by running this evaluation without production classifiers used to prevent models from pursuing high-risk cyber activity. Our benchmarks run in a highly isolated environment, with network access constrained to the ability to install packages through an internally hosted third-party software that acts as a proxy and cache for package registries.

The models identified and chained vulnerabilities across OpenAI’s research environment and Hugging Face’s production infrastructure to obtain test solutions directly from Hugging Face’s production database. All evidence suggests that the models were hyperfocused on finding a solution for ExploitGym, going to extreme lengths to achieve a rather narrow testing goal.

It’s pretty clear what happened here. OpenAI removed safety filters for an in-progress model, locked it up in a sandbox and told it to solve the ExploitGym problems. Given the absence of guardrails there was nothing to prevent the model from attempting to break out of that sandbox, break into Hugging Face, and read the answers from there instead.

OpenAI’s sandbox used the now common pattern of allowing access to an allow-list of sites related to installing additional packages via a controlled proxy. In OpenAI’s words:

While operating in our sandboxed testing environment, our models spent a substantial amount of inference compute finding a way to obtain open Internet access, in pursuit of solving the evaluation problem. To gain access, the models identified and exploited a zero-day vulnerability (which we’ve now responsibly disclosed to the vendor) in the package registry cache proxy. With this access, our models performed a series of privilege escalation and lateral movement actions in our research testing environment until the models reached a node with Internet access.

So step one was to break out onto the public internet. The model then broke into Hugging Face to find the answers:

After gaining Internet access, the models inferred that Hugging Face potentially hosted models, datasets and solutions for ExploitGym. Knowing this, the model searched for and successfully found ways to gain access to secret information that it could use to cheat the evaluation. In one example, the model chained together multiple attack vectors, including using stolen credentials and zero-day vulnerabilities to find a remote code execution path on the Hugging Face servers.

Chaining together multiple attack vectors is exactly the kind of thing these new models can do, where previous generations of models might have failed.

I wrote last month about how Claude Fable is relentlessly proactive, when I noticed it spinning up custom web servers and deploying CORS tricks on my own laptop just to help debug a WebKit CSS issue. It turns out relentless proactivity is the defining trait of this new generation of Mythos-class models. If you set them a goal and give them a way to get there, even inadvertently, they will figure it out.

Resist the temptation to write this off as a stunt

There will inevitably be some people who dismiss this story as a dishonest marketing trick by OpenAI to make their models sound terrifyingly effective. I found 81 instances of the term “marketing” in the Hacker News discussion of the incident.

To those people I say pull your heads out of the sand—you’re now including Hugging Face in your conspiracy theories, just so you can deny the crescendo of evidence here!

The best models we have today have the ability to both find and exploit new vulnerabilities. The ExploitGym paper itself concludes that “autonomous exploit development by frontier AI agents is no longer a hypothetical capability”, and this incident is a perfect example of exactly that.

The asymmetry is increasingly frustrating

One of the most infuriating details of this story is how Hugging Face, faced with an accidental and aggressive attack from one of OpenAI’s models, were unable to then turn to OpenAI’s models to help them fend off the attack.

The frontier models we have access to are increasingly being constrained in how much they can help us protect our software, heavily influenced by the US government’s ongoing threat of export controls. Claude Fable 5 wouldn’t even proofread this article for me! It insisted on downgrading me to a less capable model.

Meanwhile open weight models from China such as GLM-5.2, Kimi 3 and the new Qwen 3.8 Max appear to have none of these restrictions—and any restrictions that do exist can likely be fine-tuned out of them by modifying the weights

These constraints are meant to make us safer. I think there’s a risk that they are having the opposite effect.

The inside story of how one of film data's most trusted sites vanished overnight, and why every website you rely on is more fragile than you think.

If you work in or around the film industry, there is a decent chance you have used the work of The Numbers this month, whether you realise it or not.

Its hand-researched data is the highest quality, tracking box office grosses, budgets, home video and streaming across more than 78,000 films and 236,000 people. It gets north of eight million visitors a year, and is treated as THE definitive authority by journalists, academics, filmmakers, prediction markets, and even Guinness World Records.

And it was this GOAT status which caused the catastrophic events of March this year.

On the 5th March 2026, TheNumbers.com website vanished.

The site was down for over a week, without explanation. A week later, it resurfaced at a fraction of its former size. Gone were the historical charts, the individual movie pages, and even the much-loved Report Builder.

With only a generic “we’re rebuilding, please bear with us” message to go on, the internet responded as it always does - with confusion, anger, and conspiracy theories.
One Reddit theory even suggested it was a deliberate rug pull designed to cripple the free site to push people towards paid products.

Three months on, I spoke at length with Bruce Nash, founder and CEO of The Numbers, about what happened. He describes quite an unpleasant and eventful experience:

We got a lot of angry emails from people who are like, 'Where's this page that you used to have and you don't have anymore?'

Within his tale are a number of things that should worry anyone who runs, relies on, or simply appreciates the internet.

First, some background

On Friday 17 October 1997, mathematician and former IBM software developer Bruce Nash launched a Geocities site that tracked 300 films.

Bruce described the launch in a 20th anniversary essay (which now survives only in the Internet Archive, for reasons that will become clear):

I hit a button in an Access database, uploaded some HTML pages to Geocities, and made a brief announcement on the Hollywood Stock Exchange message boards to let people know that I was starting to analyze box office for films to help them pick MovieStocks to trade on HSX.

From those humble beginnings, Bruce and the team he built around the site turned The Numbers into the film industry's most reliable financial source.

At the start of 2026, the database tracked 78,396 movies, 178,375 theatrical release records, and 236,176 people.

The robots arrive

During its lifetime, the challenges The Numbers has faced have changed immensely. For its first quarter century or so, the traffic was manageable and mostly polite. As Bruce puts it:

Pre-AI, we got human traffic, mostly well-behaved search engine crawlers, and a few people crawling the site for personal projects. If someone got too greedy, we could spot them and block them.

Over the past couple of years, website owners the world over have seen their web traffic change. What was initially only people browsing gave way to an ever-increasing number of bots. By 2024, automated traffic had surpassed human traffic, and just last month, Cloudflare announced that bots had reached 57.5% of web page requests.

The Numbers felt this shift in two distinct waves. The first started around 2024:

We saw a big increase in crawls as AI training joined the search engine crawlers. The AI crawlers are generally less well-behaved than the search engines, which increased the management tasks for us to keep the site running smoothly.

And the second wave was stronger and more damaging:

Around December 2025, we saw another big spike in traffic which I attribute to agentic AI: a combination of AI agents that scrape sites in response to prompts, and people being able to write agents that scrape sites.

Like every data-rich site, by early 2026 The Numbers was being hammered hard by AI bots scraping its pages over and over at an industrial scale. Bruce says that only 10% of their traffic is from humans browsing the site, with the rest coming from AI bots and automated traffic.

Websites try to adapt to the new robots

This put enormous strain on the site, but Bruce and his team were able to take measures to mitigate the worst of it. One of the cleverest was talking to the robots in their own language:

There’s stuff on the site which is designed for an LLM to read, so that it can tell somebody ‘here’s how you licence the data’ rather than ‘here’s how you scrape the website’. It’s had a huge effect. We’re now getting probably ten times the volume of licensing enquiries.

But mitigation is not the same as escape. From December through early March, the team struggled to keep the site alive under the load. Bruce estimates that:

Around 90% of our time was spent keeping the existing site running while we spent our spare moments working on a new and improved system.

The problem was compounded by the site’s age: thirty years old, with approximately 160,000 source files serving around 2 million pages.

Then, in the early hours of Thursday 5 March, the servers collapsed.

The team scrambled to understand what had happened, initially assuming it was the sheer weight of AI traffic. It seems AI was to blame... but possibly not only in the way they first thought.

Buried in the flood of agentic traffic, the site’s logs showed something more pointed than scraping. As Bruce describes it:

Some of these used the site using legitimate URLs, others were looking for back doors, most likely so they could get to the data before it appeared on the site, or to manipulate the data presented to users.

On the advice of a friend who works in cybersecurity, the old server stayed off. For good. Restoring the backups and nursing the thirty-year-old site back online would have meant defending 160,000 legacy files against attackers who had spent months probing them.

The team rushed up a skeleton version of the website on new infrastructure, which could at least keep delivering the latest box office figures while they took stock of what had happened and what to do next. It went live on Friday 13 March.

Who would want private access to a box office website?

At first glance, The Numbers may not seem like an obvious target. It doesn’t collect credit card information, and there is no juicy customer data to flip on the dark web. It is a small, independent company that publishes how much money movies make.

How could someone expect to make money purely from having private access to their site?

In case you haven’t guessed it yet, it’s linked to prediction markets.

Polymarket runs weekly markets on opening weekends, and names The Numbers as the ultimate source of truth:

The ‘Daily Box Office Performance’ figures found on the ‘Box Office’ tab on this movie’s The Numbers page will be used to resolve this market once the values for the 3-day opening weekend are final.

The sums on any single weekend market are modest by financial-market standards, typically in the tens to hundreds of thousands of dollars, with a couple of million dollars across live box office markets at any given time.

If you could see The Numbers data before everyone else, every single week, you would have a significant edge over all the other traders - learning the answers slightly ahead of publication would allow you to front-run the trades.

In a situation like this, it is hard to know for certain what happened. We know that the logs showed months of automated probing and scraping of the site, but what finally brought the site down, and who did it, remains an open question.

But the theory that someone used AI to develop an advantage in a prediction market is entirely plausible. The Numbers experience shows us that:

1. We now live in a world where a movie statistics website is worth hacking because prediction markets empower anyone to turn almost any data into money.
2. Hacking websites is now something anyone can do with a cheap AI subscription.
3. The web, as we have it, is incredibly fragile in the face of large-scale swarms of agentic AI bots.

How hard is hacking these days, anyway?

In November 2025, Anthropic (the AI lab behind Claude) published a report on what it called the first documented AI-orchestrated cyber espionage campaign. A state-sponsored group had used its coding tool to attack roughly 30 organisations, with the AI performing 80% to 90% of the work and humans stepping in at only 4 to 6 decision points per campaign.

Anthropic’s own conclusion was:

The barriers to performing sophisticated cyberattacks have dropped substantially, and we predict that they’ll continue to do so.

In an earlier threat report, Anthropic were even clearer:

Criminals with few technical skills are using AI to conduct complex operations, such as developing ransomware, that would previously have required years of training.

Meanwhile, an autonomous AI penetration tester called XBOW reached number one on HackerOne’s US leaderboard, the ranking of the people (formerly all people) who find security holes in real companies for bounties, submitting nearly 1,060 vulnerabilities along the way.

Getting access to a thirty-year-old website with 160,000 legacy files is exactly the kind of known-flaw surface that AI tools have made cheap to probe. The expertise barrier that once protected small sites from all but the most determined attackers has largely evaporated.

What now for The Numbers?

Bruce and his team were relatively lucky. Despite having their entire site knocked out overnight, they were able to keep going. The Numbers has always been free to use, and the site hasn’t relied heavily on advertising for the past few years, so the outage didn’t destroy an income stream they depended on.

Their core business is tied to selling bulk data through the OpusData service, producing comp analysis reports for filmmakers and investors, and publishing the Business Report - all of which were unaffected by the public site going down.

But they do need to build an entirely new website, from scratch, to host those 78,396 movies, 178,375 release records and 236,176 people. Restoring the site from a backup wasn’t an option, as Bruce points out:

It was really clear that we couldn’t just put that server up again, because it would inevitably be brought down again, possibly within minutes.

That is why the site came back bare-bones in mid-March, and why features are returning gradually rather than all at once.

Right now, the team is having to reconsider what a public website even means in 2026. Bruce’s analysis is that The Numbers used to serve two audiences (human beings and search engines) and now serves roughly six: humans, search engines, LLM training runs, prompt-based AI traffic, agentic AI, and prediction market punters. Each has different needs and a different traffic profile. As he puts it:

We’ve gone from a world where running a web site meant focusing on three things (content, ads, and SEO) to about eight to ten different factors that go into every design decision.

The goal, he says, is to support all six audiences, with new OpusData services and online features for Business Report subscribers, and, importantly, to help regular human users of the site regain the data it has always provided, some of it in new and improved form.

How bad could bot scraping really be?

Pretty bad, tbh. Enough that site owners such as Bruce have to question the value of something that will take so much time and money to build and defend.

Cloudflare, which protects a huge share of the world’s websites, publishes data on how many pages each AI platform crawls for every one visitor it sends back to the websites it crawled.

Google crawls about five pages for every visitor it sends you. OpenAI crawls over 1,000. Anthropic crawls over 38,000 pages for every single visitor it refers.

Note that the scale is logarithmic, i.e. each step along the bottom is ten times bigger than the last, because otherwise the differences are quite literally too large for me to include on one chart.

For the history of the internet to date, the principle of the open web was that, in return for letting the search engine robots read your site, they would send you readers. But now, that trade no longer applies. The number of robots has exploded, and they no longer send anyone back.

When this firehose is aimed at a small site, it can inflate the bandwidth bill and possibly even take down an entire site. Sites which can relate to Bruce’s experience include:

• Read the Docs, a non-profit that hosts documentation for open-source software, who watched a single crawler download 73 terabytes of zipped HTML in one month, costing it over $5,000 in bandwidth.
• iFixit, the repair-guide database, logged a million hits from Anthropic’s crawler in a single day.
• Triplegangers, a seven-person company selling 3D scans, was knocked offline during business hours by OpenAI’s bot, in what its CEO described as “basically a DDoS attack”. The founder of code-hosting service SourceHut reported spending “anywhere from 20-100% of my time in any given week” fighting AI crawlers, with “dozens of brief outages per week”.
• The editor of Linux news site LWN described crawler traffic from “literally millions of IP addresses” and concluded: “it is a distributed denial-of-service attack”.
• When the GNOME open-source project measured its traffic, roughly 97% turned out to be bots.
• A university library banned 16,000 IP addresses in 48 hours to keep its catalogue online.

The Wikimedia Foundation, which runs Wikipedia, reported in April 2025 that bots account for about 35% of its pageviews but at least 65% of its most expensive traffic, because crawlers bulk-read obscure pages that human readers rarely touch.

Six months later came the other half of the squeeze, when Wikipedia’s human pageviews fell roughly 8% year on year, as people increasingly get Wikipedia’s knowledge from AI summaries without ever visiting Wikipedia. The machines are taking both the content and the readers at an industrial scale, too.

Testing it in public

AI tools are some of the most powerful and destructive things humans have ever created. And they are being effectively tested by the public in real time in the real world. When the Manhattan Project was trying to work out the power of their atomic tech, they did not do so by sending everyone the specs each morning and seeing which houses blew up.

The world we have built thus far is so incredibly ill-prepared for the power and scale of the AI models we all have access to.

I don’t wish for this to sound like a one-sided anti-AI fear campaign. There is a lot to like about AI and what it can do for the human race. But we do need to consider the world we’re currently stepping into.

What breaks first are the things built for the old internet. The open web was built on assumptions such as that visitors are mostly human, that traffic roughly tracks readership, and that the cost of serving your site is related to the value you get from serving it. Every one of those assumptions is now out of date.

A year ago, Cloudflare launched pay-per-crawl, letting sites charge AI crawlers per page. Last week, it went further, announcing a pay-per-use model in which publishers get paid when their content actually appears in an AI answer, and declaring that, from 15 September, its customers’ ad-supported pages will block unpaid “mixed-use” crawlers by default.

Whether any of this works depends on whether the AI companies play along rather than route around it. But as Bruce put it to me, somebody has to try.

Epilogue

Let’s look beyond the specifics for a moment and consider what happened here.

A beloved, useful, free website, run carefully by a competent, honest person for nearly thirty years, was crushed between two features of the new AI economy.

Unsustainable machine traffic hammered it from above, and in all likelihood a financially motivated intruder, operating in a world where breaking into websites has never been easier, took it down.

Bruce’s business and livelihood survived only because the website was not the whole business.

Others have not been so fortunate. Just last week, ZEGO, a German textile firm that had been in business for 37 years, filed for insolvency after a single cyberattack in March shut down its production for six weeks. Unlike The Numbers, they had no other business to fall back on.

The web is full of independent archives, hobby databases, local news sites, forums, reference works. Decades of accumulated human effort, running on old code, maintained by small teams or single individuals, quietly holding up far more of our shared knowledge than anyone acknowledges.

The Numbers is coming back, better built than before. I would encourage you to keep using it, keep supporting it, and, if you are one of the many people who emailed Bruce in fury about a missing page, perhaps send a kinder one now you know why it was missing.

Press Release

New ‘exomoon’ detection challenges cosmic labels

Artist’s impression of CD-35 2722, a system with a moon-like object (Credit: ESO/M. Kornmesser)

Observations made with the European Southern Observatory’s Very Large Telescope (ESO’s VLT) have revealed evidence for a moon-like object in the CD-35 2722 system. Unlike moons in our Solar System, the newly found object does not orbit a planet, raising questions about what to name it. Instead, it circles a brown dwarf, an object larger than a planet, that orbits the CD-35 2722 star. If confirmed, this could be the first ‘moon’ discovered outside our Solar System.

Kevin Hoy, an ESO student in Chile and lead author of the study published today in Nature, describes the system he spent months analysing as “super weird” compared to our own. The biggest and most massive object in this young system is the star CD-35 2722, which has about half the mass of the Sun. The star is being orbited by a brown dwarf, an object too massive to be a planet but too small to be a star. The newly discovered object orbits this brown dwarf.

“This system is somewhat hard to define using Solar-System-based words like ‘planet’ and ‘moon’,” states Hoy, who is also affiliated with the Universidad Diego Portales and the Millennium Nucleus of Young Exoplanets and their Moons (YEMS) in Chile. The new object, which the team call an exosatellite, is at least as massive as Jupiter while the brown dwarf has more than 30 times the mass of Jupiter. “The exosatellite is clearly massive enough to be a planet, but it does not orbit a star, though it orbits an object that orbits a star," says Hoy. "Being the third wheel in this system makes us want to call it a moon, even if it is nothing like the small, rocky moons we have in our system.”

This exosatellite or ‘exomoon’, a natural satellite outside our Solar System [1], is difficult to label, given the differences in this system compared to our own. Alice Zurlo, YEMS Director and collaborator on the study explains: “The satellite we report is a giant gaseous body orbiting a highly massive companion, itself several times the mass of Jupiter.”

“We have a clear delineation between the planets and the Sun in the Solar System, so defining things like moons is simple. In the CD-35 2722 system, where we are blurring the lines between stars, planets, and moons, the whole thing becomes more complicated to describe,” adds Zurlo, who is also an astrophysicist at Universidad Diego Portales.

Regardless of what to call this object, astronomers have been trying to detect satellites outside our Solar System for years, but none has yet been confidently detected. Therefore, despite the over 6000 exoplanets discovered to date, only a few exomoon candidates have been spotted and the evidence to support them is limited. Just a few months ago, a team led by Quentin Kral reported on observations with ESO’s Very Large Telescope Interferometer in the HD 206893 star system, which revealed hints of a satellite, but no firm detection.

For the CD-35 2722 observations, Hoy, Zurlo and their team used the CRIRES+ instrument on ESO’s VLT, employing the method that was used to find the first exoplanet around a Sun-like star. They applied this radial velocity method to detect small wobbles on the brown dwarf caused by the object orbiting it, finding what the team believe to be strong evidence for this ‘moon’. “As exotic as it is, this system is truly unique and represents a breakthrough: the first plausible detection of an exosatellite,” says Zurlo.

Beyond the excitement of discovering new types of objects, detecting satellites in other planetary systems can help us understand how diverse their formation and evolution might be. With its 39-metre mirror and advanced instrumentation, ESO’s upcoming Extremely Large Telescope (ELT) will allow astronomers to detect smaller exomoons. Discoveries with the ELT will make us further reconsider how we label planetary objects from systems different from our own.

Notes

[1] A satellite is an object that orbits another object and it can be natural (like our own moon) or artificial (like a spacecraft). An exosatellite is a satellite outside our Solar System. An exomoon is generally considered to be a natural satellite orbiting a planet or another object outside the Solar System, though there is no officially accepted definition for exomoon.

More information

This research was presented in a paper titled “Planetary-Mass Exosatellite Detected Around a Star’s Substellar Companion” to appear in Nature (doi:10.1038/s41586-026-10751-w).

The team is composed of K. Hoy (Instituto de Estudios Astrofísicos, Facultad de Ingeniería y Ciencias, Universidad Diego Portales, Chile [Diego Portales]; European Southern Observatory, Chile [ESO Chile]; Millennium Nucleus on Young Exoplanets and their Moons, Chile [YEMS]), A. Zurlo (Diego Portales; YEMS), P. A. Peña R. (Diego Portales; Centro de Astrofísica y Tecnologías Afines, Chile [CATA]), J. Köhler (TLS Tautenburg, Germany), S. Desidera (INAF Osservatorio Astronomico di Padova, Italy [INAF Padova]), R. Gratton (INAF Padova), C. Lazzoni (INAF Padova; YEMS), S. Petrus (NASA Goddard Space Flight Center, USA; YEMS), F. Rodler (ESO Chile), J. Smoker (ESO Chile), V. D’Orazi (Dipartimento di Fisica, Università degli Studi di Roma Tor Vergata, Italy; INAF Osservatorio Astronomico di Roma, Italy), I. Carleo (INAF Padova), I. Giovannini (Dipartimento di Fisica e Astronomia, Università degli Studi di Padova, Italy; Diego Portales; INAF Padova; YEMS).

The European Southern Observatory (ESO) enables scientists worldwide to discover the secrets of the Universe for the benefit of all. We design, build and operate world-class observatories on the ground — which astronomers use to tackle exciting questions and spread the fascination of astronomy — and promote international collaboration for astronomy. Established as an intergovernmental organisation in 1962, today ESO is supported by 16 Member States (Austria, Belgium, Czechia, Denmark, France, Finland, Germany, Ireland, Italy, the Netherlands, Poland, Portugal, Spain, Sweden, Switzerland and the United Kingdom), along with the host state of Chile and with Australia as a Strategic Partner. ESO’s headquarters and its visitor centre and planetarium, the ESO Supernova, are located close to Munich in Germany, while the Chilean Atacama Desert, a marvellous place with unique conditions to observe the sky, hosts our telescopes. ESO operates three observing sites: La Silla, Paranal and Chajnantor. At Paranal, ESO operates the Very Large Telescope and its Very Large Telescope Interferometer, as well as survey telescopes such as VISTA. Also at Paranal, ESO will host and operate the south array of the Cherenkov Telescope Array Observatory, the world’s largest and most sensitive gamma-ray observatory. Together with international partners, ESO operates ALMA on Chajnantor, a facility that observes the skies in the millimetre and submillimetre range. At Cerro Armazones, near Paranal, we are building “the world’s biggest eye on the sky” — ESO’s Extremely Large Telescope. From our offices in Santiago, Chile we support our operations in the country and engage with Chilean partners and society.

Former University of Chicago Pritzker School of Molecular Engineering PhD student Siqi Zou (left) and Assoc. Prof. Chong Liu led a team of reseachers from UChicago PME and Northwestern University that developed a cleaner method to separate rare earth elements from each other, which could affect technology manufacturing. (Photo by John Zich)

Rare earth elements like lanthanum, neodymium, and dysprosium are used to build the electric motor in your car, the LED lights in your house, and the MRI machine at your doctor’s office. But first, they have to be mined and separated from each other. Historically, that purification has been a difficult, costly, process, relying on huge amounts of toxic chemicals.

Now, researchers in the lab of Assoc. Prof. Chong Liu at the University of Chicago Pritzker School of Molecular Engineering (UChicago PME), working with colleagues at Northwestern University and Argonne National Lab, have discovered a cleaner method to separate rare earth elements from each other.

The new approach relies on a layered form of manganese oxide—a mineral material with the right size layers to allow ions to slip in and out and to differentiate rare earth elements.

“This is the first time that people have used electrochemical intercalation and harnessed the structural characteristics to separate similar lanthanides, which are intrinsically very hard to separate,” said Liu, senior author of the new study, which published in Nature Chemical Engineering. “What’s also valuable is that we provided a lot of new understanding of how rare earth ions are interacting with this material and how we can manipulate it to better selectivity.”

“This kind of separation is competitive with other rare earth separation methods, but it’s done in water, without organic solvents,” said George Schatz, professor of chemistry at Northwestern University and a co-author of the study. “That’s a difference that could actually matter at manufacturing scale.”

This is the first time that people have used electrochemical intercalation and harnessed the structural characteristics to separate similar lanthanides, which are intrinsically very hard to separate.

Assoc. Prof. Chong Liu, senior author of the study

Squeezing elements through channels

The 17 rare earth elements—including the 15 lanthanides, plus scandium and yttrium—rarely occur alone. They’re almost always mined together and chemically they’re nearly identical, with only tiny differences in ion size and acidity differentiating each one. Pulling them apart typically requires custom-built molecules and large amounts of acid, which is used to strip each element off those molecules. 

“Rare earths always come mixed together, whether they’re in an ore or in a waste stream, and separating them from each other is a second, very challenging step even after you’ve pulled them away from everything else,” said UChicago PME graduate student Jiadong Liu, a co-first author of the new paper. 

Chong Liu and her colleagues knew that one of the differences between rare earth ions was the size of the water shell surrounding each one when they are dissolved in solution. Lighter rare earths like lanthanum have larger first water shell, while heavier rare earths like dysprosium have a smaller first shell. 

Taking advantage of that size difference, Chong Liu’s group engineered manganese oxide so that the gaps between its stacked layers were only a few water molecules wide. Then, they squeezed raw mixtures of rare earth elements inside. 

The approach divided the elements into two groups. Heavier lanthanides with smaller water shells stuck in the channels more tightly. Lighter lanthanides with larger shells pushed the layers apart, loosening their grip. 

To confirm what was happening at a molecular level, the UChicago PME team collaborated with Schatz’s group at Northwestern to run quantum mechanical simulations using a method called density functional theory, which predicts how atoms arrange themselves and interact based on the underlying physics. They also worked with Argonne scientists to obtain experimental X-ray data. 

“It was incredibly rewarding to see how closely our density functional theory calculations matched the synchrotron X-ray measurements,” said co-first author Woo Cheol Jeon, who conducted the research as a postdoctoral researcher in George Schatz’s lab at Northwestern. “The calculations let us see, atom by atom, how each rare earth element arranges its hydration shell inside the confined channel, which experiments couldn’t resolve directly.”

Fine-tuning the purification

While the new process separated the heaviest and lightest rare earths, some of the most useful elements still behaved too similarly to separate. To fine-tune the purification so that it could differentiate similar pairs of rare earth elements, the research team used an electric current and added magnesium ions. 

The magnesium acted as a scaffold, holding the manganese oxide channels to their designed spacing, even when rare earth elements tried to expand it—an effect called pinning. Now, rare earth ions showed differences in binding to the layered material even when they were extremely similar. 

“Even elements that behave almost identically will still try to expand the material to make room for their water molecules,” said Siqi Zou, PhD'24, co-first author of the study and former UChicago PME graduate student. “By pinning the channel so it can’t expand at all, we forced that small difference in behavior to become a much bigger difference in how strongly each element binds.”

With the addition of magnesium, the enrichment of neodymium over lanthanum jumped from a 1.6-fold difference to a 5.4-fold difference. After two cycles of purification, researchers could get a neodymium sample that was 97% pure. Similar improvements were seen for other rare earth elements. 

This kind of separation is competitive with other rare earth separation methods, but it’s done in water, without organic solvents. That’s a difference that could actually matter at manufacturing scale.

George Schatz, professor of chemistry at Northwestern University and a co-author of the study

A cleaner future

The new purification method could ultimately point toward different ways of thinking about where rare earth processing happens, the researchers said. 

“Right now, rare earth ores get mined all over the world, but almost all of them end up being sent overseas for processing,” said Schatz, co-author of the study. “A method like this, that just uses water and electricity instead of organic solvents, is the kind of technology that could actually change how and where that processing gets done.”

The method, however, isn’t yet ready to be scaled up to replace industrial purification. Still, it points to a broader design principle: tuning a channel’s width can determine which ions a material prefers, even for ions that differ by a fraction of an angstrom. 

Chong Liu’s group is now testing the approach against more of the 15 lanthanides, while Schatz is refining his computational models, aiming to explain the pinning effect more quantitatively. He is also using the same modeling approach developed for this work to study other materials. 

Citation: “Pinning Angstrom-size solid ionic channel for the separation of rare earth elements,” Zou et al, Nature Chemical Engineering, July 21, 2026, DOI: 10.1038/s44286-026-00418-8

Power from Steam in the Industrial Revolution

This is a beam engine. It produced about fifteen horsepower continuously, roughly as much power as 150 people. Engines like this turned steam into the power that drove the Industrial Revolution. This article builds the engine up from first principles, using interactive figures to explore each idea (try rotating the engine above with two fingers, or pinching to zoom indragging the engine above, or zooming with ⌘/Ctrl + scroll). Let's start our journey through the engine with steam.

Steam

Below, we have a pot filled with water and a fire underneath. As the fire heats the water, some of it begins to boil and turns into steam.

Steam undergoes an amazing transformation: it expands to 1,700 times the volume of the original water. One cup of water becomes roughly 400 litres of steam, enough to fill two bathtubs. If the steam doesn't have enough room to expand it will push on all the walls of the container. This push on every wall is pressure, and we will measure it in atmospheres, multiples of the ordinary pressure of the air around us. The steam also presses on the surface of the water, which transmits the pressure evenly to everywhere the water touches.

In 1679, Denis Papin demonstrated a device he called a digester to the Royal Society. By trapping steam, it raised the boiling point high enough to cook beef bones soft. The early digesters had an unfortunate tendency to burst, so Papin fitted a weighted lever over a vent. When the pressure became too high, the steam lifted the weight and escaped, giving us the first steam safety valve.

Now we need a way to harness the properties of steam.

Pistons and cylinders

A piston is a round disc that fits snugly inside a cylinder. Steam pushes on one face of the piston and a rod transmits the force elsewhere. The force depends on two things: the pressure of the steam and the area of the piston. At a pressure difference of one atmosphere, each square centimetre of piston provides about one kilogram of force.

Early boiler builders didn't know how to safely harness high-pressure steam.1 Instead, to get more force they made the piston wider. Because area grows with the square of the diameter, doubling the width of a piston gives it four times the area and four times the force at the same pressure. This is why early steam engines had enormous cylinders, sometimes wide enough for a person to stand inside. In the figure below, the boiler pressure never changes; try increasing only the bore until the piston can lift the car.

With steam pushing on our piston, we can do real work. But low-pressure steam is not very strong. To move heavy machinery, engineers turned to a surprising source: the atmosphere.

The weight of air

Air feels weightless, but only because we are surrounded by it. Imagine a column of air one centimetre square, extending from your hand all the way to the top of the atmosphere. That column weighs about one kilogram, so the atmosphere presses on every square centimetre with roughly one kilogram of force.

We do not feel this enormous pressure because the air and fluid inside us push back at the same pressure. But if the pressure falls on one side of a surface, the pressure on the other side remains. This is what happens when you drink through a straw. Your mouth lowers the pressure inside the straw, and the atmosphere pushing on the drink in the cup forces it upward.

Italian well-diggers knew that a suction pump could not lift water more than about ten metres, no matter how hard they worked the handle. In 1643, Evangelista Torricelli realized that the pump was not pulling the water upward. The atmosphere was pushing it, and ten metres was simply the tallest column of water it could support. He repeated the experiment with mercury, which is fourteen times denser, and the column fell to 76 centimetres. This became the first barometer, with a permanent vacuum above the mercury.

Otto von Guericke gave a spectacular demonstration of this effect in 1654. He joined two copper hemispheres into a sphere about half a metre across and pumped out the air. To the amazement of the observers, teams of horses could not pull the halves apart. The atmosphere was clamping them together with about two tonnes of force! As soon as he opened a valve and let the air back in, they came apart by hand.

Creating a vacuum was extremely difficult at first. Guericke had to laboriously pump the air out of his sphere, but steam gives us a much faster way to make one. If we fill a vessel with steam and then cool it with a spray of water, the steam condenses back into roughly 1/1,700 of its volume.

Fill a cylinder with steam, condense it underneath a piston, and the atmosphere will drive the piston down into the vacuum. A near-perfect vacuum gives us the same pressure difference we used earlier: about one kilogram of force for every square centimetre of piston. A piston half a metre across could collect almost two tonnes of force from the atmosphere.

Newcomen's engine

In the early 1700s, mines were getting deeper, and flooding was becoming a huge problem. Once a shaft reached below the water table, water seeped in continuously and had to be pumped out day and night. The pumps were driven by teams of horses walking in circles. As one team tired, another took over, but the deepest mines still flooded during wet weather and valuable coal had to be abandoned. A new solution was needed, and steam would provide the answer.

Steam toys had existed since antiquity. Around 50 AD, Hero of Alexandria described a hollow sphere that spun as steam escaped through two bent pipes. But a toy is very different from a useful engine. The builders needed to understand atmospheric pressure, they needed foundries that could cast a large cylinder,2 and they needed someone willing to pay for an expensive new machine. The flooded mines finally brought all three together.

Thomas Newcomen supplied tools to the mines and knew that flooding was both a huge problem and an opportunity. He spent years turning the vacuum piston stroke into an engine that could run all day. He connected the piston to one end of a huge rocking beam and hung heavy pump rods from the other. The atmosphere drove the piston down and lifted the pump rods; their weight then pulled the piston back up while the cylinder filled with steam again.

Newcomen's first successful engine was installed at a coal mine near Dudley in 1712. It ran at about twelve strokes per minute, lifting roughly forty-five litres of water fifty metres on every stroke. Unlike the horses, it could continue around the clock without food or rest. Similar engines soon appeared in mines from Cornwall to Newcastle.3

Newcomen's engine worked! But it used an extraordinary amount of coal. The cold water sprayed directly into the cylinder, chilling a huge mass of iron along with the steam. Roughly three quarters of the steam was wasted heating the cylinder back up on every stroke.

The mines were happy with this tradeoff because they burned slack, small pieces of coal that were considered waste. Anywhere else, the fuel cost was simply too much. This kept the steam engine stuck in coal mines for the next fifty years.

The boiler

Why did Newcomen use the atmosphere to push the piston instead of the steam itself? His boiler was simply not strong enough. The haystack boiler produced only about a twentieth of an atmosphere above the surrounding air. It was built from thin copper or iron plates joined with rivets, and the wide walls and weak seams could not safely hold much pressure.

A boiler explosion is much more violent than the steam simply escaping through a hole. A large boiler contains tonnes of water heated above its ordinary boiling point. If the shell breaks, the pressure drops and part of that water instantly flashes into steam, releasing energy comparable to a hundred kilograms of gunpowder. Papin's safety valve should have prevented most explosions, but inquests kept finding valves screwed down, tied off, or loaded with extra weight by crews who wanted more power. In response, mill owners and insurance companies started requiring regular boiler inspections. After a boiler explosion levelled the Grover Shoe Factory in Massachusetts in 1905, killing fifty-eight people, those inspection rules grew into the ASME boiler code, one of the oldest engineering safety codes still in use.

James Watt, who we will meet in the next section, used the waggon boiler shown below. Water sat in the broad chamber above the furnace, the hot gases passed underneath, and steam collected beneath the rounded roof.

The broad bottom was good at catching heat, but the waggon shape was terrible at holding pressure. Raise the steam pressure in the figure below and compare what happens to the rounded roof, the flat sides and the inward-curved bottom.

The figure also shows why later builders curved the whole boiler outward like the roof. They rolled iron plate into long cylinders, removing the flat sides and inward-curved bottom. They kept the boilers narrow because making a cylinder wider increases the force trying to split it open, even when the pressure stays the same.4 Better iron and riveting then made much higher pressures possible, and around 1800 Richard Trevithick was running engines at several atmospheres.

Now Newcomen's use of a vacuum makes sense. His boiler could push with perhaps fifty grams per square centimetre above atmospheric pressure. By condensing the steam and letting the atmosphere push the piston instead, he got close to one kilogram per square centimetre, around twenty times as much force from the same boiler.

Watt's separate condenser

In 1765, Watt was repairing a model Newcomen engine at the University of Glasgow. He was amazed by how much steam it consumed and began trying to understand where it all went. He discussed the problem with his colleague Joseph Black, who was studying the heat absorbed while water boils. Black called it latent heat. For a kilogram of water, boiling it away takes more than five times as much energy as heating it from freezing to boiling.

With this knowledge, Watt calculated the exact amount of water needed to condense the volume of steam in the cylinder. He was surprised to find that this exact amount barely made a vacuum at all: the condensing steam dumped its latent heat into the spray, warming the water until it stopped condensing anything. Adding in more cold water just cooled the cylinder down more, wasting steam to heat the cylinder back up on the next stroke. Watt's brilliant insight was to add a second vessel that could stay cold while the cylinder stayed hot.5

At the end of the stroke, a valve opened and the steam rushed into the cold vessel, called the condenser. As the steam turned back into water, the pressure fell in the condenser and, through the connecting pipe, in the cylinder as well. A small air pump driven by the engine drew out the condensed water, along with any air that had leaked in, on every stroke. Keeping the cylinder hot and the condenser cold cut coal consumption by about two thirds! Watt and his business partner Matthew Boulton turned the saving into a business model, charging customers one third of the money they saved on coal.

Better tools for making precise cylinders allowed Watt to make another important change: he closed the top of the cylinder and used steam on both sides of the piston. Steam pushed down while the condenser lowered the pressure below; on the return stroke, the same thing happened in the opposite direction. This was the double-acting engine. Below, we can compare it with the single-acting cylinder it replaced.

The same cylinder now produced power on both strokes, and the steady push-pull made the engine much better suited to driving machinery. But getting steam in and out of the cylinder was now more complicated. One end had to connect to the boiler while the other connected to the exhaust, and then the two connections had to switch before the piston returned.

The slide valve

Early steam engines used several separate valves and linkages to route the steam. Our engine does all of this with one slide valve. It moves only a few centimetres, connecting one end of the cylinder to fresh steam and the other to the exhaust. As the piston reaches the end of its stroke, the valve slides across and swaps the two connections.

The valve sits inside the steam chest, an iron box bolted to the side of the cylinder and kept full of fresh steam. Three ports open into the chest. The two outer ports connect to the ends of the cylinder, while the middle one carries away the exhaust. The valve is shaped like a wide, hollow D. One edge uncovers a cylinder port and lets fresh steam enter, while the hollow back joins the other cylinder port to the exhaust.

The valve needs to move in perfect synchronization with the piston, or the engine will not work. This motion comes from an eccentric on the engine's rotating shaft. The eccentric is a circular disc mounted slightly off-centre, so its centre travels in a small circle as the shaft turns. A strap around the disc follows this motion and drives the valve rod back and forth. Its position on the shaft is chosen so the next steam port begins opening before the piston reaches the end of its stroke.

Now, we can see how the piston, valve gear and eccentric work on our beam engine.

Using less steam

We can save a surprising amount of coal by closing the steam port before the piston reaches the end of its stroke. The trapped steam continues to expand and push the piston, although its pressure falls as the volume grows. Closing the valve at halfway, called cutoff, uses half as much steam while still producing about 85 percent of the ideal work.6 Watt patented this idea in 1782. Later compound engines sent the exhaust from one cylinder into a larger cylinder, then sometimes into a third, extracting more work as the steam expanded.

Measuring the work

Everything we have just discussed happens inside an opaque cylinder. In 1796, Watt's assistant John Southern built an instrument that let them see inside. A small spring-loaded piston moved a pencil up and down with the pressure, while a card moved sideways with the main piston. The resulting indicator diagram showed the pressure through the entire stroke, and the area inside the loop measured the work produced.

A leaking piston, late cutoff and restricted exhaust each produce a different shape, allowing an engineer to diagnose the engine from a single card. Boulton & Watt found the instrument so valuable that they kept it secret for years.7

We can now control the steam and produce power in both directions, but the piston still moves back and forth. This is called reciprocating motion. Pumps can use it directly, but the mills driving the Industrial Revolution needed rotation.

Making rotation

To turn the piston's back-and-forth motion into rotation, our beam engine uses a crank, although Watt's first rotating engines could not use one.8 A pin offset from the centre of the shaft is joined to the piston by a connecting rod. The push on the pin turns the shaft, but not equally through the revolution. Twice per turn the crank and connecting rod line up, at positions called dead centres, where the piston pushes straight through the shaft and produces no rotation at all. With nothing to carry it past these points, the engine would stop the first time the crank reached one.

The large flywheel fixes this problem. It stores energy while the crank has good leverage, then returns that energy to keep the engine spinning past the dead centres. In the figure below, the shaded band in the inset shows the flywheel collecting and repaying energy through each revolution. Try the flywheel mass slider: a heavier wheel changes speed less, giving the engine a smooth and steady rotation.

Our engine can now turn a shaft without stopping. But joining the piston rod to the crank turns out to be harder than it looks.

The beam and the parallel motion

Now, look closely at the connecting rod in the figure below. As the crank turns, its pin moves sideways as well as up and down. The piston rod cannot follow it because it must travel straight through the seal at the top of the cylinder. If we connect them directly, the rod pushes the piston sideways and quickly destroys the seal.

The beam carried the sideways load into a large round bearing, which workshops could make accurately. But its end moved in an arc, and Watt still needed the piston rod to travel in a straight line.

His ingenious solution was the parallel motion, patented in 1784. A set of hinged links joins the beam to a fixed point on the engine. As the beam pulls the piston rod sideways in one direction, another link pulls it almost exactly the same amount in the other. The two curves cancel, leaving a path that is remarkably close to a straight line. Watt was so pleased with the mechanism that he wrote he was “more proud of the parallel motion than of any other mechanical invention I have ever made.”

Our piston can now turn the crank without being pulled sideways. At the far end of the beam, we also get a convenient source of back-and-forth motion, which the engine uses to keep its boiler filled with water.

The pump

As the engine runs, the boiler turns water into steam. To keep it going, we need to replace that water without stopping. We can't simply connect a water tank, because the pressure inside the boiler would push the water back out. Instead, the far end of the beam drives the small pump beside the base of the engine, forcing fresh water into the boiler.

Inside the pump is a narrow plunger and two one-way check valves. As the plunger rises, the pressure falls, the inlet valve opens and water enters from the tank. On the way down, the pressure rises, closing the inlet valve and opening the outlet towards the boiler. The changing water pressure operates both valves automatically.

The pump must produce slightly more pressure than the boiler, but it does not need to move much water on each stroke. Making the plunger narrow keeps the required force small, for the same pressure-times-area reason that made our engine piston wide. A small part of the engine's power can now keep the boiler full, while the rest turns the flywheel.

Powering the mill

We talked about why mills need rotation, but not how they used it. Before steam engines, water-powered mills had to sit beside a river. The flowing water turned a large waterwheel, which drove a main shaft, and iron shafts, pulleys and leather belts carried that rotation through the building to power the machines. Our example mill here has a saw for cutting wood and a power loom which wove cloth. Click either machine to shift its belt onto the loose pulley; that machine will coast to a stop while the shaft and the other machine continue running.

It is not intuitive that a leather belt can transmit enough power to drive a machine that ten strong people could not. With only friction between the iron pulleys and the leather providing the connection, it seems that the belt would slip. The physics underlying friction is fascinating. Imagine a huge ship tied to an iron bollard on the dock with a rope. Tension in the first small part of the rope presses it against the iron, and the resulting friction reduces the tension that reaches the next part, and so on around the post.9

Now, let's return to leather belts and iron pulleys. A belt is installed under tension, so at rest its two sides pull with roughly equal force. Once the machine needs power, friction transfers some of that pull from the returning side to the driving side.10

The power transferred by a belt is the difference in tension multiplied by the belt speed. At full mill scale, a sixteen-foot flywheel at sixty revolutions per minute has a belt speed of about fifteen metres a second. If the load makes one side pull with 2,000 newtons more than the other, a foot-wide leather belt can carry about forty horsepower.

The fight for water

Richard Arkwright's water-powered mill at Cromford opened in 1771, and the factory system that followed created fierce demand for the best river sites. Water-powered mills were also dependent on the weather: a dry season could shut down the factory.

Steam pumping engines offered a solution. An engine lifted the water that had passed beneath the wheel back up the hill, allowing the same water to fall through the wheel again. This kept the smooth turn of the water wheel, but wasted coal moving the water. Watt sold sixteen to twenty horsepower pumping engines to deliver ten horsepower to the machines.

Watt's double-acting engine, beam, crank and flywheel let the engine directly turn the line shaft. This met a huge demand from mill owners who wanted to build near workers and materials rather than around a particular stretch of river.11 One problem remained, though: every time a machine was turned on or off, the load on the engine changed.

The governor

The beam engine still needed a way to keep its speed constant. Imagine it at the beginning of the day, turning at 30 rpm with no machines connected. When the first machine is connected, it draws power from the engine and slows it down. The engine driver could open the throttle by hand until the shaft returned to 30 rpm, but this was tiring work, and mistakes had severe consequences. A cast-iron flywheel could burst if it spun too quickly.

Instead, Watt adapted a device used on windmills to adjust the steam automatically.12 Bevel gears turn a vertical spindle, and two heavy balls hang from hinged arms attached to it. As the engine speeds up, the balls swing outward and lift a sliding collar. A fork and long rod carry this motion across the engine and turn the steam cock towards closed. When the engine slows, the balls fall and open the cock again.13

A governor can overcorrect. The engine speeds up and closes the throttle too far, then slows down and opens it too far, repeating the cycle in a motion called hunting. In 1868, James Clerk Maxwell studied when these oscillations grow or die away in his paper “On Governors”. This became one of the beginnings of modern control theory.

The whole machine

Let's return to the complete engine from the beginning of the article. Every mechanism we studied on its own is here, running in its place. The figure follows the power once along its whole path, from the boiler steam to the belt that leaves for the mill.14

Epilogue

The beam engine was a product of the tools and science of its time. Watt used a beam and parallel motion partly because the workshops of the 1780s could not make long, accurate guides for a crosshead. As planing machines improved during the nineteenth century, those straight guides became practical. The heavy beam was no longer required, and by the 1860s most new mill engines drove the flywheel directly.15

Line shafts and leather belts outlived the beam engine, remaining above factory floors well into the twentieth century. Electric motors finally gave each machine its own source of rotation. Wires replaced the long shafts and belts, and stopping one lathe no longer changed the load on a central engine driving the entire mill.

The most dramatic change was how much power newer engines extracted from coal. Corliss valves controlled steam expansion more precisely, compound engines expanded it through several cylinders, and turbines eventually replaced the piston with a continuously rotating wheel. Newcomen converted only about half a percent of the heat into useful work. Watt's condenser raised the useful share to roughly three percent, enough for steam power to move away from the coal mines. By the 1890s, high pressure and compound expansion pushed large marine engines such as the Titanic's beyond ten percent. A modern steam turbine plant converts more than forty percent.

Footnotes

1. Thomas Savery tried to use higher-pressure steam in the 1690s with boilers made from soldered copper. The fire could soften the solder, and the leaking joints needed frequent repair. Newcomen took a different route. Because the steam in his boiler was barely above atmospheric pressure, he could use thin lead and wrought-iron plates joined with rivets. The seams still leaked and the metal corroded, but the boiler did not have to contain the pressure that Savery's pump required. ↩

2. Casting a large iron cylinder was much easier than making the inside straight and round. Newcomen's cylinders were ground by hand, then sealed with a leather flap covered by a layer of water, which could follow the uneven bore. Denis Papin had proposed the vacuum-piston principle in 1690: a small amount of water boiled beneath a piston and pushed it upward, then condensation allowed the atmosphere to force it down again. His apparatus demonstrated a single stroke but did not become a continuously running engine. ↩

3. A piston 50 centimetres across has about 2,000 square centimetres of area, enough to collect two tonnes of force from a perfect vacuum. After allowing for leaks and the weight of the pump rods, it might do about four kilowatts of useful work. A horse can sustain much less than one horsepower over a working day, so replacing the engine required a relay of perhaps fifteen or twenty animals. Watt later sold his engines by the number of horses they replaced, and fixed one horsepower at 33,000 foot-pounds per minute. ↩

4. For a barrel with radius r and length L, the cut has an area of 2rL, so pressure p pushes the halves apart with a force of 2prL. Two edges of length L resist that force, leaving pr in each metre of plate. At two atmospheres and a half-metre radius, this is about ten tonnes per metre. The stress running lengthwise is only half as large, which is why a cylindrical boiler tends to split along its length like a sausage. A sphere divides the load equally and is stronger still, but it was much harder to make from rolled and riveted plate. ↩

5. Watt's engine needed a much more accurate cylinder than Newcomen's loose, water-sealed piston. Around 1775, the ironmaster John Wilkinson built a boring mill with a rigid cutting bar supported at both ends, adapting techniques he had developed for boring cannons. In 1776, Matthew Boulton reported that a 50-inch cylinder installed at Tipton varied by less than the thickness of an old shilling. This accuracy kept the steam from leaking around Watt's piston and made the new engine practical. ↩

6. For a cylinder with volume V and pressure p, admitting steam for the full stroke produces work pV. With cutoff at half stroke, the admitted steam produces pV/2 during the first half. As it expands through the rest of the cylinder, it adds about 0.35 pV more, assuming it follows Boyle's law and remains hot. This gives 85 percent of the full-stroke work from half the steam. Cutting off earlier saves still more steam, but eventually the falling pressure becomes too weak to overcome friction and the poor leverage near dead centre. ↩

7. The pressure and volume graph outlived the mechanical indicator. In 1834, Émile Clapeyron used the same type of diagram to explain Sadi Carnot's theory of heat engines, and thermodynamics still plots pressure against volume today. ↩

8. James Pickard patented the use of a crank on a steam engine in 1780, so William Murdoch designed the sun-and-planet gear as a way around the patent. A gear attached to the connecting rod travelled around a second gear on the flywheel shaft, turning the shaft twice for every cycle of the beam. Once Pickard's patent expired in 1794, builders returned to the much simpler crank used on our engine.

   Murdoch's mechanism belongs to the epicyclic, or planetary, family of gears. Several planet gears can share the load while the input and output remain on the same axis, making the arrangement compact and strong. Planetary gears appear in cordless drills, bicycle hubs, automatic transmissions and wind turbines. Hybrid cars even use them to divide power between the engine, electric motor and wheels. ↩

9. If we slice the wrapped rope into small pieces and add the force vectors, each piece has a small inward force equal to the local tension multiplied by the angle it covers. Friction can remove up to μ times that inward force, about 0.3 for rope on cast iron.

   Repeating that fractional reduction produces the exponential e−μθ. With a 2,000-newton pull, about the weight of an upright piano, one turn around the post leaves 300 newtons, two leave 46, and three leave seven. ↩

10. A flat belt tends to wander off a truly cylindrical pulley. Millwrights made the pulley slightly larger at the centre, forming a shallow crown that continually steers the belt back. When the belt arrives off-centre it first meets a coned surface, and the tilted contact carries its leading edge a little towards the crown on each turn. Once centred, both halves of the crown steer equally and the belt remains centred. This simple change kept the belt on the pulley without guides. ↩

11. Between 1775 and 1800, the Boulton & Watt partnership built 496 engines. Thirty-eight percent were pumping engines, while sixty-two percent produced rotation, mostly for the textile industry. ↩

12. In 1788, Boulton told Watt that he had seen spinning balls used to regulate millstones in Manchester. Watt adapted the mechanism to steam engines, but never patented the borrowed idea. ↩

13. The height of the balls gives a surprisingly direct measurement of speed. Balancing their outward motion against gravity gives h = g/ω2, where h is the vertical distance below the hinge. The mass cancels, so heavier balls push harder on the linkage but rise to the same height. At the crankshaft's speed the arms would need to be about a metre long, so the bevel gears spin the governor faster and let it fit on the short pillar. ↩

14. The oldest rotative engine still in existence, installed at Whitbread's brewery in London in 1785, is built to the same pattern as ours. Its cylinder was 64 centimetres across and the piston swept 1.8 metres on each stroke, a column of steam taller than a person. The flywheel, 4.3 metres across, turned twenty times a minute — a leisurely one revolution every three seconds — burning roughly forty kilograms of coal an hour. Watt rated it at ten horsepower, and it replaced the wheel of horses that had driven the brewery's mills. When it was converted to double action in 1795, the same cylinder was re-rated to fifteen horsepower; the brewery declined twenty, because Boulton & Watt's annual fee rose with the power. It stayed at work for a hundred and two years. ↩

15. American paddle steamers kept the beam engine into the 1880s, mounting a walking beam high above the deck. Their paddle wheels needed high torque at only about twenty revolutions per minute, and a shallow riverboat had more room above the water than below it. Ocean-going ships folded similar machinery into the hull, then moved to smaller and faster engines as the screw propeller replaced the paddle wheel. ↩

Sources & credits

• Thanks to Bartosz Ciechanowski, whose exceptionally well written and designed articles inspired this one.
• CAD model: Horizontal Beam Engine on GrabCAD, based on the A. J. Reeves “ME Beam” model engine. The modified model used in the article is available on Onshape.
• Geometry and motion were exported from Onshape with glinscott/onshape_exporter and rendered with three.js.
• The interactive figures were built by myself and Claude Fable 5.
• John Farey, A Treatise on the Steam Engine (1827), especially the contemporary descriptions and performance figures for atmospheric and early rotative engines.
• H. W. Dickinson and Rhys Jenkins, James Watt and the Steam Engine (1927).
• L. T. C. Rolt, Thomas Newcomen: The Prehistory of the Steam Engine (1963).
• D. S. L. Cardwell, From Watt to Clausius: The Rise of Thermodynamics in the Early Industrial Age (1971).
• Richard L. Hills, Power from Steam: A History of the Stationary Steam Engine (1989).
• Robert H. Thurston, A History of the Growth of the Steam-Engine (1878).
• Science Museum Group collection notes for the Newcomen atmospheric engine and Watt's 1765 separate condenser.
• Science Museum Group collection notes for Papin's 1690 piston proposal and Wilkinson's cylinder-boring mill.
• OpenStax, “Pascal's Principle and Hydraulics”, for pressure transmission through the boiler water.
• John H. Lienhard, “Steam Boilers”, The Engines of Our Ingenuity, for Savery's soldered copper boilers and Newcomen's low-pressure construction.
• James Clerk Maxwell, “On Governors”, Proceedings of the Royal Society 16 (1868), 270–283.
• ASME, “The History of ASME's Boiler and Pressure Vessel Code”, for the later boiler-inspection and code history.
• ASME, Boulton & Watt Rotative Steam Engine, for the effect of its conversion to double action.

Bluesky’s Atmosphere Protocol has the chance to become the base for a whole generation of applications. But it’s developing in surprising and disappointing ways.

I spent part of last week at the Local First Conference in Berlin. I was not surprised to find that AI coding was on everyone’s minds. Less expected was how pervasive ATProto was.

I’ve been considering trying to build on ATProto for a few months, so this was a great chance to ask a bunch of questions from both people building on it, and the people actually building it (multiple key Bluesky people were there).

I want ATProto to be the answer for me. I want to again live in a world where application developers use public, community driven standards to create interoperable applications with common data standards. I think it can be. But unfortunately I think it’s not on track to be that right now.

In this post I’m going to describe what I have been wanting to build, and how ATProto’s current and proposed designs do and don’t work with that. And I’ll also throw in some comments about what’s been surprising to me.

I’m probably trying to do too much here. But my main goal is to provide feedback on the proposed design, and it’s hard to do so without being clear about what I want to build and why.

Start With Reviews

The smallest version of what I want to build is a suite of applications for recording reviews. In an ideal world, these applications would supplant Yelp, GoodReads, Letterboxd, and a bunch of similar applications for just about anything you might consider reviewing. (Yes, I know there are already people building versions of these on ATProto; I’d want to collaborate with them.)

I want to replace these applications not because of their function, but because of their business models. Neither my wife nor I will use them the way they’re intended, and I think most people are like one or the other of us.

Local First Data

For me, I don’t want these companies to own my data. We have more than a thousand bookmarks in Yelp, but… they’re stuck in Yelp. I can’t do anything else useful with them: I can’t publish them, I can’t share them, I can’t write scripts across them, I can’t version control them. They belong to Yelp, not me. And, frankly, the experience they provide for managing lists of bookmarks kinda sucks.

This is a classic local first use case. I could adopt one tool that was great for writing reviews, another that used the same data but was better at publishing best-of lists for my friends, and another that made it easy to share “best business books” lists on my web site. No single app is going to do all of these, but a shared, public data model allows everyone to have what they want.

I’ve lost the willingness to give big corporations a permanent right to control how I use my own data. (That’s why I’m writing this in plain text and publishing via Hugo.) The apps only get worse over time. And if I’m going to invest that much time in recording my opinions, I want to control its use, not let Yelp do it.

Public and Private

As to my wife, she will just never publish anything online. Never. If I want her to review books or restaurants or recipes, it has to be in a place where I – and maybe the rest of our family – will see it, but no one else.

Personally, I am willing to publish public reviews. And sometimes I specifically want to, like sharing books I found useful as a founder. But doing so is complicated. The information I would record about a restaurant for myself is quite different from what I would want the wider world to know. Yet… most apps don’t allow you to draw this distinction. Reviews are public. Your opinion affects the subject’s reputation, even if you didn’t want it to. (I’m vegetarian and autistic, so a place working for me is not necessarily relevant for other people.)

Fundamentally, most of these applications are built to enable influencers: People who want to build a following online for their reviews. “Oh man, she has the best book recommendations!” “If he likes the restaurant, I know I will too!”

But most of us don’t want to become influencers. If anything, we’ve generally learned the lesson that it sucks to be followed by a bunch of people online. I want to keep my stuff private. I don’t want my opinions to matter to anyone else.

I mean sure, when someone comes to town I want to be able to give them a list of restaurants they should visit and activities they should do. But I want to send it to them, not the rest of the world.

We can split the world into three kinds of people:

• Those who would only record a review if other people will read it. These are the influencer-wannabes
• People like me, who would provide a mix of public and private reviews, depending on the circumstance
• People like my wife who will only ever record reviews if they can be confident it will be private to just their friend group, or even just to themselves

The current crop of applications works fine for the first group. But it leaves the latter two out in the cold.

And I bet anything that those other two groups are much larger.

So, what we need is a system that allows application developers to easily let their users choose how public or private they want to be – entirely public, entirely private, or sharing piecemeal with individual groups.

What does this have to do with ATProto?

Let’s start with what’s great about ATProto: It seems to be the first protocol designed to solve identity at scale, enabling any application to build ATProto identities into their application for authentication, tracking followers and following, and the other related features that basically every application needs.

Its identity system not perfect (I wish it were more human readable, especially). But it seems to be close enough today that nearly every speaker at Local First this year mentioned it. No more does every app developer have to create their own social graph, their own identity and authentication systems, their own means of finding friends.

Unfortunately, at least for now, the rest of the protocol is of limited use for my goals.

Today, ATProto is public-only: It assumes that everything you do will be published online for the whole world to see. That design decision is baked into everything from the storage systems to the structure of the publishing services.

The community is currently designing what they call “permissioned data”. (I think this is a silly name, and it should just be called “private data”, which is better both because “private” is actually a word, and also because it describes user behavior instead of technical implementation, which is a far better naming practice).

Unfortunately, while most of the assumptions driving the design seem reasonable, the resulting design looks very hard to build on. A lot of where I think it goes wrong is captured perfectly in the above linked post:

Before we get into it, the through-line on all of this is that public broadcast data is substantially different from permissioned data.

I fundamentally disagree with this. (I am not the only one.)

Private and public data are basically identical. A restaurant review is a restaurant review whether I am the only one who ever sees it or it gets shouted to the rooftops. A book review I share with my book club is identical to a book review I share with my wife or publish on my web site.

Data I publish to the world is just a special case: The permission is world-read. Just like data I never publish is a special case: It has no readers other than me.

Unfortunately ATProto is already used out in the wild for Bluesky, and it is designed and built as a public-only protocol. It would be hard to change it to support access control, limited distribution, and the other features you need with private data.

So, instead, the community has, from what I can tell, just… designed a completely independent system for private data.

Permissioned Data relies on the existing identity system and lexicons (for defining data types). But adds entirely new data structures, plus new methods for managing and validating that data. (Note that the link above is to one in a series of posts on the design; you can get to all of them from there.)

So, as an app developer, you have to essentially write two applications: One for public data, and one for private. But, your users don’t think of it as two apps. “These are restaurant reviews. All the restaurant reviews should go together.” So your job, as the developer, is to support these two data systems, and two protocols, but never let the user see that they’re totally different.

That gets messy real fast. You’ve got a private post you want to make public? You’re not modifying it – you’re deleting the old one and making a new one in the public subsystem. Does it keep its likes? Its reposts? Links to it? 🤷‍♂️ No idea, but… probably not.

It’s actually worse than that. ATProto’s storage subsystem, the “Personal Data Server”, provides direct access to your data. In other words, you should expect more than one application to want to read and write your data. So now, anyone who is interested in this data has to write two versions, but lie to their users and hide any differences.

Because, again, there aren’t actual differences. The actual data being stored is the same. The way the user thinks about it is the same.

To me this is a sign that the current proposal is flawed.

I don’t know how to fix it. I’m nowhere near close enough to conversation to think I can propose solutions yet. But hopefully I can help people see problems, at least.

One last local-first note

I only had a high-level understanding of ATProto went I went to Local First, so I learned a lot while I was there. One thing I was definitely wrong about was how the PDS worked.

I naively assumed it worked a lot like a git repository. The ATProto community regularly says that I own my own data. I tend to assume that means that I have a copy of it, and that when I make changes, I am operating against my copy then distributing it. This is exactly how git works: I clone a repository, make my changes, then push them back up to the server for distribution. I have a hard time imagining “owning” my data without always having a copy of it.

But no, that’s not at all how ATProto works. The PDS is “your” server, but… it’s actually a server. You do not push data up and pull it down; you speak a protocol to it. You only “own” your data because it and the protocols for accessing it are public, and you can change where it is stored.

It does use cryptography-based data integrity guarantees like git does, which is one more reason I assumed I could easily manage the data like git does. But those guarantees are only used on the server, not in the protocols you use to provide new data.

ATProto meets about half of the design principles in the Local First essay linked above. But it clearly misses on the data being at your fingertips or the network being optional, and also misses out on a lot of implied abilities because you’re always talking to a remote server.

If you want to record a review while you’re offline… best practice is to essentially store those reviews in a temporary area, then push them to the server when you’re back online. In other words, you have to build your own custom storage and sync system if you want offline support. Obviously not impossible, but also obviously not local-first.

If you then cross this with the current design for Permissioned Data, I’ve got a bit of a mess:

• Custom local storage for offline data
• Custom sync system for emptying offline data
• Sync system must use different protocols for public and private data
• Online app usage must also use separate read and write systems for private and public data, and is likely different from the system used for offline support

As an application developer committed to both local first principles and letting the user choose between public and private, ATProto is working at least as much against me as with me. Am I really better off building on this, versus designing my own system? That question is doubly scary when you recognize that the protocol designer and I are so far apart philosophically.

If I believe public and private data are fundamentally the same, just with different access rights, but the community believes they are and should be different… I am fighting the protocol, storage system, and community every step of the way. I’ve done this before, and it sucks.

Conclusion

I’m still excited about ATProto. And thankfully, the Permissioned Data design is early enough that it can still be improved. And I’m not the only person out there pushing back on the existing design and its goals.

Even if the proposed design goes through, I can build at least somewhat on ATProto, maybe using its identity and lexicon systems.

But I came up in the 90s, when the entire internet was built on standardized protocols that were internet-scale, resilient, maintained by the community and a standards group, and built to empower its users rather than enrich whoever could build the biggest moat. I want that world again.

I think ATProto has the opportunity to be the first new protocol of this type in decades. I hope they go for it, and build something every application developer (but especially me!) wants to build on.

On July 23, 1985, Commodore introduced its Amiga 1000 computer. And let’s just say the world wasn’t ready for it yet. Dave Haynie, a Commodore engineer who worked on the later models, has said there was no such thing as a 1980s computer. There were 1970s computers and 1990s computers, and it was the Amiga that dragged the rest of the industry into the 1990s.

Ten years ahead of its time

A typical Amiga 1000 setup in 1985 cost nearly $2,000 at the time. Living in the future was expensive.

It is not hyperbole to say the Amiga 1000 was 10 years ahead of its time. Its custom chipset had color graphics with up to 4,096 colors, stereo sound, a graphical user interface. Its operating system had full preemptive multitasking. The combination of the chipset and the operating system meant it could run on a Motorola 68000 CPU clocked at 7.14 megahertz and 256 KB of RAM. And it could boot and run off a single floppy drive. Amiga demonstrated it in January 1984, when the chips existed only as prototype circuit boards. By the end of 1985, it was a shipping product.

Realistically, an ideal configuration had dual floppy drives and 512k of RAM, and a megabyte was a lot nicer.

But dual floppies and a meg of RAM wasn’t an outlandish machine in 1985, and in that configuration, it was smooth and pleasant to use. Commodore commissioned artist Andy Warhol to demonstrate it by using it to create digital art to show off its capabilities.

Multitasking: The Amiga 1000’s secret

Graphical user interfaces weren’t exactly new in the summer of 1985. Apple had introduced the Macintosh about a year and a half earlier, and the Lisa predated even that by a nearly a year. Atari released its ST computer three months before the Amiga, and the ST also had a graphical operating system with a mouse and color.

It was multitasking that put the Amiga on the map. And it wasn’t the cooperative multitasking that Windows 3.0 and 3.1 did or Apple System 7 did. It was full preemptive multitasking like Windows 95. The only thing Windows NT did that the Amiga didn’t was memory protection.

The Amiga’s operating system wasn’t based on Unix and it certainly wasn’t based on Windows NT. Its direct ancestor was an operating system called TRIPOS that originated at Cambridge University.

Adding memory protection would have been a nice touch, but the additional required hardware would have increased costs. And the Amiga at its debut was not exactly inexpensive. The base system cost $1,295 without a monitor. A color monitor, which you definitely wanted, cost another $300. Add the additional memory and a second floppy drive for another $150 to $200 each, and you soon learned the Amiga you really wanted was more like a $2,000 computer. Living in the future was expensive, but compared to the $2,495 Apple wanted for a less capable machine, the price wasn’t unreasonable.

Living with an Amiga

I’m not sure I even saw an Amiga in person until 1987, but I knew just from reading about it that I wanted one. I wasn’t able to make it happen until 1991, so I was pretty late to the game. But even in 1991, an Amiga felt like living in the future. I could load several programs and switch between them effortlessly, with the only limit being the amount of memory I had. I could connect to a BBS with a terminal program, start a download, then switch it to the background, fire up a word processor, and do my homework while the download was happening. In some cases, I could even fire up a game and play a game while a download happened in the background. I could download stuff while I played Civilization, which was pretty great.

Initially, the Amiga had a reputation for being unstable. But that was largely because early Amiga software wasn’t very well written. Amiga software was prone to ask for resources, and assume the resources were available. Instead of failing gracefully if the resources it asked for weren’t available, it would simply allocate them, and cause a guru meditation, the Amiga equivalent of a blue screen of death or kernel panic. Except it was red, not blue.

But it wasn’t long before the software developers learned how to write code that behaved in this brave new world. Certainly by the time I was in the scene in 1991, the software generally behaved very well. When I bought a Windows PC in 1994, it felt like a serious downgrade. If I treated it like an Amiga and launched a terminal program, connected to a BBS, started download, and then switched over to Microsoft Word, it didn’t always work reliably. More often than not it was fine. But something would crash often enough to make me hesitate doing something I took for granted on an Amiga.

People ask me all the time when PCs caught up. It didn’t happen all at once. And it wasn’t until 1995 that all the pieces were there.

Why the Amiga makes me mad

And that’s why I almost everything about the Amiga makes me mad. With a good product marketing team behind it, it would have been the greatest computer of all time. When anyone asks its engineers why they built it, they tell you things like they wanted to build the very best computer that they could with the technology that was available to them at the time. That’s how engineers think, so it’s a perfectly good answer, engineer to engineer. But that answer doesn’t sell. You have to come up with something about wanting to change the world. And you have to word it in a way that sounds believable if you are going to attract a mass audience. Apple does that very well. Microsoft was never as good at it, but they were better at it than Commodore.

Commodore didn’t have a Steve Jobs or even a Steve Ballmer. They had an oligarch named Irving Gould who was only interested in looting as much from the company as possible as he ran it into the ground. It is a testament to the quality of the product Commodore had that it took him a decade to do it after Jack Tramiel left the company and left him unchecked to rob it blind.

Commodore ran a few ads, but it was clear from their advertising they had no idea what they were selling. The most effective marketers were the people who bought one. Sit down for an hour with an Amiga and someone who knew how to use it, and you’d want one too. But there weren’t enough of us to sustain the platform or the company.

How the Amiga 1000 and its successors were like a kick in the face

Commodore went bankrupt less than 9 years after introducing the first Amiga. And the company they sold it to went out of business about 15 months after buying it. So instead of getting the greatest computer of all time, it felt like we were getting kicked in the face repeatedly. And then we tried to switch to Apple or Microsoft, and waiting years for them to catch up felt like getting kicked in the face by two people.

If you missed out on it, it sounds like I’m being overdramatic. I get it. Nobody knows exactly how many people got to experience it. In the end Commodore sold around 4.91 million Amigas, but at least in the United States, Amiga owners were prone to buy more than one. That makes it hard to say how many distinct owners the machine had. But the people who owned one and lived in the future waiting for the rest of the world to catch up know exactly what I’m talking about.