Over the last few days there has been a lot of discussion about open-weights models, especially those from China. Reports suggest that some US officials are considering banning the use of Chinese open-weights models by US companies. In response, many tech companies have signed a letter supporting open-weights models, and some people have even accused Anthropic of wanting to ban open-weights models as a means of protecting our business. Anyone who has read my past writing should know that I don’t regard such bans as a useful measure, but let me state it clearly so that there is no doubt: Anthropic has never advocated for a ban on open-weights models.

Open-weights models that don’t have dangerous capabilities are a public good: they don’t cost anything besides the compute needed to run them, and they provide value to businesses, developers, and researchers.

Protectionist bans would not address my most serious national security concerns. Specifically, I am worried about two nightmare scenarios. I laid these out in my essay The Adolescence of Technology six months ago1, and have held these positions consistently for many years:

1. My primary concern is the risk that authoritarian governments—not solely the Chinese Communist Party (CCP), although the CCP is clearly the most capable threat—build AI models that are more powerful than those built by the US, and use them to achieve permanent military superiority or perpetrate incredibly deep repression of their own people. This concern is widely shared within the US government: Vice President Vance warned in Paris last year that “authoritarian regimes have stolen and used AI to strengthen their military, intelligence, and surveillance capabilities,” and the Intelligence Community’s 2026 Annual Threat Assessment found that “other global powers’ robust progress in AI is challenging US economic competitiveness and national security advantages.” It is irrelevant whether these models are released with open weights, and certainly irrelevant whether they are used by US businesses. In fact, the most dangerous model may be one that is trained in secret and handed only to the People’s Liberation Army for use in drones and the Ministry of State Security for surveillance and repression.
2. My secondary concern is the risk that powerful AI models may be misused to carry out cyberattacks or biological attacks, and may have serious alignment problems. Open-weights models—it does not matter whether they come from China or anywhere else—do potentially present a higher risk than closed models, because it is very difficult to apply guardrails to them or monitor their usage, and once weights are released they cannot be withdrawn2. But banning the use of these models by US businesses does nothing to address this risk, because bad actors are unlikely to be legitimate US businesses. It would protect US AI companies from competition, but that has never been my goal.

To address these concerns, I do support the following three measures, which I and Anthropic have consistently advocated for:

• We should not sell powerful chips or chipmaking equipment to China, and we should crack down on the rampant smuggling3 and workarounds used to obtain access to such chips. China has limited domestic production capacity, and therefore, due to the scaling laws, cannot build more powerful models than the US without US chips. This is the most efficient and direct way to block threat #1, and by hampering the training of models that are out of reach of US law, it also indirectly helps with threat #2.
• We should crack down on industrial-scale distillation operations. Distillation is a much more compute-efficient process than training models from scratch. It allows China to build much better models than its number of chips would ordinarily enable, and thus partially evade chip bans. Distillation does not allow the CCP to obtain equivalent or superior AI capabilities to the US, but it can bring the Chinese frontier to within a few months of the US frontier. It is true that many of the companies carrying out these operations release open-weights models—but the open weights are far less relevant than the fact that the operations are backed by an authoritarian state seeking to overtake the US at the frontier. We should have policy interventions to deter this behavior. A blanket ban on open-weights models is neither the correct remedy nor something we have called for4.
• All sufficiently capable models, open and closed, should go through mandatory safety testing. The best way to address threat #2 is to just directly test models for cyber, biological, and alignment risks before release. I think this idea is actually close to a consensus: I have been heartened both that the Trump administration has moved in this direction in recent months, and by recent industry proposals that would apply such testing to the most capable models regardless of their country of origin or whether they are open or closed (while exempting less capable models, such as those from startups and academia, entirely). Whether open models do or don’t pose an increased risk, and whether that risk can be mitigated, is something that should emerge from testing, rather than be decided in advance—and there may be promising methods for improving the safety of open-weights models, including recent research from Anthropic on modular training strategies. Note that to be effective, testing would need to be global, which means even the CCP would need to be on board. I think this may actually be possible: as I wrote in The Adolescence of Technology, limited cooperation around preventing AI biological weapons may be possible because it is in China’s interest too.

This brings me to the open letter. I agree with much of it: open weights expand access to the AI economy, they strengthen competition at least for some use cases, and they give customers greater control. Concerns about distillation should be addressed through targeted legal and commercial frameworks—the same measure I described above. But I don’t agree with the letter’s assertions that open-weights models necessarily make it easier to develop safeguards or that broad access to capabilities necessarily helps defenders more than attackers. It seems at least as likely to me that the opposite will be true. For example, I worry that biology will have a strong attacker-defender asymmetry, where sufficiently capable models may be able to quickly weaponize pandemic-level viruses with widely available materials, whereas defense against these agents is a multi-year operational task in the best case (as we saw with Operation Warp Speed)5. Questions like this should be empirically answered by rigorous pre-release testing, not assumed in advance.

To summarize my and Anthropic’s position, we have not and are not advocating for a ban on open-weights models as a category. We should instead focus on keeping powerful chips out of authoritarian hands, stopping industrial-scale distillation, and requiring safety testing of all sufficiently capable models, open and closed.

Footnotes

1. See Sections 3 and 2 of that essay for discussion of misuse for seizing power and discussion of biological risks, respectively.
2. See this report from the UK AI Security Institute, specifically: “The same openness underpinning these benefits precludes many of the safety measures that closed model developers can use to detect and disrupt misuse, iterate on safeguards as vulnerabilities emerge, control user access and withdraw models. Once open-weight models are released, these options are lost permanently: safeguards can be removed, and copies can be downloaded, redistributed, and run on private systems beyond monitoring. For models with dangerous capabilities – including highly cyber-capable models – open weight release therefore creates a persistent and irreversible risk of misuse.”
3. See also here, here, and here for more reports from the US Department of Justice.
4. At Anthropic we’re committed to cracking down on industrial-scale distillation through our own practices, including identifying and banning accounts that use our models in this way. This is challenging—for instance, the relevant accounts can often only be identified after substantial distillation has occurred, and distillation often involves creating large numbers of fake accounts that form a moving target. The practices of any individual company cannot entirely solve the problem, which is why we have called for policy on this issue.
5. See Section 2 of The Adolescence of Technology for a more detailed discussion of biological threats and the offense-defense balance. To summarize, what I believe currently keeps us safe in biology is not “defenders”, or even the availability of materials, but a negative correlation between intellectual capability and desire to commit catastrophic harm. Previous technologies like internet search or even DNA synthesis were nowhere near powerful enough to break this correlation, but I worry that at its current rate of progress, AI will do so very soon. Another way to say it is that a sufficiently powerful technology removes all barriers and exposes whether the attacker or defender has an inherent structural advantage, and I worry in biology it is the attacker.

A hot potato: A federal case in Atlanta is raising questions about a privacy-focused mobile operating system, with prosecutors arguing that its features were used to erase evidence. The US Department of Justice is attempting to prosecute Atlanta resident Sam Tunick under a federal statute that makes it a crime to destroy property in an effort to prevent it from being seized.

The case centers on Tunick's use of GrapheneOS, an open-source operating system that works on Google Pixel phones and lets users enter a passcode to wipe a device clean.

Experts said the legal approach is unusual and may be the first time the law has been aimed at an operating system. "It's concerning – and sends the message that [GrapheneOS] is criminal by default," said Christophe Boutry, a cybersecurity and surveillance expert. Boutry and Bill Buddington, senior staff technologist at the Electronic Frontier Foundation, both said they had not seen a similar case.

The incident began at Hartsfield-Jackson Atlanta International Airport on January 24 of last year. Tunick had just returned from a trip to the Dominican Republic when he was stopped for questioning. According to court testimony, federal agents had already circulated his name and photo internally, saying he was under investigation for "suspected terrorism activities" because of his alleged association with the movement against Cop City.

Tunick was taken to a secondary screening room, where multiple agents questioned him. A motion filed by his defense argues the interrogation focused on child sexual abuse material as a pretext for investigating his connections to the protest movement. The motion also states that Tunick asked four times to speak with a lawyer and was denied each time. According to the same filing, agents did not present a warrant or read him his rights.

Government attorneys and agents pushed back during Monday's hearing. They described the encounter as a routine airport inspection. Larry Findley, a Customs and Border Protection officer, said agents were "looking for anything that's prohibited."

During the questioning, agents repeatedly asked Tunick to unlock his phone and warned they would seize it if he refused. When he finally provided a passcode, the phone appeared to restart. The defense motion states that "the screen went blank, flashed several times, and the phone appeared to restart," resulting in the loss of data.

The wipe is now central to the case. Prosecutors are treating it as an intentional act to destroy evidence, while the defense argues that the search violated Tunick's constitutional rights and that the evidence should be suppressed.

The case raises questions about which constitutional rights apply at US borders, including international airports, where authorities have broader search powers. A judge is not expected to rule on the defense motion until at least late October.

GrapheneOS is designed to improve privacy and security on Pixel phones. Supporters say those tools are legitimate security protections, not evidence of criminal intent. Boutry pointed to France and Spain, where authorities have struggled to gain access to secured devices. He said authorities have treated the use of GrapheneOS itself as suspicious. In Catalonia, Spain, police have been profiling people carrying Pixel phones, assuming they have GrapheneOS installed and are drug dealers or gang members.

"The main goal [of the operating system] is protection of privacy," Boutry said. "They're our phones and the state can't tell us how to use them."

The case is tied to ongoing opposition to Cop City, a $109 million police training facility that opened last spring. The project has drawn opposition from activists concerned about police militarization and environmental impacts. Law enforcement officials have defended it as necessary for training and recruitment.

Previous attempts to prosecute protesters at the state level have foundered, while federal authorities have more recently stepped in, including a separate indictment announced last month.

We have proof automation now

I've long had a soft spot for dependently-typed languages like Coq Rocq and Lean. They offer the possibility of a type system capable of encoding and enforcing arbitrarily subtle invariants. The sort of thing that, in regular languages, ends up (at best) as a comment, and which quickly gets lost as the size of the team grows. Then you get subtle misunderstandings and components that don't quite fit together. It's often the case that those components have grown to a sufficient size that, when the problem is noticed, aligning either of them is a wearying prospect. Perhaps, say dependent types seductively, you could write those invariants formally and have a machine check them.

(p.s. Coq changed its name! I remember many years ago at a Coq conference in Princeton, I tried suggesting that, in an English-speaking world, having a programming language called Coq was an impediment. I don't think the audience agreed at the time. I also joked that many of the talks there sounded like a speech by Tyrion Lannister, there being so many Coqs and Hoares. A joke that was hilarious and timely, even though it fell completely flat, coming as it did before the final season of that show and our collective memory-holing of it.)

The problem has always been that with great type-system power comes great proof effort. I can certainly attest to entire days spent proving really quite simple things. Doing proofs is actually quite fun: it's challenging, interactive, and there's a clear goal. But gosh, does it take a lot of time, especially if, like me, you don't know what you're doing. There's also the periodic, galling experience, at the end of many hours of effort, where you realise that the goal that you're trying to prove is, in fact, false. The classic result here is the retrospective from the seL4 effort that found that, even though the project was large enough for the engineers to develop considerable experience, they spent about 10 times as much time proving as they did designing and implementing. They ended up with more than 20 times as many lines of proof code as they did C code.

That overhead has made programming in dependently-typed languages extremely niche. It has also spurred people to try and automate it away. The attempt I'm passingly familiar with is F*, where the system tries to have an SMT solver automatically discharge the obligations. That certainly works for simple cases, but it's very easy to craft something that causes the SMT solver to go off into space and run for hours, leaving you wondering whether it's ever going to finish. I've seen that people who use these languages a lot have to develop a sixth sense for what is going to make the solver happy, and then craft everything around that. It can help, but to an extent it converts the problem into mysticism: you end up serving a complex and fickle god.

A critical fact is that, at least in theory, once the statement is correct, the contents of its proof are irrelevant: only its existence matters. This is not entirely true because of two complicating factors: first, what the seL4 group called “proof engineering”: the need to structure proofs so that the effort of realigning them after code changes is reduced. And, second, sufficiently complicated proofs can cause even type checkers to blow up and consume vast amounts of memory.

We now have LLMs which, combined with proof irrelevance, promise to be an extremely capable form of proof automation. With sufficient amounts of automation perhaps you don't need to worry about proof engineering nearly so much. You still need to avoid blowing up the type checker but, in my limited tests, LLMs can avoid that. Potentially, LLMs suddenly make dependent-type systems dramatically more practical. I wanted to play around with this so built a Zstandard decompressor in Lean, mostly because I was also curious about Zstandard.

Zstandard seems like it's winning the competition to replace gzip as the canonical compression utility. It's another LZ77-style compressor, but it offers better entropy coding and a careful design that allows it to achieve very impressive decompression speeds. It will never be as beautiful as bzip2, but the shining elegance of the Burrows–Wheeler transform doesn't count for too much in the face of significant practical advantages:

(Measurements taken on the standard reference computer, i.e. whatever the author was using at the time. And note the log scale on the y-axis: gzip and Zstandard are in their own speed class. This is an Apple machine and Apple's gzip is especially optimised; expect gzip to be slower elsewhere.)

Zstandard (by Yann Collet, building on the seminal ANS work by Jarek Duda) has an RFC, but it is quite terse. It contains all the information you need to implement a decompressor, but unless you're already quite familiar with compression, I think you'll need to re-read it a few times to understand what's going on. I, at least, had to read section 4.1 half a dozen times before I felt that I had a decent grasp of it. Too late into this process, I discovered that my colleague, Nigel Tao, has written a better write-up of Zstandard than I was going to manage anyway. So, if you want to understand Zstandard, you should read that. I'm just going to give an explanation of the most interesting bit, the entropy encoder, and mix that in with some evangelism about Lean.

The job of an entropy encoder is, given a set of symbols with non-uniform probabilities, to encode a sequence of those symbols using the fewest number of bits. The classic entropy coder is a Huffman encoder. Huffman encoders build a binary tree with symbols at the leaf nodes, and Huffman showed that a very simple algorithm produces an optimal prefix-tree: you take the list of symbols, you find the two with the least probability, and you form a tree node with them as children. That tree node then has a probability that is the sum of its two children, and then you repeat the algorithm with two fewer symbols, but now with a tree node in the mix. Obviously each step of this algorithm reduces the size of the set of elements by one, so it terminates, and it also produces an optimal tree. Huffman trees are very fast because you can build a table indexed by the next n bits (where n is the length of the longest code). The table entry tells you what symbol you've decoded and how many bits to unread. The drawback of Huffman trees is that they can only use a whole number of bits for each symbol: if you have a symbol where -log2(p) = 2.3 then ideally you want to use 2.3 bits to encode it. But Huffman forces you either to round up to 3 bits or to round down, which will force some other symbols to consume more bits.

Zstandard uses Huffman trees, but it also has a higher-compression entropy encoder called FSE. FSE is a state machine. There are more states than symbols, and each symbol gets a fraction of the states that mirrors its probability of occurrence in the stream. So if there's some symbol that is expected to appear 50% of the time, it gets ~50% of the states. Each state has three values: the symbol for that state, a number of bits to read from the bitstream when in that state, and a baseline state number that is added to those bits to get the next state. Now, if you recall, the problem with Huffman trees was that they could only use a whole number of bits, and these states also read a whole number of bits. But the trick is that if you are aiming to read one and a half bits for a given symbol, then half of its states will read one bit and half of them will read two bits. Then you hit your target on average. The table of states is never transmitted. The RFC prescribes an algorithm for building the table from a list of symbol probabilities, and so only the probabilities need to be transmitted.

Let's do an example. Let's say we have four symbols and we're going to use 16 states. So we have to approximate the symbol probabilities in terms of 16ths. (If you want a more accurate approximation of the probabilities, you can use a larger number of states; zstd actually never uses fewer than 32 states.)

Any symbol may follow any other symbol, and a symbol might only have a single state. So every symbol must be able to reach every state. Take a look at state three, which is the only state for symbol D. Because it's the only one, it has to read four bits, which is sufficient to encode any other state. But if you look at a symbol like B, its states only demand that you read one or two bits. However, the set of 16 possible next states is exactly partitioned between those states for symbol B. So, for any particular state, there is exactly one state for symbol B that can reach it.

FSE decoding table cells tiling the state space. A 16-cell decoding table with symbol B occupying states 2, 5, 8, 11 and 15. Arrows connect each of B's cells to its destination range below: state 11 covers next states 0 to 1, state 15 covers 2 to 3, state 2 covers 4 to 7, state 5 covers 8 to 11, and state 8 covers 12 to 15. Together the five ranges cover all sixteen states.

Again, consider symbol B, which we said had a probability of 5/16. The ideal number of bits to encode that symbol is -log2(5/16) = 1.68. There are three symbol B states that read two bits and two that read one bit. The states aren't used equally often and, weighted by how often they're used, the average comes out to almost exactly the right value for the quantised probabilities. If you want to capture the true symbol probabilities with more accuracy, use a bigger table.

The central trick is that, by giving multiple states to more common symbols, the encoder doesn't just pick a symbol: it also picks which of that symbol's states to land in, and that choice carries information forward to the next symbol. That's where the fractional bits of information go. But this entropy encoder is still just table based, and so it runs very quickly.

The wrinkle is that you can't work forwards. Assume that you want to encode C, D. Which C state do you start in? Well, D only has one state so it has to be the C state which can reach that one. If D had multiple states then you would need to worry about what came after D to know which of those you needed. FSE forces you to start at the end of the sequence and work backwards. (That's not too bad because you usually need to know the whole sequence in order to calculate the symbol probabilities anyway.) Furthermore, a Zstandard compressor thus encodes symbols back-to-front, but writes output incrementally, so the decompressor has to seek to the end of a block and read the bits backwards in order to straighten it out! That's getting into broader details of the format that I'm not going to cover; see Nigel's piece.

Basic entropy encoders do not care about inter-symbol probabilities. I.e. they can't use the fact that the letter Q is disproportionately followed by the letter U (in English). There has to be some other encoding that is exploiting those redundancies. In Zstandard, that's a traditional Lempel–Ziv structure where it encodes either literal bytes or back references to previously decoded data. So FSE is primarily used for efficiently encoding these back reference offsets and lengths.

Lean

Let's talk about Lean! Above I said that it's a dependently-typed language, and that is a concept better articulated in examples than in a complicated definition. So here's the type of a function that reads n bytes from a stream and, if it doesn't throw, returns a byte array that the type system knows is n bytes long.

def IO.FS.Stream.readExact (st : Stream) (n : Nat) :
    IO {ba : ByteArray // ba.size = n} := …

Here is a function that returns two numbers and a byte array such that the first number is prime, the sum of the two numbers is divisible by six, and the byte array is at least as long as the smaller of those two numbers.

def getResult :
    IO (Σ a b : Nat, { bytes : ByteArray //
      Nat.Prime a ∧
      6 ∣ a + b ∧
      Nat.min a b ≤ bytes.size }) := …

That is not a type that anyone will ever need. It's just demonstrating that you can go as wild with this as you want. Dependently-typed languages are sufficient to encode even very complicated mathematical structures, and Lean's dominant use at the moment is as a formal language for stating and proving mathematics. The recent book, The Proof in the Code, is a short, well-written articulation of the story of how Lean came to be. The author does completely butcher constructive mathematics for a few paragraphs but, other than that, I enjoyed it!

Lean is a purely functional language like Haskell, although it has a few properties that make it potentially a lot more convenient as a programming language. Firstly, Lean is strict, while Haskell is lazy. Strictness means that arguments to functions are evaluated before the call happens, whereas in Haskell the evaluation of arguments is deferred until the value is actually required. So in Haskell it's free to write expensive expressions and pass them into functions, because they'll only actually be computed if they end up being used. But it also means that computation can happen in very surprising places in the program. This is a contentious topic but, while I appreciate the elegance of laziness, boy, it can make the performance of programs hard to reason about.

Next, Lean has some nice helpings of sugar. Its monadic do notation contains for loops and return statements and break statements. If you want to program in an imperative style, you can do so pretty reasonably!

Lastly, Lean has an optimisation where it will make mutating updates to objects as long as their reference count is equal to one. So you can mutate an array in place as efficiently as in an imperative language, as long as you are careful not to have a reference to it someplace else. Unfortunately, Lean does not have any aspects of a linear type system that I'm aware of, so it does not help you in ensuring that there is only a single reference to a value. It's a bit of a sharp edge that a seemingly minor tweak to the code can completely crater its performance by holding on to a reference to a large array somewhere inconspicuous. But it does mean that if you are trying to optimise the performance of something, you have a lot more tools at your disposal.

Here's an example of some of this, from the zstd decoder I sketched:

while true do
    let some blockHeaderBytes ← input.readExactOrEof 3 | break
    let some blockHeader := BlockHeader.fromBytes blockHeaderBytes frameHeader
      | throw (.userError "invalid block header")
    let blockBytes ← input.readExact blockHeader.contentSize

    match hty : blockHeader.type with
    | .rle =>
      let b := blockBytes.val[0]'(by
        rw [blockBytes.property, blockHeader.contentSize_rle hty]; omega)

Focus on line 9. There's an array index there, which is exactly the sort of place that implicit invariants live: blockBytes had better not be empty! C-like languages will give you undefined behaviour in that case. Modern languages will throw at run-time, or only give you an optional value to avoid that. Lean has another option: prove that it's not empty. That's what line 10 does. blockBytes.property is the fact that it's as long as the requested read, i.e. exactly blockHeader.contentSize bytes long. blockHeader.contentSize_rle is this:

theorem BlockHeader.contentSize_rle (h : BlockHeader) (hty : h.type = .rle) :
    h.contentSize = 1 := by
  simp [contentSize, hty]

That's a proof that, when the type is rle, the contentSize is always one. With those facts, Lean can figure out the rest.

It's a really short proof and probably I could have figured that out myself, but we can aim much higher:

I wrote an implementation of the FSE table construction algorithm from the RFC. The RFC contains “test vectors” for it: three sample outputs from given probabilities. Obviously those go into unit tests. But, in Lean, we can also prove universal properties of the function:

theorem ofDistribution_wellFormed (h : ofDistribution accuracyLog probs = some t) :
    t.entries.size = 2 ^ accuracyLog ∧
    (∀ s : Fin probs.size,
      t.entries.toList.countP (fun e => e.symbol == s.val) = probCells probs[s]) ∧
    (∀ (i : Nat) (hi : i < t.entries.size) (v : Nat), v < 2 ^ (t.entries[i]'hi).nbBits →
      (t.entries[i]'hi).baseline + v < 2 ^ accuracyLog) ∧
    (∀ (s : Fin probs.size), 0 < probCells probs[s] → ∀ x < 2 ^ accuracyLog,
      ∃! i : Nat, ∃ hi : i < t.entries.size,
        (t.entries[i]'hi).symbol = s.val ∧ (t.entries[i]'hi).baseline ≤ x ∧
          x < (t.entries[i]'hi).baseline + 2 ^ (t.entries[i]'hi).nbBits) := …

Repeating that, in words:

Assuming that the table construction function, when given the “accuracy” constant and a list of symbol probabilities, produces a value, then:

1. The table has the correct size for that accuracy.
2. The number of states for a given symbol is correct given its probability.
3. For all states, reading nbBits bits and adding the baseline value for that state produces a valid state number.
4. For all symbols with non-zero probability, and for all target states, there's exactly one state for that symbol which can reach the target state.

These are the subtle assumptions that an optimised decoding inner-loop requires, and things that can only ever be implicit or mere comments in weaker type systems. Proving strong statements like that is part of the 10× effort that the seL4 retrospective described, and a major barrier to the adoption of dependent types in regular software. Several LLMs can do it automatically now in about 20 minutes, and using only a fraction of a $20/month subscription quota. It'll probably be table-stakes next year. I must admit that they needed to change the table-generating code when doing so: I had used too much Id.run (i.e. dropping into imperative mode) and that's harder for the proof machinery to work with. (But Lean are working on it.) I confirmed that the proofs type-check and that there are no sorrys.

Combining dependent types and LLMs is not a new idea, but not much has been done on applying the combination to quotidian software engineering. Lots more experience would be needed. Very strong types can amplify the scope of changes as they have to be propagated out through all the derived types. Perhaps the proof effort scales poorly in larger systems, such that even modern LLMs can't keep up. Lean is a high-level language, and that's not suited to everything. (My toy Zstandard decoder is 10× slower than zstd on the command line.) Still, proof automation is here now and we, practically speaking, have a new type of programming language available to us. That's exciting!

(I'm not publishing the code because, frankly, for a small, well-defined case such as this, the LLMs can probably do a better job than I did. I did this to learn Lean a little and I don't hold my explorations up as an exemplar. This was inspired by lean-zip which does much more, includes a compressor, and proves round-tripping!)

Aside: verified assembly

AWS made LNSym: a semantics and simulator for AArch64. That's cool. Perhaps we could use it to show equivalence between an optimised assembly implementation of some functions, and their Lean counterparts, and then use the assembly code at run-time? Then we could let LLMs rip at optimisation and they couldn't introduce any functional bugs. Verified assembly is well-trodden in crypto implementations, but perhaps now it could be cheap?

I put some (mostly LLM) time into trying this. The small popcount example from the repo uses bv_decide, which is a certifying SAT solver, and that example requires more memory than my system has, which doesn't bode well. Tiny functions do work, and it is possible to get an equivalence proof to tiny Lean functions, and then to use extern to call them at run-time! But I, and a few LLMs, couldn't get it to scale any further.

Go 1.26 made Green Tea the default GC. We observe its cache-friendliness with perf, visualize the heap to see how Go allocates, and poke at a sparse-page problem a non-moving collector like Go's struggles with.

Go 1.25, released last year, introduced a new garbage collector: Green Tea. And in Go 1.26, released a few months ago, Green Tea became the default. That linked article is excellent. We’ll recap it and take a look at a few programs that benefit the most. And we’ll look at some programs that don’t benefit, that trigger Go’s residual garbage collector bugaboo: its non-moving collector cannot reclaim sparse pages.

Taking a step back, Go manages memory by allocating objects of the same size class (an object’s size is rounded up to the nearest size class) within a contiguous chunk (or span in Go terminology) of one or more 8KiB pages. Size-segregated allocation is common in some malloc implementations (like tcmalloc, which Go’s allocator descends from).

Let’s observe this happening in Go, and then compare it with C#. We’ll randomly allocate objects of three different sizes (small, medium and large). Then we’ll check their heap addresses, walk the address space and print out when we hit one of our objects, one character printed out for each 32-bytes we walk.

First install Go and C#.

sudo apt update -y
sudo apt-get install -y dotnet-sdk-10.0
curl -fsSL https://go.dev/dl/go1.26.0.linux-amd64.tar.gz | sudo tar -C /usr/local -xz
export PATH=$PATH:/usr/local/go/bin

Here’s the pseudocode we’re going for.

struct Small  { a [32]byte }
struct Medium { a [64]byte }
struct Large  { a [128]byte }

constructors = [Small, Medium, Large]
live = [] # stop objects from being collected
for i in range(100):
  live.push(new constructors[rand() % len(constructors)])

for pass in [0, 1]:
  if pass == 1:
    runtime.gc() # trigger the GC

  records = []
  for obj in live:
    records.push((runtime.addressof(obj), runtime.typeof(obj), runtime.sizeof(obj)))
  records.sort(key = r -> r.address)

  cell = 32
  cursor = records[0].address
  for (addr, typ, size) in records:
    while cursor < addr: # no object of ours here
      print("."); cursor += cell
    head = typ.name[0]
    print(upper(head) + "-" * (size/cell - 1)) # "S" / "M-" / "L---"
    cursor += size

Let’s build it in Go.

package main

import (
	"bytes"
	"cmp"
	"fmt"
	"math/rand"
	"reflect"
	"runtime"
	"slices"
)

type (
	Small  struct{ _ [32]byte }  // 32 bytes
	Medium struct{ _ [64]byte }  // 64 bytes
	Large  struct{ _ [128]byte } // 128 bytes
)

type object struct {
	addr uintptr
	size int
	name byte // 'S' / 'M' / 'L'
}

func main() {
	allocs := []func() any{
		func() any { return new(Small) },
		func() any { return new(Medium) },
		func() any { return new(Large) },
	}
	live := make([]any, 100) // keep refs so GC can't reclaim, and so we know each type
	for i := range live {
		live[i] = allocs[rand.Intn(len(allocs))]()
	}

	for pass := 0; pass < 2; pass++ {
		if pass == 1 {
			runtime.GC() // Go never moves objects: pass 1 is identical to pass 0
		}
		objs := make([]object, len(live))
		for i, o := range live {
			t := reflect.TypeOf(o).Elem()
			objs[i] = object{reflect.ValueOf(o).Pointer(), int(t.Size()), t.Name()[0]}
		}
		slices.SortFunc(objs, func(a, b object) int { return cmp.Compare(a.addr, b.addr) })

		fmt.Printf("\n=== pass %d (base 0x%x) ===\n", pass, objs[0].addr)
		draw(objs)
	}
}

func draw(objs []object) {
	const cell, width = 32, 60

	last := objs[len(objs)-1]
	base := objs[0].addr
	grid := make([]byte, int((last.addr+uintptr(last.size)-base)/cell))
	for i := range grid {
		grid[i] = '.'
	}
	for _, o := range objs {
		c0 := int((o.addr - base) / cell)
		grid[c0] = o.name
		for k := 1; k < o.size/cell; k++ {
			grid[c0+k] = '-'
		}
	}

	prev := -1
	for off := 0; off < len(grid); off += width {
		row := grid[off:min(off+width, len(grid))]
		if len(bytes.Trim(row, ".")) == 0 { // row holds none of our objects
			continue
		}
		if prev >= 0 && off != prev+width {
			fmt.Println("             ...")
		}
		fmt.Printf("0x%09x  %s\n", base+uintptr(off)*cell, row)
		prev = off
	}
}

heapwalk.go

Run it and you’ll see something like this.

$ go run heapwalk.go

=== pass 0 (base 0xba4841580c0) ===
0xba4841580c0  M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-
0xba484158840  M-M-M-M-....................................................
             ...
0xba48415bcc0  ............................SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
             ...
0xba4841add40  ..........................L---L---L---L---L---L---L---L---L-
0xba4841ae4c0  --L---L---L---L---L---L---L---L---L---L---L---L---L---L---L-
0xba4841aec40  --L---L---L---L---L---L---L---L---L---L---

=== pass 1 (base 0xba4841580c0) ===
0xba4841580c0  M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-
0xba484158840  M-M-M-M-....................................................
             ...
0xba48415bcc0  ............................SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
             ...
0xba4841add40  ..........................L---L---L---L---L---L---L---L---L-
0xba4841ae4c0  --L---L---L---L---L---L---L---L---L---L---L---L---L---L---L-
0xba4841aec40  --L---L---L---L---L---L---L---L---L---L---

So even though we allocated randomly among objects of different sizes, we can observe the Go runtime fitting objects of each size next to each other.

And even after we ran the garbage collector it didn’t move anything around.

Now let’s take a look at C#.

using System;
using System.Collections.Generic;
using System.Runtime.CompilerServices;

var allocs = new Func<object>[] { () => new Small(), () => new Medium(), () => new Large() };
var live = new object[100]; // keep refs so GC can't reclaim, and so we know each type
var rnd = new Random();
for (int i = 0; i < live.Length; i++) live[i] = allocs[rnd.Next(allocs.Length)]();

// A reference on 64-bit .NET is a plain 8-byte pointer, so reinterpreting one
// with Unsafe.As gives the object's address. Object sizes are measured from
// the heap: allocate several, the smallest gap between consecutive addresses
// is the (aligned) object size, header included.
var size = new Dictionary<Type, int>();
foreach (var make in allocs)
{
    var keep = new object[16];
    var a = new nint[keep.Length];
    for (int i = 0; i < keep.Length; i++) keep[i] = make();
    for (int i = 0; i < keep.Length; i++) a[i] = Unsafe.As<object, nint>(ref keep[i]);
    Array.Sort(a);
    nint best = nint.MaxValue;
    for (int i = 1; i < a.Length; i++)
        if (a[i] - a[i - 1] > 0 && a[i] - a[i - 1] < best) best = a[i] - a[i - 1];
    size[keep[0].GetType()] = (int)best;
}

for (int pass = 0; pass < 2; pass++)
{
    if (pass == 1) GC.Collect();

    // An address is only valid until the next collection, so pause the GC
    // while we take them.
    var addrs = new nint[live.Length];
    GC.TryStartNoGCRegion(1 << 20);
    for (int i = 0; i < live.Length; i++) addrs[i] = Unsafe.As<object, nint>(ref live[i]);
    GC.EndNoGCRegion();

    var objs = new (nint Addr, int Size, char Name)[live.Length];
    for (int i = 0; i < live.Length; i++)
        objs[i] = (addrs[i], size[live[i].GetType()], live[i].GetType().Name[0]);
    Array.Sort(objs, (x, y) => x.Addr.CompareTo(y.Addr));

    Console.WriteLine($"\n=== pass {pass} (base 0x{(long)objs[0].Addr:x}) ===");
    Draw(objs);
}

static void Draw((nint Addr, int Size, char Name)[] objs)
{
    const int cell = 32, width = 60; // cell = the smallest object's size

    var last = objs[^1];
    nint b = objs[0].Addr;
    var grid = new char[(last.Addr + last.Size - b) / cell];
    Array.Fill(grid, '.');
    foreach (var o in objs)
    {
        int c0 = (int)((o.Addr - b) / cell);
        grid[c0] = o.Name;
        for (int k = 1; k < o.Size / cell; k++) grid[c0 + k] = '-';
    }

    int prev = -1;
    for (int off = 0; off < grid.Length; off += width)
    {
        var row = new string(grid, off, Math.Min(width, grid.Length - off));
        if (row.Trim('.').Length == 0) continue; // row holds none of our objects
        if (prev >= 0 && off != prev + width) Console.WriteLine("             ...");
        Console.WriteLine($"0x{(long)b + (long)off * cell:x9}  {row}");
        prev = off;
    }
}

class Small  { public long a, b; }
class Medium { public long a, b, c, d, e, f; }
class Large  { public long a, b, c, d, e, f, g, h, i, j, k, l, m, n; }

HeapWalk.cs

Build and run it.

$ dotnet run HeapWalk.cs
=== pass 0 (base 0x7aea1080a1e0) ===
0x7aea1080a1e0  L---L---L---M-M-SL---L---SM-M-M-M-SL---L---L---L---M-L---SSL
0x7aea1080a960  ---L---M-M-L---M-SL---L---SM-L---M-L---L---L---M-L---L---M-L
0x7aea1080b0e0  ---SL---SSL---L---M-L---L---M-L---L---L---SM-SL---L---SL---L
0x7aea1080b860  ---L---SSSL---L---M-SL---SM-L---SL---M-L---M-L---M-L---SSL--
0x7aea1080bfe0  -L---SSSM-SM-L---M-M-L---M-L---

=== pass 1 (base 0x7aea1080a1e0) ===
0x7aea1080a1e0  L---L---L---M-M-SL---L---SM-M-M-M-SL---L---L---L---M-L---SSL
0x7aea1080a960  ---L---M-M-L---M-SL---L---SM-L---M-L---L---L---M-L---L---M-L
0x7aea1080b0e0  ---SL---SSL---L---M-L---L---M-L---L---L---SM-SL---L---SL---L
0x7aea1080b860  ---L---SSSL---L---M-SL---SM-L---SL---M-L---M-L---M-L---SSL--
0x7aea1080bfe0  -L---SSSM-SM-L---M-M-L---M-L---

And we immediately notice that objects of the same size are not grouped together. (Later on, in a different workload, we'll also notice C# moving objects around in memory.)

The documentation for Go and C# will tell you as much about the behavior of both, but I think it’s nice to also see it demonstrated like this.

Now that we’ve seen how Go memory is allocated, let’s look at how it’s cleaned up.

Mark and sweep

The garbage collector starts at specific roots (e.g. globals and locals) and, historically in Go, follows each pointer until the GC visits all accessible objects. The mark phase. Then, in a second pass, the GC frees any allocated objects that have not been visited. Since these now-freed objects were not accessible from the root tree in the mark phase, they are by definition dead. The sweep phase.

A challenge arises when you have objects A that point to objects B/C/D of different sizes. Objects of different sizes are allocated in different sections of memory in Go. Or even if you have objects A that point to other objects A that were created at very different times; they’re going to exist in very different parts of memory. In both cases, the GC following pointers now introduces random memory access which is measurably less cache friendly.

In Green Tea, Go now scans a memory span for objects and pointers and queues up future spans for scanning based on pointers it has found, rather than following every pointer roughly as it sees one. And while we can’t show this random access behavior happening without applying patches to Go itself (so that we could observe the mark path as it visits each object), we can observe it happening with perf showing both fewer cache misses (per kilo instruction) and faster overall program runs.

Here’s the pseudocode for our workload.

struct Node {a,b,c,d *Node}

mode = packed | scattered

nodes = new [2_000_000]*Node
for i in 0..nodes.len:
  nodes[i] = Node{
    a: nodes[(mode == packed ? i + 1 : rand()) % nodes.len],
    b: nodes[(mode == packed ? i + 2 : rand()) % nodes.len],
    c: nodes[(mode == packed ? i + 3 : rand()) % nodes.len],
    d: nodes[(mode == packed ? i + 4 : rand()) % nodes.len]
  }

for i in 0..100:
  trigger_gc()

keepalive(nodes) # prevent `nodes` from being garbage collected

In order to make the program measurement a little fairer (the scattered version has to do significant work generating random numbers) we’ll separate out the generation of node index offsets:

import array
import random
import sys

n = 2_000_000
order = sys.argv[1] if len(sys.argv) > 1 else ""

if order == "packed":
    a = array.array("I", ((i + k) % n for i in range(n) for k in (1, 2, 3, 4)))
elif order == "scattered":
    r = random.Random(1)
    a = array.array("I", (r.randrange(n) for _ in range(n * 4)))
else:
    sys.exit("usage: gen.py packed|scattered")

assert a.itemsize == 4 and sys.byteorder == "little"  # matches Go's uint32 cast
with open(order+".idx", "wb") as f:
  a.tofile(f)

generate_indexes.py

And the Go workload becomes:

package main

import (
        "io"
        "os"
        "runtime"
        "unsafe"
)

type Node struct {
        a, b, c, d *Node
}

func main() {
        n := 2_000_000

        raw, err := io.ReadAll(os.Stdin)
        if err != nil {
                panic(err)
        }
        idx := unsafe.Slice((*uint32)(unsafe.Pointer(&raw[0])), n*4)

        nodes := make([]*Node, n)
        for i := range nodes {
                nodes[i] = &Node{}
        }
        for i, nd := range nodes {
                nd.a = nodes[idx[i*4]]
                nd.b = nodes[idx[i*4+1]]
                nd.c = nodes[idx[i*4+2]]
                nd.d = nodes[idx[i*4+3]]
        }

        for i := 0; i < 100; i++ {
                runtime.GC()
        }

        runtime.KeepAlive(nodes) // keep `nodes` from seeming to fall out of scope
}

readorder.go

Now generate the index files from the Python script. Then build two versions of the Go workload: one with Green Tea and one without.

python3 generate_indexes.py scattered
python3 generate_indexes.py packed
go build -o readorder_greentea readorder.go
GOEXPERIMENT=nogreenteagc go build -o readorder_oldgc readorder.go

Let’s time the two garbage collectors and the two workloads with perf while collecting information on cache misses.

$ for bin in readorder_oldgc readorder_greentea; do
  for input in packed.idx scattered.idx; do
    echo "=== $bin < $input ==="
    perf stat -e cache-references,cache-misses -r 5 \
      sh -c "exec ./$bin < $input" > /dev/null
  done
done

=== readorder_oldgc < packed.idx ===

 Performance counter stats for 'sh -c exec ./readorder_oldgc < packed.idx' (5 runs):

     1,130,709,755      cache-references                                                        ( +-  0.70% )
       290,434,782      cache-misses                     #   25.69% of all cache refs           ( +-  1.02% )

             4.230 +- 0.145 seconds time elapsed  ( +-  3.44% )

=== readorder_oldgc < scattered.idx ===

 Performance counter stats for 'sh -c exec ./readorder_oldgc < scattered.idx' (5 runs):

    13,247,268,612      cache-references                                                        ( +-  0.38% )
     2,325,799,796      cache-misses                     #   17.56% of all cache refs           ( +-  0.15% )

            11.052 +- 0.154 seconds time elapsed  ( +-  1.39% )

=== readorder_greentea < packed.idx ===

 Performance counter stats for 'sh -c exec ./readorder_greentea < packed.idx' (5 runs):

       481,414,281      cache-references                                                        ( +-  0.27% )
       257,894,055      cache-misses                     #   53.57% of all cache refs           ( +-  0.04% )

           2.69560 +- 0.00385 seconds time elapsed  ( +-  0.14% )

=== readorder_greentea < scattered.idx ===

 Performance counter stats for 'sh -c exec ./readorder_greentea < scattered.idx' (5 runs):

     3,398,491,016      cache-references                                                        ( +-  1.02% )
     2,195,902,796      cache-misses                     #   64.61% of all cache refs           ( +-  0.10% )

            6.9610 +- 0.0108 seconds time elapsed  ( +-  0.16% )

We see very clear improvement for each workload with the new GC. But cache misses seem to have increased with the new GC? That’s not what we expected.

There are two things at play here. First, cache-references and cache-misses in perf usually correspond to L3 cache. So while the percentage of L3 cache misses in the new GC might have gone up, it doesn’t really tell us anything about behavior at the L1 or L2 cache level. And again, we saw the speedup ourselves. So we’re missing something.

Second, the program runtime largely changed between the old GC and the new GC and we haven’t normalized that. perf has an instructions metric we can key on to calculate a standard normalized metric: Misses Per Kilo Instructions (MPKI).

So let’s run perf again and ask for instructions too and calculate MPKI ourselves.

import json, subprocess

for binary in ["readorder_oldgc", "readorder_greentea"]:
    for inp in ["packed.idx", "scattered.idx"]:
        out = subprocess.run(
            ["perf", "stat", "-j", "-e", "instructions,cache-misses", "-r", "5",
             "sh", "-c", f"exec ./{binary} < {inp}"],
            stdout=subprocess.DEVNULL, stderr=subprocess.PIPE, text=True).stderr

        c = {}
        for line in out.splitlines():
            line = line.strip().lstrip("[").rstrip("],")
            if line.startswith("{"):
                r = json.loads(line)
                c[r["event"].split(":")[0]] = float(r["counter-value"])

        print(f"{binary:<20} {inp:<14} {1000 * c['cache-misses'] / c['instructions']:6.2f} MPKI")

perf.py

Run it.

$ python3 perf.py
readorder_oldgc      packed.idx       1.59 MPKI
readorder_oldgc      scattered.idx   12.70 MPKI
readorder_greentea   packed.idx       1.81 MPKI
readorder_greentea   scattered.idx   15.39 MPKI

And we’re still not seeing what we expect! L3 MPKI actually went up with the new GC. But we saw the performance improvements ourselves!

At this point if you want to keep hunting for the cache improvements to show up you’re going to need to switch to a bare metal x86/amd64 Linux machine because most virtual machines do not expose PMU counters needed for L1 events.

I grabbed myself a Vultr bare metal machine and kept going. We will ask perf for L1-dcache-loads and L1-dcache-load-misses to get L1 metrics. Then we’ll calculate MPKI for both the L1 cache and the L3 cache.

import json, statistics, subprocess

print(f"{'binary':<21}{'ordering':<12}{'elapsed(s)':>10}{'L1miss%':>9}"
      f"{'L1-MPKI':>9}{'L3miss%':>9}{'L3-MPKI':>10}")
print("-" * 80)

for binary in ["readorder_oldgc", "readorder_greentea"]:
    for order in ["packed", "scattered"]:
        runs = []
        for _ in range(5):
            out = subprocess.run(
                ["perf", "stat", "-j", "-e",
                 "instructions,duration_time,L1-dcache-loads,L1-dcache-load-misses,"
                 "cache-references,cache-misses",
                 "sh", "-c", f"exec ./{binary} < {order}.idx"],
                stdout=subprocess.DEVNULL, stderr=subprocess.PIPE, text=True).stderr
            c = {}
            for line in out.splitlines():
                line = line.strip().lstrip("[").rstrip("],")
                if line.startswith("{"):
                    r = json.loads(line)
                    c[r["event"].split(":")[0]] = float(r["counter-value"])
            runs.append(c)

        secs = [r["duration_time"] / 1e9 for r in runs]
        ins = statistics.fmean(r["instructions"] for r in runs)
        l1 = statistics.fmean(r["L1-dcache-load-misses"] for r in runs)
        l1l = statistics.fmean(r["L1-dcache-loads"] for r in runs)
        l3 = statistics.fmean(r["cache-misses"] for r in runs)
        l3r = statistics.fmean(r["cache-references"] for r in runs)
        el = f"{statistics.fmean(secs):.2f}±{statistics.stdev(secs):.2f}"
        print(f"{binary:<21}{order:<12}{el:>10}"
              f"{100 * l1 / l1l:>9.1f}{1000 * l1 / ins:>9.2f}"
              f"{100 * l3 / l3r:>9.1f}{1000 * l3 / ins:>10.2f}")

perf2.py

Give it a run.

$ python3 perf2.py
binary               ordering    elapsed(s)  L1miss%  L1-MPKI  L3miss%   L3-MPKI
--------------------------------------------------------------------------------
readorder_oldgc      packed       4.47±0.32      0.9     2.23     26.2      1.61
readorder_oldgc      scattered   11.44±0.85     12.9    31.57     17.5     12.76
readorder_greentea   packed       2.70±0.01      1.0     1.98     53.8      1.81
readorder_greentea   scattered    7.01±0.04      7.3    14.06     63.2     15.37

And now we finally start to see what we’re expecting: while L1 cache miss percentages stay the same or decrease, L1 MPKI more markedly decreases. More of the reads fit into L1 (and possibly L2) cache and didn't even need to get to the L3 cache. The L3 cache misses stop mattering as much. So here, concretely, is at least one of the areas that the Green Tea GC improved on.

Now let’s take a look at one of the areas where the Go garbage collector still struggles.

Worst-case cleanup

Because Go will never move memory around to compact or defragment it, we can easily get into a seemingly-ridiculous scenario where we free up a huge percentage of our objects but Go cannot entirely reclaim unused memory.

Let’s go back to our initial program drawing out memory allocation for S/M/L objects. This time, after we allocate our objects, we’ll free 90% of them. We’d hope for memory usage to shrink 90%, but we’ll see it actually does not happen because Go does not compact or move objects around.

package main

import (
	"bytes"
	"cmp"
	"fmt"
	"math/rand"
	"reflect"
	"runtime"
	"runtime/debug"
	"slices"
)

type (
	Small  struct{ _ [32]byte }  // 32 bytes
	Medium struct{ _ [64]byte }  // 64 bytes
	Large  struct{ _ [128]byte } // 128 bytes
)

type object struct {
	addr uintptr
	size int
	name byte // 'S' / 'M' / 'L'
}

func main() {
	allocs := []func() any{
		func() any { return new(Small) },
		func() any { return new(Medium) },
		func() any { return new(Large) },
	}
	live := make([]any, 50000) // keep refs so GC can't reclaim, and so we know each type
	for i := range live {
		live[i] = allocs[rand.Intn(len(allocs))]()
	}

	for pass := 0; pass < 2; pass++ {
		if pass == 1 {
			for i := range live {
				if i%10 != 0 { // free 90% of the objects
					live[i] = nil
				}
			}
			runtime.GC()         // but the survivors can't be moved,
			debug.FreeOSMemory() // so nothing closes up and nothing goes back
		} else {
			runtime.GC()
		}
		objs := make([]object, 0, len(live))
		for _, o := range live {
			if o == nil {
				continue
			}
			t := reflect.TypeOf(o).Elem()
			objs = append(objs, object{reflect.ValueOf(o).Pointer(), int(t.Size()), t.Name()[0]})
		}
		slices.SortFunc(objs, func(a, b object) int { return cmp.Compare(a.addr, b.addr) })

		fmt.Printf("\n=== pass %d (%d live, base 0x%x) ===\n", pass, len(objs), objs[0].addr)
		draw(window(objs, 16*1024)) // first 16 KiB of the heap, same region both passes
		stats(objs)
	}
	runtime.KeepAlive(live)
}

// window returns the objects sitting in the first n bytes of the heap.
func window(objs []object, n uintptr) []object {
	end := objs[0].addr + n
	for i, o := range objs {
		if o.addr >= end {
			return objs[:i]
		}
	}
	return objs
}

func stats(objs []object) {
	const span = 8192 // Go's span/page granularity

	liveBytes := 0
	spans := map[uintptr]bool{}
	for _, o := range objs {
		liveBytes += o.size
		spans[o.addr&^(span-1)] = true
	}

	var m runtime.MemStats
	runtime.ReadMemStats(&m)
	kib := func(x uint64) float64 { return float64(x) / 1024 }

	fmt.Printf("  live data      %8.1f KiB\n", float64(liveBytes)/1024)
	fmt.Printf("  spans pinned   %8d      (%d if the survivors were packed)\n",
		len(spans), (liveBytes+span-1)/span)
	fmt.Printf("  runtime        HeapInuse %.1f KiB | HeapIdle %.1f KiB | HeapReleased %.1f KiB\n",
		kib(m.HeapInuse), kib(m.HeapIdle), kib(m.HeapReleased))
}

func draw(objs []object) {
	const cell, width = 32, 60

	last := objs[len(objs)-1]
	base := objs[0].addr
	grid := make([]byte, int((last.addr+uintptr(last.size)-base)/cell))
	for i := range grid {
		grid[i] = '.'
	}
	for _, o := range objs {
		c0 := int((o.addr - base) / cell)
		grid[c0] = o.name
		for k := 1; k < o.size/cell; k++ {
			grid[c0+k] = '-'
		}
	}

	prev := -1
	for off := 0; off < len(grid); off += width {
		row := grid[off:min(off+width, len(grid))]
		if len(bytes.Trim(row, ".")) == 0 { // row holds none of our objects
			continue
		}
		if prev >= 0 && off != prev+width {
			fmt.Println("             ...")
		}
		fmt.Printf("0x%09x  %s\n", base+uintptr(off)*cell, row)
		prev = off
	}
}

heapwalk_free.go

Run it.

$ go run heapwalk_free.go

=== pass 0 (50000 live, base 0x10a2674ac000) ===
0x10a2674ac000  L---L---L---L---L---L---L---L---L---L---L---L---L---L---L---
0x10a2674ac780  L---L---L---L---L---L---L---L---L---L---L---L---L---L---L---
0x10a2674acf00  L---L---L---L---L---L---L---L---L---L---L---L---L---L---L---
0x10a2674ad680  L---L---L---L---L---L---L---L---L---L---L---L---L---L---L---
0x10a2674ade00  L---L---L---L---....L---L---L---L---L---L---L---L---L---L---
0x10a2674ae580  L---L---L---L---L---L---L---L---L---L---L---L---L---L---L---
0x10a2674aed00  L---L---L---L---L---L---L---L---L---L---L---L---L---L---L---
0x10a2674af480  L---L---L---L---L---L---L---L---L---L---L---L---L---L---L---
0x10a2674afc00  L---L---L---L---L---L---L---
  live data        3639.9 KiB
  spans pinned        464      (455 if the survivors were packed)
  runtime        HeapInuse 6320.0 KiB | HeapIdle 5520.0 KiB | HeapReleased 5488.0 KiB

=== pass 1 (5000 live, base 0x10a2674acc00) ===
0x10a2674acc00  L---..........................L---............................
0x10a2674ad380  ................L---............................L---........
0x10a2674adb00  L---....................................................L---
0x10a2674ae280  ................L---........L---............L---............
0x10a2674aea00  ............L---............................................
0x10a2674af180  ........L---............L---....................L---........
0x10a2674af900  ............................L---
  live data         364.6 KiB
  spans pinned        463      (46 if the survivors were packed)
  runtime        HeapInuse 6320.0 KiB | HeapIdle 5552.0 KiB | HeapReleased 5512.0 KiB

The interesting thing is that (in simple scenarios) we can “move” (well, copy) objects ourselves to avoid fragmentation and to recover memory.

package main

import (
	"bytes"
	"cmp"
	"fmt"
	"math/rand"
	"reflect"
	"runtime"
	"runtime/debug"
	"slices"
	"unsafe"
)

type (
	Small  struct{ _ [32]byte }  // 32 bytes
	Medium struct{ _ [64]byte }  // 64 bytes
	Large  struct{ _ [128]byte } // 128 bytes
)

type object struct {
	addr uintptr
	size int
	name byte // 'S' / 'M' / 'L'
}

func main() {
	allocs := []func() any{
		func() any { return new(Small) },
		func() any { return new(Medium) },
		func() any { return new(Large) },
	}
	live := make([]any, 50000) // keep refs so GC can't reclaim, and so we know each type
	for i := range live {
		live[i] = allocs[rand.Intn(len(allocs))]()
	}
	// pass 2 moves the survivors in here, by hand
	var packedS []Small
	var packedM []Medium
	var packedL []Large

	for pass := 0; pass < 3; pass++ {
		switch pass {
		case 1:
			for i := range live {
				if i%10 != 0 { // free 90% of the objects
					live[i] = nil
				}
			}
		case 2:
			for i, o := range live {
				switch v := o.(type) {
				case *Small:
					packedS = append(packedS, *v)
				case *Medium:
					packedM = append(packedM, *v)
				case *Large:
					packedL = append(packedL, *v)
				}
				live[i] = nil
			}
		}
		runtime.GC()
		debug.FreeOSMemory()

		objs := make([]object, 0, len(live))
		for i := range packedS {
			objs = append(objs, object{uintptr(unsafe.Pointer(&packedS[i])), 32, 'S'})
		}
		for i := range packedM {
			objs = append(objs, object{uintptr(unsafe.Pointer(&packedM[i])), 64, 'M'})
		}
		for i := range packedL {
			objs = append(objs, object{uintptr(unsafe.Pointer(&packedL[i])), 128, 'L'})
		}
		for _, o := range live {
			if o == nil {
				continue
			}
			t := reflect.TypeOf(o).Elem()
			objs = append(objs, object{reflect.ValueOf(o).Pointer(), int(t.Size()), t.Name()[0]})
		}
		slices.SortFunc(objs, func(a, b object) int { return cmp.Compare(a.addr, b.addr) })

		fmt.Printf("\n=== pass %d (%d live, base 0x%x) ===\n", pass, len(objs), objs[0].addr)
		draw(window(objs, 16*1024)) // first 16 KiB of the survivors' range
		stats(objs)
	}
	runtime.KeepAlive(live)
	runtime.KeepAlive(packedS)
	runtime.KeepAlive(packedM)
	runtime.KeepAlive(packedL)
}

// window returns the objects sitting in the first n bytes of the range.
func window(objs []object, n uintptr) []object {
	end := objs[0].addr + n
	for i, o := range objs {
		if o.addr >= end {
			return objs[:i]
		}
	}
	return objs
}

func stats(objs []object) {
	const span = 8192 // Go's span/page granularity

	liveBytes := 0
	spans := map[uintptr]bool{}
	for _, o := range objs {
		liveBytes += o.size
		spans[o.addr&^(span-1)] = true
	}

	var m runtime.MemStats
	runtime.ReadMemStats(&m)
	kib := func(x uint64) float64 { return float64(x) / 1024 }

	fmt.Printf("  live data      %8.1f KiB\n", float64(liveBytes)/1024)
	fmt.Printf("  spans pinned   %8d      (%d if the survivors were packed)\n",
		len(spans), (liveBytes+span-1)/span)
	fmt.Printf("  runtime        HeapInuse %.1f KiB | HeapIdle %.1f KiB | HeapReleased %.1f KiB\n",
		kib(m.HeapInuse), kib(m.HeapIdle), kib(m.HeapReleased))
}

func draw(objs []object) {
	const cell, width = 32, 60

	last := objs[len(objs)-1]
	base := objs[0].addr
	grid := make([]byte, int((last.addr+uintptr(last.size)-base)/cell))
	for i := range grid {
		grid[i] = '.'
	}
	for _, o := range objs {
		c0 := int((o.addr - base) / cell)
		grid[c0] = o.name
		for k := 1; k < o.size/cell; k++ {
			grid[c0+k] = '-'
		}
	}

	prev := -1
	for off := 0; off < len(grid); off += width {
		row := grid[off:min(off+width, len(grid))]
		if len(bytes.Trim(row, ".")) == 0 { // row holds none of our objects
			continue
		}
		if prev >= 0 && off != prev+width {
			fmt.Println("             ...")
		}
		fmt.Printf("0x%09x  %s\n", base+uintptr(off)*cell, row)
		prev = off
	}
}

heapwalk_free_manual.go

And run it.

$ go run heapwalk_free_manual.go

=== pass 0 (50000 live, base 0xa615561a0c0) ===
0xa615561a0c0  M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-
0xa615561a840  M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-
0xa615561afc0  M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-
0xa615561b740  M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-
0xa615561bec0  M-M-M-......................................................
             ...
0xa615561dcc0  ............................SSSS
  live data        3654.9 KiB
  spans pinned        465      (457 if the survivors were packed)
  runtime        HeapInuse 6224.0 KiB | HeapIdle 5680.0 KiB | HeapReleased 5632.0 KiB

=== pass 1 (5000 live, base 0xa615561a0c0) ===
0xa615561a0c0  M-..........................M-......M-......................
0xa615561a840  ..M-......M-..M-......M-..........M-........................
0xa615561afc0  ..............M-..................................M-........
0xa615561b740  ....M-
  live data         373.0 KiB
  spans pinned        465      (47 if the survivors were packed)
  runtime        HeapInuse 6232.0 KiB | HeapIdle 5672.0 KiB | HeapReleased 5672.0 KiB

=== pass 2 (5000 live, base 0xa6155710000) ===
0xa6155710000  M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-
0xa6155710780  M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-
0xa6155710f00  M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-
0xa6155711680  M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-
0xa6155711e00  M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-
0xa6155712580  M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-
0xa6155712d00  M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-
0xa6155713480  M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-
0xa6155713c00  M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-
  live data         373.0 KiB
  spans pinned         48      (47 if the survivors were packed)
  runtime        HeapInuse 2768.0 KiB | HeapIdle 9104.0 KiB | HeapReleased 9040.0 KiB

Which is kind of neat.

Let’s compare it though to if we take our C# sample and clean up 90% of the objects.

#nullable enable
using System;
using System.Collections.Generic;
using System.Runtime.CompilerServices;

var allocs = new Func<object>[] { () => new Small(), () => new Medium(), () => new Large() };

// A reference on 64-bit .NET is a plain 8-byte pointer, so reinterpreting one
// with Unsafe.As gives the object's address. Object sizes are measured from
// the heap: allocate several, the smallest gap between consecutive addresses
// is the (aligned) object size, header included.
var size = new Dictionary<Type, int>();
foreach (var make in allocs)
{
    var keep = new object[16];
    var a = new nint[keep.Length];
    for (int i = 0; i < keep.Length; i++) keep[i] = make();
    for (int i = 0; i < keep.Length; i++) a[i] = Unsafe.As<object, nint>(ref keep[i]);
    Array.Sort(a);
    nint best = nint.MaxValue;
    for (int i = 1; i < a.Length; i++)
        if (a[i] - a[i - 1] > 0 && a[i] - a[i - 1] < best) best = a[i] - a[i - 1];
    size[keep[0].GetType()] = (int)best;
}

var live = new object?[50_000]; // the only refs to our objects: nulling one frees it
var rnd = new Random();
for (int i = 0; i < live.Length; i++) live[i] = allocs[rnd.Next(allocs.Length)]();

for (int pass = 0; pass < 2; pass++)
{
    if (pass == 1)
        for (int i = 0; i < live.Length; i++)
            if (i % 10 != 0) live[i] = null; // free 90% of the objects

    // Full compacting collection that also returns freed memory to the OS.
    GC.Collect(GC.MaxGeneration, GCCollectionMode.Aggressive);

    int n = 0;
    foreach (var o in live) if (o != null) n++;

    var addrs = new nint[n];
    GC.TryStartNoGCRegion(1 << 20); // addresses are only valid until the next GC
    for (int i = 0, j = 0; i < live.Length; i++)
        if (live[i] != null) addrs[j++] = Unsafe.As<object?, nint>(ref live[i]);
    GC.EndNoGCRegion();

    var objs = new (nint Addr, int Size, char Name)[n];
    for (int i = 0, j = 0; i < live.Length; i++)
        if (live[i] != null)
        {
            objs[j] = (addrs[j], size[live[i]!.GetType()], live[i]!.GetType().Name[0]);
            j++;
        }
    Array.Sort(objs, (x, y) => x.Addr.CompareTo(y.Addr));

    int w = 0; // the objects sitting in the first 16 KiB of the range
    while (w < objs.Length && objs[w].Addr < objs[0].Addr + 16 * 1024) w++;

    Console.WriteLine($"\n=== pass {pass} ({n} live, base 0x{(long)objs[0].Addr:x}) ===");
    Draw(objs[..w]);
    Stats(objs);
}
GC.KeepAlive(live); // keep `live` from seeming to fall out of scope

static void Draw((nint Addr, int Size, char Name)[] objs)
{
    const int cell = 32, width = 60; // cell = the smallest object's size

    var last = objs[^1];
    nint b = objs[0].Addr;
    var grid = new char[(last.Addr + last.Size - b) / cell];
    Array.Fill(grid, '.');
    foreach (var o in objs)
    {
        int c0 = (int)((o.Addr - b) / cell);
        grid[c0] = o.Name;
        for (int k = 1; k < o.Size / cell; k++) grid[c0 + k] = '-';
    }

    int prev = -1;
    for (int off = 0; off < grid.Length; off += width)
    {
        var row = new string(grid, off, Math.Min(width, grid.Length - off));
        if (row.Trim('.').Length == 0) continue; // row holds none of our objects
        if (prev >= 0 && off != prev + width) Console.WriteLine("             ...");
        Console.WriteLine($"0x{(long)b + (long)off * cell:x9}  {row}");
        prev = off;
    }
}

static void Stats((nint Addr, int Size, char Name)[] objs)
{
    const int chunk = 8192; // 8KiB = Go's span size, for an apples-to-apples density measure

    long liveBytes = 0;
    var chunks = new HashSet<nint>();
    foreach (var o in objs)
    {
        liveBytes += o.Size;
        chunks.Add(o.Addr & ~(nint)(chunk - 1));
    }

    Console.WriteLine($"  live data      {liveBytes / 1024.0,8:f1} KiB");
    Console.WriteLine($"  8KiB chunks    {chunks.Count,8}      ({(liveBytes + chunk - 1) / chunk} if the survivors were packed)");
}

class Small  { public long a, b; }                                     // 32 bytes: 16-byte header + 2 longs
class Medium { public long a, b, c, d, e, f; }                         // 64 bytes
class Large  { public long a, b, c, d, e, f, g, h, i, j, k, l, m, n; } // 128 bytes

HeapWalkFree.cs

Compile and run it.

$ dotnet run HeapWalkFree.cs

=== pass 0 (50000 live, base 0x7e928000aad0) ===
0x7e928000aad0  M-M-L---M-M-SSL---M-L---M-M-L---M-L---M-M-SM-L---L---SM-M-M-
0x7e928000b250  SM-SL---L---L---L---SM-L---SM-SM-M-M-M-L---SSM-L---L---L---L
0x7e928000b9d0  ---M-SM-M-M-SSL---M-SSL---L---L---SM-M-M-M-SL---M-M-SL---M-L
0x7e928000c150  ---M-SL---M-L---L---L---L---M-L---SM-SM-L---M-L---L---L---L-
0x7e928000c8d0  --M-SL---M-M-SSSL---M-SL---SM-L---L---L---L---M-SM-M-M-M-M-L
0x7e928000d050  ---L---SL---L---M-SL---SSSL---M-L---M-SM-M-L---SL---L---SL--
0x7e928000d7d0  -.L---M-M-L---SSL---SL---L---M-SL---M-SM-L---M-M-SM-SM-M-SSS
0x7e928000df50  SL---M-M-M-M-SM-M-L---L---M-M-M-SL---L---M-SSM-L---L---M-M-S
0x7e928000e6d0  L---L---L---M-SM-SM-SM-M-L---L---
  live data        3646.1 KiB
  8KiB chunks         458      (456 if the survivors were packed)

=== pass 1 (5000 live, base 0x7e9282c0aa70) ===
0x7e9282c0aa70  M-M-L---L---M-M-SL---M-M-M-M-M-SL---SSSM-M-L---L---M-SL---SS
0x7e9282c0b1f0  M-L---M-L---M-L---L---L---SL---SSM-L---L---SSL---L---SM-SSL-
0x7e9282c0b970  --SSSM-L---SSSSSL---M-SL---SSM-L---M-M-L---SSM-SSL---M-M-L--
0x7e9282c0c0f0  -M-SSM-SM-M-SSM-M-M-M-L---L---L---SM-M-SSL---L---M-M-M-M-L--
0x7e9282c0c870  -SM-L---L---M-L---M-SL---SSM-L---L---SL---L---M-L---L---SM-S
0x7e9282c0cff0  M-L---L---M-SL---SL---SSM-M-SL---L---M-L---SSSSL---SM-M-SSM-
0x7e9282c0d770  M-L---L---L---M-M-L---L---SL---SM-SSM-SM-L---SL---M-M-SL---L
0x7e9282c0def0  ---L---SM-SM-SM-SM-M-L---L---L---L---SL---SM-SSM-SM-SM-L---L
0x7e9282c0e670  ---L---M-M-M-M-SM-SSL---SL---L---
  live data        364.6 KiB
  8KiB chunks          47      (47 if the survivors were packed)

Which looks pretty good!

The gigantic wildfire ravaging southwest France has created an extremely dangerous "pyrocumulonimbus" -- a phenomenon never before seen in the country -- the spokesman for the country's National Firefighters Federation of France (FNSPF) told AFP.

The main wildfire burning in France's southwest has created a 'fire cloud', which generates its own winds © ROMAIN PERROCHEAU / AFP/File

Such a formation, also called a cumulonimbus flammagenitus cloud, can be seen during volcanic eruptions -- or, as in this case, when a wildfire roasts the air near the ground, causing it to rise quickly.

The rising air cools and becomes a cloud, or fire cloud: a turbulent mass that can generate its own winds that worsen fires. If it becomes very large it can even generate lightning.

Hitherto, they have only really been seen in wildfire-prone Australia and North America. NASA calls them the "fire-breathing dragon of clouds".

What causes them?

FNSPF spokesman Lieutenant Colonel Eric Brocardi:

"The rise in ground temperature at the heart of the blaze is so extreme that the intense heat surges upward in a column, meeting cooler air masses at the top. This contact creates a pyrocumulonimbus -- or fire cloud -- though unfortunately one devoid of moisture. This smoke cloud generates its own weather system, causing anything in its path to spontaneously combust.

"Lightning forms within the cloud, striking the embers on the ground and further fuelling the fire ... It produces a low, rumbling sound.

"While this type of fire occurs regularly in Canada and Australia, it is unprecedented in France."

Can it be brought under control?

"We are dealing with a so-called 'convective' fire that creates its own winds -- winds that constantly shift direction, unlike a standard fire that spreads in a conical pattern. The result is that it spreads in every direction and develops multiple fire fronts, making the situation completely unpredictable.

"We don't know how the blaze will spread; the fire front is constantly shifting. We cannot fight it directly. It's a David-versus-Goliath scenario: the idea is that, at some point, we'll find a weak spot and strike there.

"Salvation will come either from the sky -- in the form of rain, though it would need to pour heavily for three days -- or by finding a way to steer the fire toward a place where it will die out on its own. The sea, for example."

Do the firefighters have any hope?

"We have reached a point of 'operational impossibility' that requires us to accept the need for strategic retreat -- to understand and acknowledge a natural force beyond our control.

"It calls for ... knowing what can be done and, above all, what cannot. The situation on the ground is defensive rather than offensive; it is a war-like scenario where we are under constant bombardment and must get everyone to safety.

"We realise we are facing a situation where we are powerless. Yet, that powerlessness leaves no room for despair. It is certain that all the efforts being made will yield results. It is out of the question for the firefighters to feel that their work on the ground is in vain.

"Every single action -- defending critical sites, rescuing animals -- breathes new life into their efforts, driven by their fierce determination. There is a support force ready to relieve them.

"However, not all firefighters from across France can simply converge here, because the risk exists everywhere. We have to hold the line."

(Interviewed by AFP journalist Juliette Collen)

Colossus cracked Germany’s encrypted communications

Operators Dorothy Du Boisson and Elsie Booker working Colossus, the world’s first large-scale programmable electronic digital computer, at Bletchley Park in England.

Geopix/Alamy

One summer day in 1941, a British radio operator was monitoring German military frequencies and heard something unexpected in her headphones. A later report called it “strange new music.” Sounding unlike the familiar Morse dit-dit-dah of enciphered messages sent over the German Enigma network, the “new music” was a rhythmic warble of binary teletype code being transmitted at high speed.

Germany’s wartime engineers had developed a radically new encryption and transmission system. It was way more advanced than Enigma, which was patented in 1920.

To break the complex new cipher, engineer Tommy Flowers built Colossus, the world’s first large-scale programmable electronic digital computer. Flowers previously built Enigma-related codebreaking equipment for Alan Turing, the British mathematician.

Colossus was installed in the British codebreaking headquarters at Bletchley Park, about 80 kilometers from London. The room-size machine weighed around a tonne.

The computer is being commemorated as an IEEE Milestone. The dedication ceremony is scheduled to be held 29 September at Bletchley Park.

Decrypting Germany’s strange new music

Britain’s top codebreakers were quickly all over the new “music” being picked up by the intercept stations. Identifying it as encrypted teletype code was the easy part. The real problem was figuring out how the encryption machine worked. Its manufacturer was discovered at the end of the war: Berlin engineering firm C. Lorenz.

But in 1941, the Lorenz machine was just a black box to the British. They codenamed it “Tunny,” a British term for tuna fish. The Enigma breakers had set a precedent for using piscine codenames such as Dolphin, Lumpsucker, and Porpoise.

Enigma had three or four encrypting wheels. The codebreakers guessed that the Tunny machine also used a system of rotating wheels to encrypt messages. An important clue was that all the intercepted messages shared a curious feature: Each began with an uncoded list of 12 common German names, including Anton, Bertha, Conrad, and Dora. The codebreakers guessed that Tunny had 12 wheels and that the 12 names and their order somehow told the receiving operator which combination they should twist the wheels to before decrypting the message.

Then the British had an extraordinary piece of good fortune. John Tiltman, head of the research section at Bletchley Park, started analyzing a pair of intercepted messages, each around 1,200 characters long. Unusually, both began with the same sequence of names. The second message turned out to be a retype of the first, with minor differences in punctuation, a few abbreviations, and other small divergences. Tiltman managed to decrypt the two ciphertexts using a mixture of educated guesswork and intuition. The resulting 1,200 or so pairings of ciphertext and plaintext characters proved to be enough information to deduce the workings of the Tunny machine.

That was thanks to Bill Tutte, a quiet young codebreaker who spent weeks poring over the pairings. One day, he shyly announced to his superiors how Tunny worked. His description was uncannily accurate.

The next step in the Tunny machine’s downfall was achieved by Turing, fresh from his successes against Enigma.

Knowledge of how the Tunny machine worked was not enough to decrypt the messages. Codebreakers also required detailed information about how the wheels of the sender’s machine had been set up. There were adjustable pins around the circumference of each wheel: In one of its two possible positions, a pin would contribute a 1 to the encryption process, and in the other, a 0. The pins were reset from time to time.

The codebreakers also needed to know the wheels’ positions at the start of the message—which the German operators gave away in the list of 12 names.

Turing invented a tricky method, called “Turingery,” that enabled codebreakers to deduce the positions of the pins from nothing but intercepted ciphertext.

After that, the message could be decrypted, using the list of names and a British replica of the Tunny machine.

The basis of Turingery was a procedure that Turing introduced, called “delta-ing” (from the Greek letter delta). Also known as “differencing,” the process used “sideways” addition: To delta the four letters ABCD, you add (at the bit level) A to B, B to C, and C to D. Turing used delta-ing to reveal information about the wheels.

Tunny messages, often signed by Adolph Hitler himself, turned out to be pure gold for the Allies. The machine was used in Berlin by the Armed Forces High Command to communicate with front-line generals directing the war in the Eastern and Western theaters.

Once the system was broken, the Allies could eavesdrop on lengthy back-and-forth communications between the architects of Germany’s battle plans.

Turingery was the codebreakers’ only weapon against Tunny for a year, during which they managed to decrypt 1.5 million letters of ciphertext.

But everything changed when those helpful lists of names at the start of each message disappeared.

At the same time, Turingery was becoming less effective. Turing’s method depended on the German sender mistakenly using the same wheel settings to encrypt two differing messages. As security tightened across the Tunny network, the blunder became rarer.

Fortunately, Tutte had been at work devising a different decryption method, based on Turing’s delta-ing but taking a novel approach.

Building the Colossus computer

Tutte had found a way of deducing wheel information from ciphertext, with no list of names or blunders by the German operators required. His method made use of statistical properties of the Tunny machine itself.

At first, it wasn’t clear how to apply his statistical method, however. The Tunny breakers worked by hand. Applying Turingery to a message was like solving a monster Sudoku or crossword puzzle.

Tutte’s statistical method required scads of routine binary math, as well as a colossal amount of counting long binary sequences. If the process were done by hand, one message could take months to decrypt. What was needed was a machine to automate the process.

Black and white portrait of a man with short gelled hair in a suit jacket, tie and eyeglasses. Engineer Thomas H. Flowers developed Colossus to break a complex new German cipher.Pictorial Press/Alamy

The first plan was to build a machine from electromagnetic relays, adding a couple of dozen vacuum tubes to speed up the counting. Electronic tubes were much faster than electromagnetic relays, which had slow-moving metal components. Problems with the circuit design bedeviled the machine’s relay-based logic unit, however.

Flowers was recommended by Turing and brought in to troubleshoot. He was on loan to Bletchley Park from the Post Office Research Station in London, where he had spent the prewar years designing experimental switching equipment involving thousands of vacuum tubes.

At the time, it was commonly believed that tubes could not be used in large numbers because each one contained a hot filament. This meant tubes were prone to sudden death. In a large installation, it would not be long before one tube blew and things stopped working properly.

Flowers discovered that switching tubes on and off stressed them, but leaving them on continuously made them more reliable than relays. He offered to build Bletchley Park a high-speed, all-electronic machine containing around 2,000 tubes.

Bletchley Park’s advisors rejected the idea, convinced that such a machine would never work reliably. But Flowers, confident of his proposed design, retreated to his London laboratory and quietly built the electronic machine that he believed the codebreakers needed. He and his small team of engineers worked day and night for 10 months to create Colossus.

In January 1944 some of his engineers showed up at Bletchley Park with the world’s first large-scale programmable electronic digital computer packed onto the back of a truck. Colossus was reassembled and functional in about two weeks, and it notched up its first German message on 5 February 1944.

The machine read the input—Tunny ciphertext—photoelectrically from a large loop of punched paper tape. The output—information about the wheels—went to a primitive printer that Flowers’ engineers had created from a manual typewriter, fitting relays to automate the keys. Once Colossus had cracked enough of the Tunny machine’s wheels, the information was passed on to the hand-breakers, who took over.

The codebreakers were astonished by Colossus.

“I don’t think they understood very clearly what I was proposing until they actually had the machine,” Flowers said in a 1977 interview. “They just couldn’t believe it!”

Colossus was described in almost loving terms in a since-declassified report written at Bletchley Park in 1945:

It is regretted that it is not possible to give an adequate idea of the fascination of a Colossus at work: its sheer bulk and apparent complexity; the fantastic speed of thin paper tape round the glittering pulleys; the childish pleasure of not-not, span, print main heading and other gadgets; the wizardry of purely mechanical decoding letter by letter (one novice thought she was being hoaxed); the uncanny action of the typewriter in printing the correct scores without and beyond human aid; the stepping of display; periods of eager expectation culminating in the sudden appearance of the longed-for score; and the strange rhythms characterizing every type of run: the stately break-in, the erratic short run, the regularity of wheel-breaking, the stolid rectangle interrupted by the wild leaps of the carriage-return, the frantic chatter of a motor run, even the ludicrous frenzy of hosts of bogus scores.

The demand for more Colossi

Bletchley Park’s managers, no longer leery of Flowers’s ideas, soon wanted additional Colossi. He finished building the second one in June 1944, days before D-Day and the Allied invasion of Europe. With 2,400 vacuum tubes—around 800 more than in Colossus I—Colossus II processed Tunny messages at an eye-watering speed of 25,000 characters per second.

Its maximized timing-pulse rate was not far short of the performance of the first Intel microprocessor chip from the 1970s, more than 30 years later.

Flowers conceded that “Colossus bore about as much resemblance to a modern computer as Stephenson’s [1829] Rocket locomotive did to the Royal Scot,” a state-of-the-art 20th-century train operating between London and Glasgow. But he emphasized that, nevertheless, Colossus “embodied all the basic features of a modern computer.” In Colossus, Flowers had pioneered clock pulses, bit-stream generators, control circuits, loops, counters, shift registers, interrupts, parallel processing, and more.

As the Allies slowly fought their way toward Germany, the Colossi poured out wheel information, and the codebreakers provided the military with an unparalleled view of German strategies, strengths, weaknesses, and tactical intentions.

Even with that mass of detailed intelligence, it took the Allies almost a year to move from Northern France to the German heartland. No one can say for sure how much longer the fighting would have lasted if the intelligence breakthrough had not occurred. But if Colossus and the codebreakers shortened the war even by only six months, the number of lives saved was in the millions.

There were 10 Colossi at Bletchley Park by the end of the war, housed in two vast, steel-frame, bombproof buildings, running day and night. Although concealed behind a thick veil of secrecy, Bletchley Park accommodated the world’s first electronic computing facility. It was directed by Max Newman, the mathematician who mentored Turing in prewar Cambridge.

I don’t think they understood very clearly what I was proposing until they actually had the machine. They just couldn’t believe it!”—Tommy Flowers

When the fighting ended, authorities decided that ultrasecrecy must be maintained, and orders were issued to break up the Colossi. Only two were spared.

“All that was left were the deep holes in the floor where the machines had stood,” Colossus operator Dorothy Du Boisson recalled in an interview for the book Colossus: The Secrets of Bletchley Park’s Codebreaking Computers. Norman Thurlow, one of Flowers’s engineers who was also interviewed, remembered being told in a staff memo that if the secrecy was ever lifted, he and his colleagues might be able to tell their grandchildren about Colossus and “the tapes that span on silver wheels.”

IEEE Milestone dedication at Bletchley Park

The Milestone plaque recognizing Colossus is to be displayed outside Block H at Bletchley Park, near Milton Keynes, England.

The plaque is to read:

Six Colossus codebreaking computers operated in this building in 1944–1945. Designed by Thomas H. Flowers of the British Post Office, they enabled deciphering of encrypted radio messages transmitted between German commands across occupied Europe, North Africa, and the Soviet Union. The resulting military intelligence saved countless lives and helped shorten World War II. As the first successful large-scale application of digital electronics to computing, Colossus anticipated subsequent computer developments.

The IEEE United Kingdom and Ireland Section sponsored the nomination.

Reviewed by the IEEE History Committee and awarded by the IEEE Board of Directors, IEEE Milestones recognize outstanding technical developments around the world that are at least 25 years old. The Milestone program is administered by the IEEE history and heritage group.

To learn more about historical figures in engineering, IEEE Milestones, and IEEE History Center programs and events, check out our IEEE Tech History collection. IEEE Spectrum also covers aspects of tech history.

Back in December we called out Google for filing a DMCA 1201 lawsuit over companies scraping Google’s results. Almost everything about the lawsuit seemed problematic, not the least of which is that Google’s entire business was built on scraping the web. To sue another company for scraping Google just felt… obnoxious. And now a judge has dismissed the lawsuit, though leaving it open for Google to refile.

Some background: now that we’re in the age of AI, access to all kinds of data has become more precious, which means we’re seeing more and more attempts to put a toll booth on parts of the open web, primarily aimed at AI companies. But the rest of us get locked out along the way. SerpAPI is one of the players in the space which (as its name implies) basically tries to create an unauthorized API for search engine result pages.

Last fall, Reddit sued SerpAPI and some others (including search AI company Perplexity), claiming that because SerpAPI was allowing others (like Perplexity) to access Reddit content via its scrape of Google, it was violating the DMCA’s anti-circumvention (DMCA 1201) clause. We found the whole thing to be an attack on the principles of the open web. It really seemed weird. Reddit had no copyright interest in its users’ posts (the users hold the copyright) and SerpAPI was scraping Google, not Reddit. Reddit has an API deal with Google, but none of the parties being sued were parties to that deal. The whole thing was just “we don’t like that this is happening, so we’re suing.”

Google’s case came a few months later and was quite similar, focused on SerpAPI. And while at least in this case (unlike Reddit) they could point out that SerpAPI was scraping their own site, it still makes no sense to claim that scraping an open website can be a 1201 anti-circumvention violation, no matter what “technological protection measures” you throw up to try to block scraping. The Reddit case continues to move forward with the defendants filing motions to dismiss, but the Google case has lapped them a bit, with the judge already dismissing the complaint, and pointing out (correctly!) that Google has no legitimate copyright claim to make here.

While SerpAPI tried a variety of different ways to kill the lawsuit, what seemed to stick is that Google was clearly stretching the way the DMCA 1201 is supposed to work. Remember, 1201 is the “anti-circumvention” part of the DMCA, and was initially written to protect DRM so that if people broke DRM (or even talked about how to break DRM) they could still be held liable for copyright infringement just for the act of circumventing the “technological protection measure.” This very broad and poorly worded law has created huge messes in its wake, including blatant abuses like companies arguing that you can’t use third-party printer ink or third-party garage door openers because of flimsy “technological protection measures” put into those devices, even though the underlying circumvention had nothing to do with copyright.

The court also looks at one of those earlier cases (regarding Lexmark’s printers), but concludes it doesn’t apply here — long story, not worth the detail, except to note that the precedent that mattered against Lexmark came from trademark law, not the DMCA, even though Lexmark had also tried (and failed) to use Section 1201 itself.

However, SerpAPI (rightly) also pointed out that Google is overclaiming what “SearchGuard” — the “technological protection measure” — actually protects here. As the court explains it, SearchGuard is basically a kind of CAPTCHA:

SearchGuard works by sending a JavaScript “challenge” to search queries that Google receives from unrecognized sources to confirm that they come from real users as opposed to automated software. Id. ¶ 29. Google’s computer system transmits JavaScript code that calls upon the user’s browser to send Google a “solve” for the challenge, i.e., to send Google specific information regarding the browser and user generating the request. Id. ¶ 29. For human users, the “solve” is relatively straightforward; their browsers run the JavaScript code and send back the required information seamlessly, without disrupting the user experience. Id. ¶ 29. However, automated systems that submit automated queries at a massive scale typically cannot solve the SearchGuard challenge. Id. As a result, SearchGuard denies them access to Google’s Search results.

But, as SerpAPI highlighted, SearchGuard has little to do with copyright. And that, at least, gets the court’s attention:

SerpApi contends that Google’s claims under the DMCA are subject to dismissal because SearchGuard is designed and functions to control access to and prevent the scraping of Google Search results regardless of whether they contain a copyrighted component, and because SearchGuard is not reasonably tailored to control access only with respect to any copyrighted component that may be included in Google Search results.

The Court agrees with SerpApi in part. To the extent that Google Search results do not contain any copyrighted content, SearchGuard cannot be said to effectively control access to a work protected under the Copyright Act. Here, Google alleges that Google Search results are compilations of publicly-available information that Google obtains from the internet and organizes for presentation to users on* google.com based on relevance. See Compl. ¶¶ 13, 14, 27. SearchGuard controls access to Google Search results because its “purpose” is “to prevent unauthorized third parties from automatically accessing Google’s Search results” to scrape them, as such scraping activities impose a “deadweight loss” on Google. See id. ¶¶ 24, 26-27, 29. However, Google does not allege that google.com or the Google Search results displayed therein are protected under the Copyright Act. Importantly, Google alleges that Google Search results are “often” accompanied by a “Knowledge Panel” that may contain some copyrighted content that Google licenses from third parties, such as copyrighted images. Google does not allege that the “Knowledge Panel” is always included in Google Search results, or that the Knowledge Panel, if included in the Search results, always contains copyrighted content. See id. ¶¶ 14-16. Accordingly, Google’s allegations indicate a mix of content, some with copyrighted material and others without.

And that cuts against Google’s argument here:

Thus, because the DMCA does not apply where the work controlled by a technological measure is not protected under the Copyright Act, Google’s claims under 17 U.S.C. § 1201(a)(1)(A) and 17 U.S.C. § 1201(a)(2) are subject to dismissal as a matter of law to the extent that they are premised on instances where SearchGuard controls access to Google Search results that do not contain any copyrighted content.

Even more damning for Google is that when it’s using SearchGuard, that has literally nothing to do with “effectively controlling access to a [copyright-protected] work.” And that’s the entire point of 1201.

SerpApi argues that Google’s claims under the DMCA fail because it does not allege that it implemented SearchGuard to protect a copyrighted work with the “authority of the copyright owner” as required under 17 U.S.C. § 1201(a)(3)(B)….

The Court agrees. The plain language of 17 U.S.C. § 1201(a)(3)(B) makes clear that, for a technological measure to “effectively control[] access to a work” it must, among other things, “require[] the application of information, or a process or a treatment, with the authority of the copyright owner, to gain access to the work.” See 17 U.S.C. § 1201(a)(3)(B). The Ninth Circuit has interpreted the “with the authority of the copyright owner” element as requiring a plaintiff to allege and later prove that the technological measure in question was implemented and functioned with the authority of the copyright owner.

Google tried to argue that it somehow has the support of copyright holders to protect their work with SearchGuard, but the court is not impressed.

Google’s arguments do not compel a different conclusion. It contends that it is not required to allege facts indicating that it had the authority of the copyright owners to implement SearchGuard because the phrase “with the authority of the copyright owner” defines who may circumvent a technological measure to gain access to protected work and does not define who may deploy a technological measure to control access to a protected work…. This argument is unavailing. Google’s authorities interpret a different provision of the DMCA, namely 17 U.S.C. § 1201(a)(3)(A), which defines what it means to “circumvent a technological measure.” See Disney Enters., Inc. v. VidAngel, Inc., 869 F.3d 848, 863 (9th Cir. 2017) (“Section 1201(a)(3)(A) exempts from circumvention liability only those whom a copyright owner authorizes to circumvent an access control measure, not those whom a copyright owner authorizes to access the work.”) (citation and internal quotation marks omitted); Universal City Studios, Inc. v. Corley, 273 F.3d 429, 444 (2d Cir. 2001) (“[S]ubsection 1201(a)(3)(A) frees an individual to traffic in encryption technology designed or marketed to circumvent an encryption measure if the owner of the material protected by the encryption measure authorizes that circumvention.”). These authorities do not address the issue here, which is whether a technological measure must function “with the authority of the copyright owner” in order to “effectively control[] access to a work” under 17 U.S.C. § 1201(a)(3)(B).

Some of SerpAPI’s other arguments fail, but for now all the DMCA claims are dismissed, though Google can (and almost certainly will) refile regarding some more narrow claims. Specifically, Google cannot file claims regarding search results for which it does not hold the copyright, but could file more narrow claims regarding content where it does (such as the Knowledge Panel). That’s much more limited, and about the only reason to keep the case going is to be a nuisance to SerpAPI.

That might be worth it to Google, which really seems to dislike SerpAPI being out there and scraping their results. But it would be a much narrower case, and (in theory) SerpAPI could simply change its scraping to avoid Google-produced content. Either way, all of this remains quite silly. Google’s entire business was built on scraping the web. Suing someone else for scraping Google sure feels like pulling up the open internet ladder up after themselves.

SerpAPI’s comments on the dismissal make this point explicitly:

We’re pleased that the court rejected Google’s attempts to expand the DMCA to assert control over access to public pages. The internet’s founding principle – open access to usable information – is essential to driving innovation and ensuring everyone benefits from the promise of data. SerpApi will continue supporting developers, AI companies, researchers, and businesses that rely on access to public search information.

One would hope that this initial dismissal from the court gets the company to rethink this anti-open-internet strategy, but somehow I fear the old adage of “young companies innovate, old companies litigate” is starting to seep into Google.

Let's design the successor to email on top of HTTP, fixing the design flaws SMTP has been dragging along for 40 years. Piece by piece. The goal is not to replace the current mail system, heaven forbid!, but to learn, have fun and discover current technologies to swap out every element: sending, receiving, gateway, keys, and so on. The system will never talk to Gmail or any classic email provider: it only talks to itself. The only thing we are going to keep is the shape of the addresses, user@domain. Everything else gets reinvented.

Since it is a mail system over HTTP, a good name for the protocol would be HMTP: Hypertext Mail Transfer Protocol (the S for Simple in SMTP gives up its seat to the H for HTTP).

Now it's time to prepare the tech stack: HTTP, TLS, WebFinger, ActivityPub, Webmention, Ed25519, HPKE, sigchains, etc.

The bill of materials

This design doesn't invent a single technology: everything already exists.

Choosing HTTP is not just pragmatism. It solves, right out of the gate, several things any new protocol would have to build at some point:

• TLS, virtual hosting and SNI: for free.
• Status codes: the catalog a mail protocol needs already exists in HTTP. 202 Accepted (queued for delivery), 429 Too Many Requests + Retry-After (rate control), 404/410 (mailbox unknown/gone), 3xx (mailbox moved), 413 (too large). Even paid anti-spam has had a code reserved since 1997: 402 Payment Required.
• Existing infrastructure: proxies, load balancers, Nginx, libraries in every language. The protocol stops being a new server and becomes a convention over HTTP, like Webmention or Micropub.

Let's move to the next level: how do we discover a user, verify their identity, deliver the message, sign it and encrypt it, all over HTTP?

1. Discovery and delegation (a better MX)

Delegation is solved with a static document:

GET https://example.com/.well-known/hmtp/ana

{
  "inbox": "https://mail.migadu.example/hmtp/inbox/ana",
  "keys": { ... },
  "devices": [ ... ]
}

The inbox can live on another host: that is the MX record, but without touching DNS. A static blog on GitHub Pages can delegate its mail to a provider by serving a JSON file. And it allows something MX never did: per-user delegation (each mailbox of a domain on a different provider). We wouldn't even need to invent the route: WebFinger (RFC 7033) does exactly this, and Mastodon has already proven it scales.

2. Identity that survives key rotation

Identity cannot be a key (they get lost, they expire); it has to be something that keys back. A proposal with two anchors:

• Cryptographic continuity: the discovery document publishes the current key and a chain of rotations, where each new key is signed by the previous one. Anyone who knew you with key N can verify key N+1 without trusting anybody. It's a sigchain, what ATProto does with DIDs or what Keybase used to do.
• Domain control as a fallback: if you lose the key without a signed rotation (your laptop gets stolen), the domain declares a new key without a chain, with a mandatory announcement period (say 30 days) during which the servers that knew you show the warning "identity re-anchored by domain, not by signature".

However, domain-anchored identity has its own Achilles heel: a domain is not owned, it is rented. If you stop paying, it expires and someone registers it, the new owner publishes their keys in your .well-known and from that moment on they receive your mail and sign as you. Nobody can tell the legitimate heir from the squatter. It's the problem ATProto tries to solve by separating identity from the domain with DIDs, at the price of another piece of infrastructure. We know it's a real problem with a known solution. Now let's move on.

3. Delivery with a queue (store-and-forward)

Delivery is a POST to the recipient's inbox:

POST /hmtp/inbox/ana HTTP/1.1
Host: mail.migadu.example
Content-Type: application/hmtp+json

The key is not the request, it's who makes it. Your client doesn't deliver directly to the recipient, but to your own server (an authenticated POST to your outbox), and it is your server that queues, retries with exponential backoff and honors Retry-After. It's admitting that SMTP's MUA/MSA/MTA separation was right. One of email's quiet strokes of genius is that if the destination server is down, your server retries for days and you forget about it.

But let's add an improvement SMTP never had. Every message carries an ID that is the hash of its content, so retries are idempotent. The receiving server deduplicates by ID and the classic "duplicate email because the ACK failed" disappears by construction.

The full cycle of a delivery, with the destination node down on the first attempt:

sequenceDiagram
    autonumber
    participant Ana as Ana's client
    participant SA as Ana's server
    participant SB as Bob's server

    Ana->>SA: POST /outbox (signed message)
    SA-->>Ana: 202 queued
    SA->>SB: GET /.well-known/hmtp/bob
    SB-->>SA: Bob's inbox and keys
    SA->>SB: POST /hmtp/inbox/bob (envelope + sealed body)
    Note over SB: down: no response
    Note over SA: queue: retries with exponential backoff
    SA->>SB: POST /hmtp/inbox/bob (retry, same id)
    SB->>SA: GET /.well-known/hmtp/ana
    SA-->>SB: Ana's signing key
    Note over SB: signature verified, deduplicated by id
    SB-->>SA: 201 delivered

Notice that the diagram contains the entire protocol: the two GETs to .well-known are discovery and verification, the POST is the delivery, and the queue lives where it should, on the sender's server.

4. Signing and encrypting with the keys we already have

The message is a signed object, not loose text:

{
  "id": "sha256:9f2c...",
  "from": "ana@example.com",
  "to": ["bruno@example.org"],
  "date": "2026-07-26T10:00:00Z",
  "in_reply_to": "sha256:11ab...",
  "subject": "Re: that idea",
  "body": { "type": "text/markdown", "content": "<encrypted>" },
  "signature": "..."
}

This buys us a lot of things:

• Authenticity at rest: stored mail carries its cryptographic proof. A forward keeps the original signature. Forging the sender becomes impossible even through chains of forwards.
• Sender verification without DKIM: the receiving server GETs the .well-known of the from domain and checks that the key signs. It's the same move as Webmention verification. The proof is fetched at the source.
• E2E: the discovery document publishes encryption keys (X25519); the body is sealed with HPKE. The envelope (from, to, id, date) stays visible for routing and filtering; the content, only for the recipient.
• Threads: in_reply_to and references by content hash. Conversations reconstructed without heuristics.
• Attachments: outside the message. An attachment is {hash, url, size} pointing to the sender's server; the receiver downloads it on demand and their server can mirror it. No more base64 bloating mailboxes.

5. Layered anti-spam

Like in any system, this is a complex problem that needs several layers of defense. With HMTP we could start with three:

1. Identity cost: signing as ana@example.com requires serving the key document at example.com. Identity is anchored to a domain, and domains cost money. It's the sybil cost that self-signed identities don't have. However, a domain gives you infinite subdomains and mailboxes, so the cost slows down the mass creation of independent identities, not of addresses. We work at the root level.
2. First-contact consent: an unknown sender doesn't get into the mailbox; they land in a "requests" box with their first message visible (like Signal's message requests). You accept, and the thread opens forever. A stranger can knock on your door, which is the essential property of mail, but they can't fill up your living room.
3. Optional postage for strangers: the server can answer a first contact with a 402. Configurable per mailbox; the cost of cold spamming stops being zero.

6. Reading gets specified too

Email standardized sending (SMTP) and reading (IMAP/POP) as separate worlds. We don't need to invent anything: reading, synchronizing and searching mailboxes over JSON/HTTP is already solved and standardized by the IETF. It's called JMAP. HMTP would define delivery; reading is JMAP with one new object type. Push to the client with SSE or WebPush. The full cycle (send, deliver, read, sync) stays on HTTP.

The conclusion

Every problem of email has a solution deployed and working: discovery in Mastodon, delivery in ActivityPub, verification in the IndieWeb, rotatable identity in Bluesky, reading in Fastmail, consent in Signal. An upgrade to email already exists, but nobody has assembled it.

We solve many problems with elegant, modern solutions:

• Reputation and social permission (SPF, DKIM, DMARC, reverse PTR, blocklists, months of warming up an IP): replaced by cryptographic verification on every message, one signature and one GET to the sender's .well-known. A property that can be verified doesn't need reputation.
• Sender spoofing: impossible by construction. The receiver checks the signature against the key published on the from domain, and the signature travels with the message even when forwarded.
• Mail delegation (the MX record): a static document in .well-known, with per-user delegation and no DNS changes.
• Content privacy (the PGP nobody ever got around to configuring): end-to-end encryption by default; the receiving server stores a body it cannot read.
• Spam (statistical filters after the fact): first-contact consent (strangers knock on the door, they don't walk into the mailbox), costly identity anchored to a domain, and optional postage with 402.
• Duplicates on retry: the message id is the hash of its content, so retries are idempotent and the receiver deduplicates by construction.
• Threads rebuilt with heuristics: in_reply_to points to the hash of the parent message; the conversation is a verifiable graph.
• Light mailboxes: attachments by reference (hash + URL), downloaded on demand.
• Simple, familiar infrastructure: everything travels over standard HTTPS, behind the same Nginx and the same certificate that already serve your website.
• Key rotation (DKIM's operational nightmare): one command. Since nobody pins your key, the new one is trusted instantly and the stolen one becomes useless just as fast.

Talk is cheap, so I implemented a working prototype in Python: github.com/tanrax/hmtp. All in a single file. The prototype covers the full transport: signed delivery, verification at the source, end-to-end encryption, first-contact consent, deduplication, threads, a queue with exponential backoff and key rotation in one command. It deliberately leaves out the pieces at the periphery: the chain of signed rotations (receivers fetch your key live on every delivery instead of pinning it, so the chain only starts paying off once nodes cache keys), the 402 postage, attachments by reference and JMAP reading. The README has the quickstart (your first message in two minutes, mailing yourself), a demo of two nodes exchanging encrypted mail on your machine, and the full production guide, from DNS to systemd. Remember it's a design experiment with unaudited cryptography; don't use it for secrets anyone depends on. Although it could be a lightweight communication system for whatever ecosystem comes to your mind.

I hope you enjoyed the ride. And if you ever send your first HMTP message, I'd love to hear about it.

This work is under a Attribution-NonCommercial-NoDerivatives 4.0 International license.

Magnolias are so old they are pollinated by beetles instead of bees

Photo: nm737/Depositphotos

Many people begin to notice the arrival of spring with the large, beautiful blooms of the magnolia flower. Magnolia trees can be found in many parts of the world, and their beautiful forms have symbolic, medicinal, and visual meaning across cultures—and have for centuries. If you’re ever near a magnolia tree, though, look closely: you’ll notice that beetles, instead of bees, will be moving amongst the flowers.

So, why beetles over bees? The answer is simpler than you might think. Magnolias are so ancient that they were around long before bees came into existence. They’ve been around for over 100 million years, in fact, and beetles have existed for even longer , approximately 300 million years.

Named after the French botanist Pierre Magnol, magnolias belong to one of the oldest lineages of flowers on Earth. (Dinosaurs still walked the Earth at this time, to put it into perspective!) Given this ancient setting, the pollinators we’re most familiar with, butterflies and bees, had not yet evolved. Beetles were the primary insect pollinators for the time, and so they became the de facto agents for the magnolia’s survival.

This partnership between the flower and the beetle reveals itself in the magnolia’s look and scent. The flowers are large and shaped like a bowl, which is ideal for beetles to climb into. Their petals also boast more muted colors, as their partner pollinators navigate better through scent than sight. Which leads to the next, and perhaps most iconic, trait of the magnolia flower: its intoxicating scent that attracts beetles to it, meant to mimic the smell of fermenting or ripening fruit.

Another aspect of the magnolia that shows its advanced evolution is the sturdiness of the petals. Where many flowers usually have reputations for being delicate, the magnolia has developed thick, leathery petals. This is to withstand the beetle’s movement within its center, which can be clumsy and at times, rough.

As far as pollinators go, the beetle isn’t the most sophisticated. They can’t hover to collect nectar (or collect nectar at all) or perform more advanced pollinating behaviors. The way they pollinate is more of a happy accident. In their search for food, beetles will plow through petals of flowers, often leaving a mess behind. But in this process, they also get coated in pollen, which they carry on to the next flower, and the one after that, as they continue their search.

The beetle's method of pollinating, though not as sophisticated as that of bees or butterflies, has stood the test of time, at least for our dear magnolias. The ancient flower’s partnership with beetles is a testament to both of these agents’ ancient origins and resilience. With sturdy petals and a rich scent, the magnolia continues to thrive today, just as it did millions of years ago: through simple, time-tested evolution.

Magnolias, the beautiful pink and white flowers that bloom in early spring, have been around since dinosaurs roamed the Earth.

Magnolias are so old they are pollinated by beetles instead of bees

Photo: courtyardpix/Depositphotos

They're so old, in fact, that they rely on beetles instead of bees to pollinate them.

Magnolias are so old they are pollinated by beetles instead of bees

Photo: Reinhold Möller via Wikimedia Commons (CC BY-SA 4.0)

Beetles, who have been around for even longer than magnolias, predate bees by hundreds of millions of years.

Magnolias are so old they are pollinated by beetles instead of bees

Photo: wasnoch/Depositphotos

The arrangement, makeup, and scent of magnolia flowers reflect their unique and ancient partnership with beetles.

Magnolias are so old they are pollinated by beetles instead of bees

Photo: matenchuk/Depositphotos

Sources: Magnolias are so ancient they’re pollinated by beetles — because bees didn’t exist yet; The Botany of Magnolias

The environmental impact of the fluorinated molecule should be studied, outside experts say

A new small-molecule glue can be removed with ethanol.

Materials scientists are on the hunt for an adhesive that binds strongly to diverse surfaces and can be easily removed and reused. Conventional polymers like epoxies achieve high strength through rigid, cross-linked networks, but they are brittle and prone to sudden, catastrophic failure under stress. Conversely, adhesives used in tapes offer ductility but lack the strength for heavy-duty bonding.

Small-molecule glues utilize noncovalent bonds for adhesion, allowing the glues to be removed easily and to leave pristine surfaces behind. But they too lack tough adhesion, which is achieved when a material has both high ductility and high adhesive strength.

Now researchers have developed a small-molecule glue that not only bonds strongly to nonstick surfaces but is easily removed with ethanol (J. Am. Chem. Soc. 2026, DOI: 10.1021/jacs.6c07886). “It combines adhesive strength and ductility together with easy washability, which is needed for recycling the adhesive as well as reusing the adhered objects,” Takuzo Aida, a polymer chemist at the University of Tokyo, says in an email.

The white crystalline adhesive, synthesized by Aida’s team, is a fluoro-crown ether phosphate called CyclicFP-fmoc. To test its adhesive strength, researchers first glued two untreated polytetrafluoroethylene (PTFE) plates by sandwiching molten CyclicFP-fmoc between them and letting it cool at room temperature for 10 min. The plates, glued to each other by a tiny contact patch of just 7 cm2, were able to hoist an 8 kg weight and registered a staggering 1.3 ± 0.1 MPa on lap-shear tensile tests, eclipsing the 0.1–0.7 MPa limits of commercial epoxies, acrylics, and silicones.

CyclicFP-fmoc binds to Teflon via strong, interfacial fluorine-fluorine (F–F) interactions, solid-state NMR spectroscopy confirmed. Meanwhile, the glue molecules hold on to each other inside the adhesive layer via hydrogen bonding between urethane units and π-π stacking of the fluorenyl rings.

Remarkably, a simple ethanol wash completely stripped CyclicFP-fmoc from the target PTFE plates, leaving no trace behind. The recovered adhesive proved repeatedly reusable, reliably maintaining its adhesive strength of ~1.2 ± 0.1 MPa across multiple cycles.

Two metal dumbbell-shaped weights hang by a hook from a white strip of material. The white strip is made of two pieces that overlap in the center.

Two polytetrafluoroethylene plates joined with a new glue, called CyclicFP-fmoc, are able to lift 8 kg of weight. Credit: Kohei Kikkawa

“Since ethanol is the main component of hand sanitizer, we believe it is possible to de-bond and recycle the adhesive by using hand sanitizer as a substitute in certain real-world situations,” Aida says. He explains that due to the noncovalent bonds, recycling the adhesive simply involves a breaking of those bonds, yielding the original monomer, which can be reused with no loss of adhesive strength. “In general, this can’t be achieved by using polymer adhesives,” he adds.

Mechanical engineer Alban Sauret and bioengineer Phillip Messersmith, who were not involved in the study, both stress that it is imperative to study this molecule’s environmental fate and impact before pressing it into service.

Sauret, at the University of Maryland, found the adhesive’s combination of strength, ductility, and ethanol removability “quite unusual.” “The removable and recyclable nature of the adhesive is promising, as we would reuse fluoropolymer parts instead of discarding them,” he says. He finds it likely to be a member of the family of per- and polyfluoroalkyl substances (PFAS) based on its fluorinated structure, “but whether it behaves like a problematic ‘forever chemical’ is a separate question.”

Messersmith, at the University of California, Berkeley, notes that existing polymer adhesives “complicate recycling efforts . . . by making it hard or impossible to separate materials in a recycling stream.” If this new adhesive is not an environmentally harmful compound, he says, “the approach could allow for easy separation and recycling of adhesively bonded materials in the future.”

The animation scaling problem

Ray tracing has become a practical part of real-time graphics, but animated geometry still presents a difficult scaling problem. The paper Ray Tracing Massive Amounts of Animated Geometry by Gruen et al., winner of the third-place Wolfgang Straßer Award (Best Paper) at High-Performance Graphics 2026, tackles a common production case: scenes with many complex objects that all deform differently. In a conventional ray-tracing pipeline, every uniquely animated mesh may require both its vertices and its acceleration structure to be updated each frame. That computational cost grows with triangle density, so geometrically dense foliage, grass, crowds, or creatures can quickly consume the frame budget. As each unique animated object may need to store its own set of bounding volume hierarchy, memory consumption may become prohibitively high.

Figure 1. Animated scene with roughly 585 million animated triangles @ 60 frames per second (AMD Radeon™ RX 9070 XT GPU at 1080p)

Decoupling animation and triangles

The central idea of the paper is to decouple animation cost from triangle count. Instead of applying animation directly to every triangle, the method builds a low-resolution tetrahedral cage around the original, non-animated mesh. During preprocessing, the mesh is split into small, disjoint pieces associated with the tetrahedra. Static mini-BLASes are then built for these pieces once and reused. At runtime, only the cage is animated. Rays that enter an animated tetrahedron are transformed into the corresponding rest-pose space, where they can intersect the static geometry. The resulting motion is an approximation: vertex movement is represented by a piecewise-linear deformation induced by the cage, rather than by a smooth global bend or by full per-vertex animation.

This makes the approach especially attractive for massive scenes where many copies of the same asset are animated in unique ways. Each unique deformation needs its own animated cage, but it can share the same rest-pose mini-meshes and mini-BLASes. The paper demonstrates this with trees, grass patches, and frogs, and shows a combined scene of roughly 585 million animated triangles rendered at 60 frames per second on an AMD Radeon RX 9070 XT graphics card at 1080p. The key message is not that tetrahedral cages replace every animation technique, but that they provide an additional new low-cost option when acceleration-structure updates, animation time, or memory footprint become the bottleneck. The paper also sits alongside similar independent work by Luton and Tricard (2026), which explores tetrahedral indirection but uses multiple hardware rays and no clipping step. Please note that tetrahedron-cage driven animations can be combined with the Microsoft DirectX® Raytracing (DXR) Functional Spec, Part 2 – e.g. partitioned top-level acceleration structures – and can also coexist with cluster level acceleration structures.

Figure 2. High-level workflow.

The trade-off is control over quality versus performance. A coarser cage is cheaper and smaller, while a finer cage can follow the original animation more closely. The method is best suited to connectivity-preserving animation, such as swaying vegetation, grass, crowds, distant characters, or animation level of detail. It is less appropriate for cases where topology changes, very small-scale motion, or sharp character deformations are the visual focus. Please note though that tetrahedron-cage driven animations can also be used in the context of partitioned top-level structures and can also coexist with cluster level acceleration structures.

What’s next for tetrahedral cages

Overall, the paper presents tetrahedral cages as a practical bridge between rich animated content and hardware-accelerated ray tracing. By animating a compact proxy and reusing static rest-pose geometry, it opens the door to scenes that would otherwise be too expensive to update every frame. Future work will focus on improving cage generation and animation quality, as well as integration with modern ray-tracing pipelines. Please note that we are hard at work to create DXR samples and a header-only C++ library that allows building high-quality tetrahedral cages for skinned objects, keyframe-animated objects, and static objects. If you’re working on animated geometry at scale, we encourage you to dive into the full paper below, and stay tuned to GPUOpen as this research continues to evolve.

• Download the Ray Tracing Massive Amounts of Animated Geometry paper and see more AMD graphics research on our Research Publications page.
• Watch a video of the High-Performance Graphics 2026 presentation of this paper.

Footnotes

Links to third party sites are provided for convenience and unless explicitly stated, AMD is not responsible for the contents of such linked sites and no endorsement is implied. GD-97.

DirectX is a trademark of the Microsoft group of companies.

Here's how Misago works currently:

1. You request https://misago-project.org.
2. Django view gathers data to render list of threads.
3. Django view renders template that contains almost complete HTML with list of threads, but also includes a JSON with same data used to render this HTML. And there is no interactivity because forms are disabled.
4. JavaScript downloads.
5. JavaScript runs, reads JSON embedded in the page's body and replaces most of HTML rendered from Django templates with HTML from React.js components. Buttons become active, and timestamps in UI swap.

This approach has some problems:

• A lot of pages in Misago are implemented twice: as Django templates and React.js components.
• People who start customizing HTML get burned because they edit Django templates and see their changes flash for a second on a site before being replaced by React.js HTML which they didn't know they also need to customize.
• Every view accessible to both users and guests needs to be done twice: as Django view with templates, and as React.js route with components. It also requires an API and JSON serializers to power data fetching.
• JSON serialization to pre-bake data for React.js app slows down response generation.
• A large part of translation messages is duplicated, living in both django.po and djangojs.po files. JavaScript translation files also increase the initial download size.
• Lots of JavaScript can kill performance, especially on older and slower mobile devices.
• Enabling plugins to replace or inject custom HTML to the page requires for those plugins to implement both Django templates and React.js components. And Misago will need to implement a JavaScript build step as part of site build in misago-docker. Plugin devs need to know both.

I've considered two solutions to this problem:

1. Drop Django views and templates, only keep those in minimal versions to keep search engine bots happy. Focus on implementing an API and having all UI as React.js app.
2. Reduce Django to API and use JavaScript framework with serverside rendering, eg. Next.js or Remix.run.

But here's a thing: there's a lot of forum software out there that still does the old way of rendering as much as possible on the server and using some JavaScript on client to improve its interactivity here and there. And people are happy with that. And this approach has none of the above issues.

Internet forum software has plenty of interactivity, but this interactivity is isolated to specific places on the page. Moderation actions, watching a thread, writing a reply, seeing last notifications, voting in a poll. All this stuff can be achieved without React.js and was achieved without it for years before we've decided that full page reload is something that needs to be avoided.

Now, what's HTMX? HTMX is a tiny library that lets developers specify parts of HTML as dynamic islands that can be swapped by new server-rendered HTML on interaction. For example, list of actual threads is one such (large) island. With little bit of HTMX included in Django template, changing current category on threads list could pull new HTML from Django only for new threads list for selected category, while keeping rest of the page's HTMX (eg. already loaded search results in navbar) unchanged. Misago's backend code would only need to be changed to return only the HTML for this island when request comes from HTMX, instead of full page. There's no need to do any JSON serialization or write dedicated JavaScript or React.js.

HTMX is declarative way of doing $.get("url", "#outlet") we've did in jQuery 20 years ago. Or Rails Turbolinks. I can't believe I am writing this, but this is the way for forum software if you hate endless scrolling or want to keep things simple otherwise.

GitHub issue: #1674

I've also decided against moving the admin panel to SPA. Currently I have a bunch of reusable Django views that take care of 90% of work, and adding new page to admin requires one to pick right base view, fill in blanks, define one or few forms and write very basic templates for those. Rest happens automagically and I like it.

The effort required to move away from React.js and to HTMX will be a big one, but it can be done as multiple small steps. Eg. one release could move Navbar to HTMX, other threads list, next thread page, another user profiles, etc. etc. But plugins/permissions/new parser still have to happen first. So HTMX migration will have to wait until later 2024. I am fine with that becasue I would also want to move to latest Bootstrap, and those guys are cooking some crazy stuff in it.

I also need to give a warning that we will likely have a transition period where parts of Misago will have no React.js, but also no HTMX either, effectively making them "multi-page application". This may sound scary but it only means that clicking a link or submitting a form in, say, user options page will cause a full page reload. But I will still write a placeholder AJAX (or just use the HTMX) where full page reload will be absolutely unbearable, eg. for liking posts or voting in polls.

For the record, JS sizes from Misago 0.39:

vendor.js: 679kb, 214kb gzipped
misago.js: 615kb, 124kb gzipped
django-i18n.js: 102kb, 25.4kb gzipped

Misago JS also includes lazily loaded:

hljs.js: 144kb, 49kb gzipped
zxcvbn.js: 820kb, 430kb gzipped

I plan to return to those numbers once I start removing React.js (and other JS dependencies) from Misago.

"Forum options" page has been replaced with "Account settings". This new page is done fully with Django views, and uses HTMX on some pages for dynamic updates.

JS size change:

misago.js: 578kb, 107kb gzipped (37kb/17kb less)

Threads lists have been rewritten almost from scratch to use Django views and HTMX instead of React.js.

JS size change:

misago.js: 530kb, 99kb gzipped (48kb/8kb less)

Today we’re announcing MAI-Cyber-1-Flash inside of MDASH, our multi-agent vulnerability identification and remediation harness. Together they deliver world-class performance at 50% of the cost of leading models.

Progress in AI has been startling and so has the new generation of cyber threats it’s unleashing. Attackers now wield increasingly powerful capabilities, probing an ever-growing mountain of code for just a single weakness that lets them in.

As the cost of finding a flaw collapses, the old model of security, where you scan occasionally and patch eventually, is now obsolete. If we’re to unlock the true benefits of AI, we must first build outstanding cyber models that help all of us harden the software the world runs on.

That’s the motivation behind MAI-Cyber-1-Flash, which has been built to find challenging vulnerabilities in complex codebases. It’s been deeply integrated into MDASH, honed by the best cybersecurity experts in the industry and hardened across the largest security estate on the planet.

This combined expertise delivers exceptional security protection, beating Mythos, Gemini and GPT on CyberGym, the gold standard benchmark for evaluating how systems reason over large codebases to find real vulnerabilities in the code.

Picking the right model for the task

Security is an always-on mission, and given the enormous volume of inbound attacks, token cost is now the real constraint for defenders. MAI-Cyber-1-Flash was designed to efficiently handle up to 90% of all tasks, enabling MDASH to use the larger and most costly models in our fleet (in this case GPT-5.4) for the 10% of exceptionally hard tasks that truly need them.

The result is that the unified system of MDASH with MAI-Cyber-1-Flash delivers 96% on CyberGym (+12 pt above Mythos).

This combination delivers a 50% cost saving when compared against our best offering in MDASH today (GPT 5.4 + 5.4 mini + 5.3 codex). That’s the power of a well-tuned, multi-model system with access to uniquely rich historical training data. It ensures you always have the best model at the best price for every task.

In this new environment, being able to go from identifying a new vulnerability to addressing it in real-time is critical. And while AI remediation of software vulnerabilities is now a key security workflow, there are many jobs to be done by Security practitioners themselves.

That’s why today we’re also launching Perception, our agentic security systems, that provides teams of agents for a variety of security workflows in MDASH, to continuously monitor, patch, and close new threat vectors. Perception will also soon use MAI-Cyber-1-Flash for many more security workflows, beyond the software vulnerability work.

Three things matter today: Model. Data. Harness.

We have jointly optimized our world-class models, our unmatched historic data, and our expert-tuned harness to ensure that our customers have a uniquely powerful security offering.

Model. MAI-Cyber-1-Flash is a compact, code-heavy security model derived from the MAI-Thinking-1 lineage, which was built from scratch, in-house, on the highest quality data. Details in our technical report.

Data. Our deepest advantage. Decades of building world-class security systems now give us trillions of daily signals across identity, endpoint, cloud, and network, and an unmatched record of real exploits and remediations. No one can manufacture this history.

Harness. MDASH, our multi-agent vulnerability identification and remediation harness, is tuned by the best security experts in the industry, who have created 100+ agents using multiple leading models to find, validate, and remediate vulnerabilities. Agentic code scanning is a critical function in the Security Operating Center and feeds Project Perception, our new agentic security system.

Built with safety first

Because MAI-Cyber-1-Flash is Microsoft’s first cyber model, we built trust into every layer of the system, from model training to customer deployment. The model was developed with a security-first calibration, rigorously evaluated by Microsoft’s AI Red Team, tested through automated and expert-led adversarial exercises, and independently assessed by a third party.

Trust extends beyond the model itself. Through MDASH, customers get enterprise-grade controls including Role-Based Controls, tenant isolation, encryption, auditability, and sandboxed execution environments with no internet access. The result is a cyber model that delivers powerful capabilities to defenders while maintaining the governance, security, and control enterprises expect from Microsoft.

Our hill-climbing machine

Cybersecurity is not just a data-rich domain; it is a live reinforcement learning loop. Every day, defenders investigate threats, triage alerts, hunt adversaries, remediate vulnerabilities, deploy protections, and learn from the outcome.

Microsoft sees that loop end to end: vulnerabilities through Microsoft Security Response Center; attacks and defenses across identity, endpoint, cloud, data, browser, and applications; more than 100 trillion security signals every day; and operational insight from 1.6 million customers. Because we can connect actions to outcomes; what was exploitable, what was contained, what was blocked, and what actually worked; we have more than data.

Our MAI reinforcement learning loop gives us the foundation to build cyber models that improve continuously and become expert cyber defenders. That’ll remain our commitment to our customers for years to come.

Build the Future With Us

We’re a lean, talent-dense team of explorers, researchers, and full-stack engineers. We move fast, sweat the details, and operate at frontier scale with a roadmap to build the world’s most powerful AI models. Most importantly, we’re united by the belief that doing this right is the only way to do it at all. If our mission resonates with you, we’d love to talk.

Explore all jobs