A Look at Recurrent Depth, Hidden Chains of Thought, and Recent Research on Looping Transformer Blocks

A lot has happened in the last few weeks. I am sure that OpenAI’s GPT-6 Astra is top of mind for everyone right now. In particular, thoughts on its performance, the looped transformer/recurrent depth aspects, and rumors that Astra is “hiding” its reasoning trace (i.e., chain of thought).

So, in this article, I want to start with some brief impressions of Astra and some thoughts on where all this is headed. Then, I will discuss, in detail, what “looped transformers” are, and how (or rather, if) this relates to hiding chains of thought.

Lastly, after covering the basics of the looped transformer, I wanted to highlight some new insights from recent research papers on the topic.

1. GPT-6 Astra impressions

First things first. Before getting into the architecture rumors and related research literature, let me briefly summarize some GPT-6 Astra observations and tidbits.

Last week, OpenAI’s new GPT-6 Astra was released with a big fanfare. I used it over the past couple of days, and it’s an exceptionally good model, likely the best I’ve used as of this writing. But what, exactly, has it improved, and how?

1.1 Astra benchmarks

Astra is the best model I’ve used so far, and it’s disproportionately good at 3D rendering and animation tasks (relative to other models). With that, I mean that while it leapfrogs its GPT-5.6 predecessor in practically all categories (writing, math, coding, and more), it especially does so when it comes to graphical demos.

We can see this also reflected in the benchmarks. For instance, GPT-6 Astra is really good at math and coding, as shown below.

Figure 1: Selection of three popular coding benchmarks and one challenging math benchmark. More benchmarks are shared on the Astra release blog: https://openai.com/index/gpt-6-astra/

One of the highlights (not shown in the figure) is that Astra also achieves 99.9% on the ARC-AGI-3 benchmark (GPT-5.6 Sol only 7.8%), which measures a mix of solving logic puzzles and generalization. However, the math, coding, and computer use benchmarks are more interesting because they are closer to real-world use.

Coming back to the Artificial Analysis Coding Agent Index v1.4 (lower right in the previous figure), which blends several agentic coding tasks, GPT-6 Astra is clearly at the frontier, but it doesn’t pull ahead by leaps and bounds. This can also be seen in the general Artificial Analysis Intelligence Index shown below, which blends different types of tasks, not just coding tasks.

Figure 2: Artificial Analysis Intelligence Index via https://artificialanalysis.ai/#intelligence

Now, the big advantage of Artificial Analysis benchmarks is that they are independent and thus may be a bit more trustworthy than self-evaluated benchmarks by model developers.

The harness setup depends on the benchmark. For example, GDPval-AA and AA-Briefcase use their open-source, minimal Stirrup harness across the different LLMs they compare. In the Intelligence Index v4.2 shown above, Terminal-Bench v2.1 uses Terminus 2, and τ³-Banking uses the τ-Bench harness. The separate Coding Agent Index also compares different coding-agent harnesses.

For evaluations that use a shared harness, this makes it more of an apples-to-apples comparison. At the same time, during model training, models are typically developed with one primary harness in mind (and fine-tuned less on other harnesses). Plus, the primary harness is often developed to suit and amplify a model’s strengths.

So, some of the agentic evaluations might underestimate how well Astra performs in its primary harness. How much this affects its Intelligence Index score would need to be tested by comparing Astra across harnesses on the same tasks.

As a side note, as a colleague recently suggested to me (as also recommended by the Claude Code lead), it’s maybe not a bad idea to delete (/archive) some of your existing AGENTS.md contents and SKILL.md files, as newer LLMs have become more efficient at understanding the prompt and solving the problem at hand. The extra hand-holding could unnecessarily constrain newer models and lead to worse solutions.

Of course, I am not suggesting never using SKILL.md files again, but for some workflows, because they can improve efficiency upon reuse, since the model doesn’t have to rediscover them. But what I am suggesting is that some workflows don’t need describing, and “old” descriptions may no longer be ideal, and the LLM may be able to come up with better solutions. So, it’s perhaps time to update or regenerate said instruction files.

1.2 Computer use capabilities

GPT-6 Astra seems to be exceptionally strong in image and rendering tasks. When these tasks involve interacting with graphical user interfaces, they also demonstrate computer-use capabilities, meaning the model operates software on your local computer through the Codex/ChatGPT app.

Computer use is where the model really shines compared to others, and anything graphic-related also makes for interesting and intuitive demos on social media platforms. There are tons of examples of impressive demos out there, from modeling rendering New York City in blender to virtual open house tours.

To pick one example, below is a comparison where I had GPT-6 Astra Medium and High redraw a picture of me in a browser version of MS Paint using the mouse on my computer (not Extra High and Max, because I didn’t want to waste all my tokens :)).

This highlights not only the model’s artistic capabilities but, more importantly, its ability to use tools on one’s computer (in this case, Paint; you can see the model using the interface via the mouse cursor).

This is not the first model that, inside a harness, is capable of general computer use. For example, I successfully used GPT models for some UI tasks (e.g., expense-related tasks in Excel) and so on since earlier this year. However, computer use is a relatively new capability, enabled by the harness, and usually feels not quite as mature yet. This makes sense. LLMs are text models, so naturally the lower-hanging fruit is writing and coding and using APIs and CLIs.

At the same time, there are many tools and software that don’t expose CLIs (yet), and instead of waiting until someone designs that interface, why not improve models to use graphical user interfaces (and, as mentioned before, this makes for pretty and impressive demos, anyway)? This is somewhat analogous to the emerging humanoid robot developments. Sure, humanoid robots are not the most efficient robots, for example, at the assembly line, where special-purpose machines exist. But they are versatile.

So, I expect the upcoming months (or years) also to be an era of computer use refinement on both the LLM and the agent harness layer. I.e., in addition to the current capabilities, and expanding their math and coding capabilities, models will be trained with an increasing amount of computer use in mind. And this will also make LLMs more accessible for everyday computer tasks outside the tech world (”Hey ChatGPT, please do my tax return” :))

1.3 Computer use training

The computer usage trend is also consistent with the recent reporting that OpenAI purchased tens of thousands of Mac Minis and Mac Studios for Reinforcement Learning. So, here the Macs are not used to literally train the models (it’s better to use GPUs for that) but rather to expose macOS during the model training for the model to learn to use said operating system and the tools therein.

So, how does computer-use training on said Macs work? In short, the Macs (or their macOS operating system, to be precise) serve as an environment that the model can interact with during training.

The basic workflow looks like this:

1. Prompt the model by giving it a task, such as “open an app xyz and do abc”.
2. Provide it with screenshots of the macOS interface (this is usually done by the harness).
3. The LLM then predicts mouse/keyboard actions (click, key presses, scrolling, and so on).
4. Execute those actions on the Mac (again, this is done by the harness).
5. Feed new screenshots of the updated environment after performing the actions in the previous step.
6. Repeat steps 2-5 until the task succeeds or fails.
7. Use success/failure signals and verifiers (or graders) as training feedback, including reinforcement learning during post-training; this is analogous to regular Reinforcement Learning with Verifiable Rewards (RLVR).

Figure 3: Overview of a computer-use training workflow.

Again, the Mac is mostly the environment here and not the machine for running or updating the model during training. The model likely sits on NVIDIA GPUs and is fed via API to said Mac. By the way, NVIDIA’s CEO mentioned that GPT-6 Astra was being trained on ~100,000 Grace Blackwell GPUs.

1.4 GPT-6 Astra is still a reasoning model

The focus on computer-use training discussed in the previous section is not a fundamental paradigm shift in the training pipeline. GPT-6 Astra (and likely any LLM in the foreseeable future) is still a reasoning model. This means the LLM is trained with reinforcement learning with verifiable rewards (RLVR) and produces intermediate reasoning traces (chains of thought)

But I will discuss the reasoning model aspects of GPT-6 Astra (especially regarding hiding chains of thought) a bit later in this article.

2. Looped transformers

That being said, about two days before the official model, the news magazine The Information published an articlereporting that, according to some inside information, Astra is using a concept called “recurrent depth” or “looped transformers.”

Figure 4: Quote from The Information (Source: https://www.theinformation.com/articles/secret-technique-behind-openais-astra-model-sparks-security-concerns)

Since LLM architectures are within my area of expertise and my passion, I created a short lecture video explaining the general looped transformer mechanism and addressing the comment about hidden reasoning chains, which you can find below.

In the following subsections, I’ll first explain what looped transformers are, and I’ll revisit the comment about the hidden chains of thought later in this article.

(The looped transformer explanation may seem a bit long, but I really think that it helps with establishing a foundational understanding of the technique, which is then useful to judging the claim that it obscures the reasoning traces or chains of thought.)

2.1 Reusing transformer blocks

So, what’s a looped transformer?

A Looped Transformer is essentially an architectural tweak, with the main idea being to pass the intermediate representations through the same transformer blocks multiple times (instead of just once). Compared to just adding more blocks, the “trick” here is that the weights stay the same across these passes.

---

Definitions & Jargon

Throughout this article, I’ll use the following terms:

• A transformer block is a unit containing attention, a feedforward module, normalization, and shortcut connections. These blocks are often called “transformer layers” in papers.
• A stack is a sequence of transformer blocks.
• A block application means running an input through a transformer block once.

---

The looped transformer is nothing new, and the basic idea already appeared in the Universal Transformers paper from 2018. But before discussing Universal Transformer, let’s start with a simpler example, Nanbeige4.2-3B, a recent open-weight LLM that came out in July and that I covered on Substack Notes and in my LLM Architecture Gallery earlier this summer.

The Nanbeige architecture, shown below, essentially looks like a regular transformer. However, notice that it has an extra (orange) arrow looping back to the beginning of the transformer stack.

Figure 5. Nanbeige4.2-3B applies the same stack of 22 transformer blocks twice. The orange arrow shows where the intermediate representations are passed back into the stack.

Let’s walk through this from the bottom up. First, as in any other transformer-based LLM, the input text is tokenized and converted into embedding vectors. These vectors then pass through 22 transformer blocks, and each of these 22 blocks has its own weights.

However, the looping transformer aspect here is that after the first pass, the hidden states are fed back through the same 22 blocks. So, block 1 is applied again, followed by block 2, and so on up to block 22.

If we were to unroll this computation, we would have 44 transformer block applications. However, compared to a conventional transformer with 44 distinct blocks, the second stack of 22 block applications reuses the weights from the first stack. For example, block application 23 uses the weights of block 1, block application 24 uses the weights of block 2, and so on.

Figure 6. Nanbeige4.2-3B unrolled into two passes through the same 22 transformer blocks, giving 44 block applications.

So, the whole idea here is that we increase the effective depth from 22 to 44 block applications without adding another set of transformer weights.

By the way, why 2 rounds, not 3, 4, or more? There are not many details in the Nanbeige paper, but they say that this was essentially the most efficient setup. Increasing the loops from 2 to 3 can increase modeling performance, but the extra computational cost wasn’t worth it.

2.2 Looping costs

So, why would we do this looping in general? This is essentially an alternative to just making the model bigger by adding more transformer blocks.

So, for instance, a model that uses 22 transformer blocks twice has roughly half as many (transformer-block) parameters compared to a model with 44 conventional blocks.

This then reduces the memory needed to store the weights. As a side note, note that the embedding and output layers, which are usually large and make up a substantial portion of the total, are separate from this comparison. (In the case of Nanbeige 4.2 3B, the embedding and output layers make up ~25% of the total 3B parameters; with weight sharing between those two, we could reduce that to 12.5%.)

hypothetical-size

Figure 7: Side-by-side comparison showing how many parameters would be required in traditional versus looping scenarios.

Of course, reusing the same blocks in a loop still requires computation. More precisely, we pass the intermediate inputs through 44 block applications during the forward pass. And, during training, gradients flow backward through both repetitions of the shared stack. So, compared to using the 22 blocks only once, this adds substantial work. Actually, it’s similarly expensive as having 44 distinct blocks (except the optimizer has fewer distinct parameters to update; backprop still runs through all 44 block applications).

There is also the KV cache, which stores the attention keys and values of previous tokens for reuse in conventional and looped transformers in each next-token generation step.

But back to the topic. Even though there is weight-sharing in looped transformers, the intermediate states that enter a block are different on the second pass. Consequently, in KV caching, the resulting keys and values are also different between these two transformer stacks (just like in the no-looping case). So, there are no KV cache-related savings either.

To make this more concrete, for example, consider block applications 1 and 23, which both use block 1 in the looped transformer setup. But each application still needs its own KV cache entries. So, since we have to keep separate caches for both passes, the repeated stack of 22 blocks has the same KV cache requirements as a conventional transformer with 44 distinct blocks.

Interestingly, the Nanbeige researchers reported in the paper that they tried sharing the KV cache between passes. This, of course, halved the KV cache size, but the model performed worse than the version with separate caches (which is the version they released).

Just to complete the Nanbeige discussion before moving on and looking at some other looped transformer designs, their technical report also discusses two other choices or trade-offs.

1. Training the looped architecture from scratch worked better than converting an already pre-trained transformer through upcycling.
2. And two passes gave their preferred trade-off, as mentioned in the previous section. More passes brought only small additional gains while slowing training and making optimization less stable.

So, the number of passes is another architectural choice we have to make. As mentioned before, in Nanbeige, this is fixed at two. But we can also make it depend on the token, as we will see next.

2.3 Universal Transformers and flexible loop counts

Now, let’s come back to Universal Transformers. In Nanbeige, we apply a stack of 22 transformer blocks twice. In the Universal Transformer paper from 2018, we repeatedly apply the same transformer block instead of repeating a stack of transformer blocks. The main idea is similar, though.

Also, the number of steps can be fixed, but the paper also explores adaptive halting. For example, a token at a particular position may only go through one or two loops. Another may go through three or four loops, and so on. This gives the model flexibility to allocate the compute to those tokens that benefit from extra computation.

How is the looping number decided? Here, the model uses a small, trained function that outputs a so-called halting probability for each position at each step. It adds up these probabilities over these successive loops and then stops looping at a given position once the sum exceeds a threshold value. In addition, a maximum loop count also limits the computation just in case.

adaptive-halting

Figure 8: Adaptive halting in a Universal Transformer.

Another example of a looped transformer is ByteDance’s Ouro, which I also covered in my LLM Architecture Gallery. For instance, Ouro-Thinking 2.6B applies the same stack of 48 transformer blocks four times. That’s 192 block applications while storing weights for 48 distinct blocks. Basically, that’s a more extreme case than Nanbeige. Additionally, a learned exit gate assigns probabilities to the different exits, and a threshold on the cumulative probability determines which pass supplies the output. So, it’s also borrowing the adaptive halting idea from Universal Transformer, which Nanbeige didn’t use. (However, there is a practical caveat here. The released Hugging Face implementation computes all configured passes before selecting an output, so it seems like the number of loops is effectively hard-coded to 4).

2.4 Routing flexible loop counts

Another approach is Mixture-of-Recursions, a paper from 2025 that is essentially a more sophisticated version of the Universal Transformer discussed earlier. Similar to the Universal Transformer, individual tokens pass the transformer blocks one or more times as illustrated in the figure below. However, the innovation is how this looping number is determined on a per-token basis.

In the following figure from the paper, the looped (repeated) stack is called a recursion block here. This contains several transformer blocks, and it sits between separate first and last transformer blocks (labeled Layer 0 and Layer L-1).

mor

Figure 9. Mixture-of-Recursions applies a shared stack a different number of times at different token positions. The highlighted text shows an example with 1, 2, or 3 passes. Figure adapted from the Mixture-of-Recursions paper.

How does the model decide how many times a token should go through the recursion block? In the previously discussed Universal Transformer, it’s based on a learned halting probability at each step. This Mixture-of-Recursion approach here uses a small, learned router. This is similar to the routing idea in a mixture-of-experts model, except that here the routing decision determines how many times to apply the shared stack.

The router operates on a token’s hidden representation, which also contains information about its context. So, we shouldn’t think of this as assigning every occurrence of a particular token the same number of passes (i.e., the word “People” in the figure above doesn’t always go through a loop of 3). The decision can change depending on where that word appears and what came before it.

Now, how does the routing work exactly? The paper explores two ways to make this routing decision, as illustrated below.

mor-routing

Figure 10. Two ways to choose the recursion depth. On the left, routers select which tokens continue at each step. On the right, a single router assigns the number of passes at the beginning. Figure from the Mixture-of-Recursions paper.

In expert-choice routing, which is shown in the left subpanel in the figure above, each recursion step selects which tokens it will process. Tokens that exit are excluded from later steps. In token-choice routing, shown on the right, the router makes one decision at the beginning, assigning each token to a path with one, two, or three passes.

In both cases, the transformer weights are reused across passes, similar to Nanbeige, etc. But the additional flexibility comes from choosing how much computation each token receives. The model and its routers are trained together, so the model learns to work with these different paths during training.

2.5 How well does this work?

The plot from the Mixture-of-Recursions paper below compares a regular transformer (Vanilla), a transformer with fixed recursion (Recursive), and Mixture-of-Recursions (MoR) for different model sizes and compute budgets (x-axis).

mor-results

Figure 11. Validation loss across four model scales and three training compute budgets. Figure from the Mixture-of-Recursions paper.

At the smallest model scale, the regular transformer performs best. For the larger models, Mixture-of-Recursions catches up and often performs better, especially at the smaller training budgets. At the largest budget, several of the curves are very close. So, the advantage depends on the model size and how much compute we spend on training.

Another detail here is that equal training compute doesn’t necessarily mean an equal number of training tokens. By skipping some computation, Mixture-of-Recursions can process more tokens within the same budget.

I think this is an interesting example because it shows that there are several choices within the looped-transformer idea, that is, how many loops there are at each position and how that’s decided.

So, in short, we can say that using looped transformers can improve model quality at a fixed compute budget if the model is large enough. (It also illustrates the importance of running some experiments at scale; e.g., just looking at the smaller 135M parameter model, we would have drawn the opposite conclusion.)

3. Side note: Recurrent Neural Networks (RNNs)

By the way, if you have a background in deep learning (or even artificial neural networks in the 1990s), the looping or “recurrent depth” idea should be somewhat familiar. Remember recurrent neural networks (RNNs)? The whole idea in RNNs is to reuse the layers (weights) from a previous iteration.

rnn

Figure 12: Illustration of an RNN (from my 2022 “Machine Learning with PyTorch and Scikit-Learn book”, https://amzn.to/3YzRnPR)

The main distinction is that RNNs reuse their weights across time steps. That is, the hidden state is carried forward from one token to the next. In the looped transformer, the looping of a token is across the architecture depth.

Or, in other words, in a conventional RNN, each step takes the next element in the input sequence and the hidden state from the previous step. So, when the RNN is processing a chunk of text, it reads one word or token at a time and carries information from the earlier words forward in its hidden state.

In a looped transformer, the intermediate representation of a given token goes through the transformer stack multiple times. The model still uses attention to pass information between tokens.

If this analogy is a bit too confusing, don’t worry about it too much. A perhaps simpler way to think about looped transformers is to think of them as reusing transformer blocks, similar to making the model bigger but with weight sharing.

rnn-vs-looped-transformer

Figure 13: Side-by-side comparison of the “recurrence” in an RNN and a looped transformer.

4. Does Astra even use looped transformers?

Before we discuss whether the looped transformer mechanism obscured reasoning traces, as rumored in The Information quote from earlier, does GPT-6 Astra even use the looped transformer concepts?

the-information-2

Figure 14: Quote from The Information (Source: https://www.theinformation.com/articles/secret-technique-behind-openais-astra-model-sparks-security-concerns)

We have to keep in mind that this is still just a rumor or scoop, with no official confirmation. If the model were open-weight, we could double-check this ourselves, of course, but in this case we have to rely on unverified reporting.

However, I think it’s highly likely that GPT-6 Astra uses looped transformer aspects. First, there is the reporting mentioned above. Second, it’s a technique that has shown promise in past studies (as discussed earlier), so why not? Third, OpenAI’s chief scientist said the following.

[...] The depth of the computation graph for our present frontier models, including Astra, is within a factor of two of GPT-4. [...]

However, this doesn’t confirm the looped transformer architecture explicitly, and it could also just mean they use twice as many regular transformer blocks.

In my opinion, the success (i.e., good modeling performance) behind Astra is likely primarily due to other reasons, namely improved training recipes and training data.

The looped transformer tweak might help a bit, but I think that The Information is overestimating its contribution.

5. Hiding chains of thought

Next, let’s finally address the elephant in the room: does looped transformer obscure the reasoning traces?

First, OpenAI has been hiding (most of) the reasoning traces from users from the very beginning, since OpenAI o1, anyway. So, for the end-user, there shouldn’t be a big difference.

So, the interpretation-concern is mostly with respect to the model developers.

Either way, I don’t think that looped transformers are significant contributors towards hiding or obscuring chains of thought. To explain my own reasoning (no pun intended), let’s take a step back and explain how reasoning models work.

5.1 Reasoning in brief

Reasoning models typically generate intermediate steps before producing a final answer. These steps use regular text token (that are optionally hidden from the user in some user interfaces) and called a reasoning trace or chain of thought.

For example, say we ask for two numbers whose sum is 10 and whose product is 21. In the figure below, the model tries 5 and 5 at first. While the sum is correct, the product is 25, not 21. Next, it then tries 3 and 7 and checks both conditions again.

backtracking

Figure 15. An illustrative LLM response annotated to show intermediate steps, backtracking, and the final answer.

The figure illustrates how a reasoning model “reasons,” including backtracking. That is, the model notices a mistake, then revisits an earlier choice, and then continues with a different approach.

Note that the model still generates one token at a time, using the prompt and previous tokens as context. So, these intermediate steps work as a scratch pad and add computation before the final answer.

The final answer can then be much shorter than the reasoning trace that preceded it, as shown in the example above. (OpenAI tends to hide most of the reasoning traces from the users.)

For more details on understanding and developing reasoning models, I recommend my book Build a Reasoning Model From Scratch.

reasoning-book

Figure 16: My Build a Reasoning Model From Scratch book covers the fundamentals of reasoning models.

5.2 Token usage and shorter chains of thought

Now, extra tokens in the reasoning trace add more computation. Looped transformers add more computation, because the tokens go through more transformer blocks. One might argue that a model with looping uses more computation internally, it doesn’t need as many external thinking tokens.

Below is a selection of the GPT-6 benchmarks with the output token number on the x-axis.

output-tokens

Figure 17: Selected GPT-6 Astra benchmarks from https://openai.com/index/gpt-6-astra/

We can see that GPT-6 Astra doesn’t necessarily use fewer tokens than its GPT 5.6 Sol predecessor across effort levels overall. However, at a fixed accuray, it is true that GPT-6 Astra uses fewer tokens than GPT 5.6 Sol.

Is this a concern for interpretability? Not necessarily. Using fewer tokens could just mean that the model is more capable and makes fewer mistakes, uses less backtracking, and so on. I.e., it might just get more things right on the first try. To me, that doesn’t raise an immediate concern regarding interpretability.

I mean, the same is true for previous models. I don’t think that anyone has strong concerns that GPT 5.6 Sol is so much less interpretable than the smaller GPT 5.6 Luna model, which uses many more tokens for the same task performance, as shown below.

luna-sol-token-usage

Figure 18: Token usage in Luna and Sol at similar task performance levels. Numbers from the Artificial Intelligence Index v4.3.

In fact, as we can see that Luna uses 80% more tokens than Sol at similar modeling performance. Does that make Sol that much less interpretable?

Rather, the more plausible answer here is that more capable (bigger, well-trained models that use more compute) can solve problems more efficiently, where “efficient” here means fewer tokens.

It’s also worth keeping in mind that a reasoning trace is not guaranteed to faithfully describe everything that happens inside the model. In my view, the only valid concern is that looped transformers purposefully mislead users by presenting “fake” reasoning traces more often than conventional transformers. But I don’t think we have any strong evidence that this is happening.

Now, Astra’s system card does state that there is also evidence of reduced monitorability of their reasoning traces, and there is a bit of regression relative to Sol. It’s mostly associated with shorter, less informative traces. But again, this doesn’t establish looping as the root cause. It could just be due to the shorter length in general, similar to the Luna vs Sol example above.

A few hours after I shared my thoughts about looped transformers with respect to hiding reasoning chains, Jakub Pachocki (OpenAI’s Chief Scientist) also shared the following clarification:

I want to prevent a race into unmonitorability kicked off by confused reporting. The depth of the computation graph for our present frontier models, including Astra, is within a factor of two of GPT-4. OpenAI has worked to preserve and utilize chain-of-thought monitoring since our very first reasoning models. We deeply care about this technique, as it can give us a view into how model alignment generalizes from its training distribution. I do think it is fragile and unfortunately trending in a negative direction, for reasons not contingent on architecture changes that I will write about soon. But there are things we can do to strengthen it, and it’s a core goal of our current research program.

The “confused reporting” likely refers to The Information’s aforementioned paragraph here, implying that the looping aspect does not have anything to do with chain-of-thought changes.

6. Looped transformer research

Lastly, I want to share some interesting papers related to looped transformer architectures beyond the ones we already discussed.

6.1 Latent reasoning

Related to the Universal Transformer, the 2025 Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach paper studies how a model can use additional loops at inference time. For this, they trained a relatively modest but also not super tiny 3.5B-parameter model on 800B tokens.

Instead of reusing the same block over and over again as in the Universal Transformer, it repeats a stack like in Nanbeige; however, in contrast to Nanbeige, it sandwiches this shared stack of four blocks between 2 initial and 2 final blocks.

Also, what’s different from Nanbeige is that the shared stack receives the output of the initial blocks at the start of every loop, in addition to the previous loop’s hidden state. These are concatenated and passed through a learned linear projection before entering the four shared blocks. You can think of this as giving the stack access to the same initial input representation on every pass. This whole layout is summarized in the figure below.

So, in short, this is an additional and interesting looped transformer variant.

latent-reasoning

Figure 19. Conceptual summary of the latent reasoning model in Geiping et al..

An interesting detail is that the researchers vary the number of loops during training. This prepares the model to work with different amounts of computation at inference time.

Here, during training, the loop count is randomly sampled. At inference, a fixed budget is chosen by whoever runs the model, such as 8, 32, or 64 loops. Additionally, they have an adaptive stopping mechanism for each token based on the next-token probability distribution. If the KL-divergence between 2 successive rounds is below a certain threshold, i.e., if the distributions are too similar, the looping is halted.

The overall benefit depends on the task. In their evaluations, HellaSwag performance largely levels off after about eight loops, while GSM8K and HumanEval benefit from more.

However, while the title of the paper mentions “latent reasoning”, the model can still generate a textual chain of thought. Looping just gives it additional computation before each output token.

6.2 Knowledge retrieval vs reasoning

There’s a useful distinction between storing information and using it to solve a problem. For instance, the Beyond Parameters: Exploring Virtual Logic Depth for Scaling Laws paper from June 2025 investigates this by measuring memorization and reasoning in an LLM separately.

First, in the memorization experiments, looping leaves the amount of stored information nearly unchanged when the parameter count stays fixed. Increasing the number of distinct parameters does increase this capacity. From this, we can conclude that looping doesn’t add or let’s the model retrieve more knowledge. This makes sense. Information retrieval is a relatively simple task once the information is stored. Also, looping in itself is computing not “storing” mechanism.

Second, in separate reasoning experiments, reusing the blocks improves performance on multi-step math problems without adding parameters. This is interesting. Here, we can conclude that extra computation can help a model solve problems even when it doesn’t have more space to store information. But again, bigger models can also improve reasoning (although they add parameters as well).

capacity

Figure 20. In this memorization test, capacity grows with parameter count but changes little with additional block applications. Annotated figure from Zhu et al.

6.3 Looping at a matched compute budget

The just-released SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers paper from September 2026 comes back to the cost comparison from section 2.2. What happens if we compare looped and conventional transformers with approximately the same compute per token, total non-embedding parameters, and KV cache requirements?

The researchers use a mixture-of-experts architecture and apply the middle half of the transformer blocks twice, kind of similar to Nanbeige except with the sandwiching in Latent Reasoning.

However, they narrow the hidden dimension to compensate for the compute needed for the extra block applications. And then, because that makes the parameter count smaller, they then add experts to recover the total parameter count. They also adjust the attention head configuration to keep the KV cache comparable.

SMELT worked example comparing model width, experts, block applications, parameters, compute, and KV cache

Figure 21. SMELT overview from the example given in SMELT, section 3.2.

The experiments scale up to 54B non-embedding parameters and so on. Then, from fitted scaling curves, the researchers estimate that SMELT requires about 6.8-18% less training compute to reach the same validation loss within the studied compute range.

So, this answers the question of whether looped transformers are worth it computationally: Yes! They give us a slightly better model when using the same compute budget.

6.4 Full-bandwidth transformer

Finally, the also very recent Full-bandwidth transformer paper from August 2026 studies recurrence across token positions. At each decoding step, it combines the previous token’s final hidden state with the newly sampled token’s embedding through a learned gate. This becomes the input for the next forward pass.

So, the next token’s computation has access to the previous token’s final representation from the bottom of the stack, which is somewhat similar to Latent Reasoning.

When using a 1B base model, they found that their latent feedback approach outputs shorter reasoning traces on MATH500 while maintaining or improving accuracy. However, the shortening effect disappears after instruction tuning.

length

Figure 22. Latent feedback shortens reasoning traces in the base model, but this effect disappears after instruction tuning. Adapted from Wang et al., Figure 6, CC BY 4.0. Definitions and caveats added.

Anyway, this is interesting because this connects directly to the earlier discussion about whether looping results in shorter reasoning traces. The result depends on both the feedback mechanism and how the model is trained, of course. Also, the experiment doesn’t establish whether those shorter traces are less faithful.

Also, the big caveat of the study is that they didn’t test whether increasing the size of the model in conventional ways (adding more transformer blocks instead of looping) has a similar effect on the reasoning trace lengths.

Conclusion

To wrap it all up, we can say that yes, OpenAI GPT-6 Astra is a very strong model. And it’s making a particularly large leap in computer use. I believe computer use will be the next big focus area for open-source and proprietary harnesses in the upcoming months. I find open-source especially important when it comes to computer use, as “with great power come great responsibilities”, and it’s nice to be able to audit the harness before giving it access to my main computer.

Besides, GPT-6 Astra is likely to use a variant of the looped transformer. Looped transformers simply give better modeling performance at a fixed compute budget.

Also, better modeling performance may decrease in shorter reasoning chains. But this is not a new trend. We have always seen that within a model family with models of different sizes (e.g., GPT 5.6 Luna versus Sol).

In my opinion, shorter reasoning traces are a side effect of more “intelligent” or capable models that make fewer mistakes and can access more compute internally inside their architecture versus using a reasoning trace as a scratchpad. In a sense, the same is true for humans. During an in-person college math exam, a smart and well-prepared student likely requires less use of the notepaper and needs to backtrack less often, and so on.

---

Thanks for reading and supporting my work!

If you’d like to learn how to build reasoning models yourself, check out my book Build a Reasoning Model (From Scratch). We start with a pre-trained LLM and add reasoning capabilities step by step, with code you can run and experiment with. It’s both fun and rewarding, and a good investment in future-self to build the fundamentals to keep up with the AI field.

Also, if you’ve read one of my books, I’d appreciate a short, honest review on Amazon. Reviews help other readers decide whether a book is right for them and are a simple way to support authors.

scratch

Figure 23: Selected illustrations from my Build a Reasoning Model (From Scratch) book, covering inference-time scaling, distillation, and reinforcement learning.

We don’t know yet how AI will reshape the economy. Will it lead to unprecedented growth? Widespread unemployment? Neither, or something else? How can we tell?

Anthropic’s Economics team built a model of how AI might affect jobs, growth, and unemployment in the US in coming years. Read about possible economic futures and make your own predictions about AI capabilities to see the economy they imply.

We study how AI is reshaping the economy because we’re committed to ensuring that this transition is beneficial for society, including workers. By providing better visibility into our possible economic future, we can take steps to make sure that everyone benefits from it.

While our Economic Index measures how AI is being used across the economy right now, this scenario explorer is about looking ahead. Based on our technical report, Economic Scenarios for Transformative AI (Korinek et al., 2026), this explorer gives you a chance to find out what the economy might look like as AI continues to get more capable.

In scenarios ranging from business as usual to an economy where AI increases growth to about twice the normal rate, unemployment stays within the historical range and wages remain flat or rise depending on the industry. But in scenarios where growth is faster than anything in economic history, there are adverse impacts on wages and job prospects for knowledge workers. In those scenarios, society is far wealthier, so the challenge is making sure that the gains are broadly shared.

As you scroll down, you’ll see an overview of how AI affects the economy. Then, you can plug in your expectations for how capable AI will be, and how extensively it will be used across the economy in the future. The model will show you what the economy in 2030 might look like if your predictions come true—and how your predictions compare to others.

The economy is made out of tasks

This model represents all the jobs people do in the economy as bundles of tasks. AI can help people do a given task better or faster. It can automate the task. It might not affect the task at all. And it can lead to new tasks.

Think of a day in the life of a nurse

You can think of any job as a bundle of tasks that someone does. She does rounds to check on a sick patient. She draws blood. She triages incoming patients, charts patients’ vitals, and orders supplies for the ward. That’s just the start of the list.

Each of the tasks listed are based on the US Department of Labor’s O*NET taxonomy, listing the tasks for each occupation.

Jobs change over time

Tasks leave the bundle (hardly anyone hand-writes paper charts anymore) and new tasks arrive (30 years ago, no one monitored patients remotely). The bundle of tasks isn’t static, and the job changes as tasks change.

Only humans can do some tasks

For instance, AI can’t bathe a patient.

Some tasks will get augmented

AI helps a human do them better, or faster. AI helps the nurse draft discharge instructions, monitor patients remotely, and plan the shift’s care schedule.

Some tasks may get fully automated

For instance, AI may chart a patient’s vitals, or order the ward’s supplies.

And new tasks will appear

Historically, new technologies have also created new tasks for workers. For a nurse, that might be checking how well an AI triages patients, or reviewing an AI-proposed care plan.

The result: the nurse’s job changes

As the nurse incorporates AI into her job, the nurse is able to oversee and accomplish more. She can spend more time talking with patients and helping them understand diagnoses. Productivity increases.

Every task happens millions of times every day, across the country

Nurses are doing their work on every ward and on every shift. As more nurses use AI, AI supports a higher percentage of these millions of instances of each task.

From tasks to the economy

Today, if you add up every single instance of tasks performed in the US, by people and by the machines and software they work with, you get the US economy: over $30 trillion of value created over the past year. So how will AI shape the economy of the future?

The answer depends on how AI affects all the tasks that make up the economy, the new tasks it creates, and how fast AI takes on this work. Will AI lead to more task augmentation or automation? How much more productive will it make us? How quickly will it be adopted by workers and companies? The answers to these questions have direct effects on GDP, the labor market, and the share of the pie taken home by workers.

There are many possible futures, but we’re highlighting three scenarios

The future will depend on how AI’s capabilities advance, and how industries and workers adopt those capabilities. The three scenarios we share capture distinct kinds of impact.

In the modest scenario, it’s hard to see the effect of AI in macroeconomic data: its economic impact is something like the internet’s. In the substantial scenario, AI makes a bigger impact than the internet, or the railroad. And in the extreme scenario, AI drives a completely transformed, unprecedented economy, likely driven by recursively self-improving AI systems and a faster rate of AI adoption.

Small economic gains

In the modest scenario, AI has roughly the same kind of impact as the internet did. It drives real economic gains, but they’re within the historical norm for new technologies, and they arrive gradually.

A revolution in knowledge work

In the substantial scenario, AI is capable of doing half of all knowledge work by 2030, the majority of it autonomously, but it’s not adopted for all of that work: most knowledge work tasks are still done without AI. The economy grows at twice its normal rate. Wages for knowledge workers don’t rise, but other workers see gains.

A profound economic transformation

In the extreme scenario, AI is more productive than humans at the vast majority of knowledge-work tasks. It does nearly all of them autonomously, and it creates essentially no new knowledge tasks for people. This scenario would likely require recursively self-improving AI, adopted quickly for knowledge work.

As AI diffuses, annual GDP growth rates reach 15% a year, leading the economy to double in size every 4.5 years. As a society, we’re far richer than we’ve ever been, but many fewer workers have jobs in knowledge work, and unemployment has risen beyond typical recessionary levels.

How might powerful AI change the economy?

How do you think AI development will go over the next few years? And what would that path mean for the economy? We invite you to consider these questions, and explore potential answers with our scenario explorer.

People’s expectations about future AI capabilities vary.

In August, we surveyed more than 10,000 Americans about their views on present and future AI capabilities, adoption, and the ease of finding new work if they have to change occupations.

The typical respondent’s answers imply outcomes close to the “substantial change” scenario: GDP is 10% higher by 2030 than it would be without AI, and the overall unemployment rate has risen to around 5%. Around 10% of respondents have views in line with the extreme scenario.

You predicted one possible future for the economy. Here’s what that future could look like.

Finding 1: GDP growth

AI grows the economy in every scenario, but some more than others

AI drives GDP growth in all scenarios, although the scale varies enormously depending on the scenario.

US GDP in 2030, by scenario (measured in trillions of dollars)*

*GDP is calculated at 2025 price levels

But growth isn’t the only economic dynamic we care about. What would these potential futures mean for how much of this growth workers receive in their paychecks, or how many people have to find new jobs?

This model isn’t a complete map of reality, but it shows us some interesting findings. The country’s GDP will grow, but a larger share of that prosperity might go to the resources and technology used to create more wealth (capital) compared to workers, even if society as a whole is much wealthier.

And in most scenarios, job reallocation and unemployment both stay within ranges history has seen before, with one exception. In the extreme scenario, if we see recursive self-improvement and rapid adoption, unemployment could spike to historic levels.

Finding 2: Job reallocation

In more transformative scenarios, more workers have to change occupations. That may mean higher unemployment.

There is always some churn in the job market—people losing jobs and finding new ones. In normal times, this process can be painful, but works relatively well from a macroeconomic perspective. Most job seekers find new jobs fairly quickly.

In our substantial and extreme scenarios, knowledge workers may see a lot of automation and displacement. At the individual level, it means coders and call service center agents may have to switch to jobs like electrician and nurse, which are less exposed to AI.

But changing occupations entirely is hard, and it takes many people a long time to land a new job. The more of this switching a scenario requires, the more people will be between jobs.

Where workers are in 2030 (percent of all workers)

As we progress from 2026 to 2030, the number of jobs available in occupations AI affects (knowledge work) decreases, while the jobs available in occupations AI doesn’t affect increase.

Switching to a new occupation is difficult for a few reasons: workers may not want to change occupations. They may need to learn new skills. And even when they do, it’s not easy to get a new job. In the extreme scenario, as large swathes of knowledge work are automated more quickly, affected workers may be unemployed for a prolonged period.

Unemployment in knowledge work rises; in other occupations, it falls

Finding 3: Wages

Across the three scenarios, average wages rise, but this increase is concentrated in occupations outside of knowledge work.

That’s because it takes time for workers to switch to occupations where demand is rising. If there’s less demand for human knowledge work, that puts downward pressure on wages. Meanwhile, as AI increases productivity within knowledge work, the demand for manual work that benefits from that productivity will increase. For example, more quickly producing designs and permitting for physical infrastructure could increase the number of construction projects, resulting in rising demand for construction workers, which pushes those wages higher. In the substantial scenario, wages for knowledge workers are essentially flat. In the extreme scenario, they fall by more than 10% by 2030.

Pay by occupation group, percent above the same economy without AI

Finding 4: Labor vs. capital share

The pie will grow, but a larger share might go to capital

Today, of each dollar the economy produces, about 60¢ goes to workers and 40¢ go to capital. If the economy grows, but AI automates more tasks, more of each dollar might go to capital. This can happen even when wages for all workers rise substantially. If capital becomes more useful for more things, it will be in higher demand, which raises its price. In that world, more of the gains from a growing economy flow to owners of capital.

We find that the labor share falls noticeably in the substantial and extreme scenarios, and the capital share rises. Average wages rise—non-knowledge workers are paid much more—but wages for knowledge workers stagnate or decline alongside worsening unemployment.

In the extreme scenario, the gains from a rapidly expanding economy are unevenly distributed. Most knowledge workers face either lower wages or unemployment, and workers overall get a smaller fraction of the larger pie. Total labor income is barely changed by 2030.

In this scenario, the main challenge is not achieving economic growth, but making sure the benefits are broadly shared and the costs aren’t unequally dispersed.

How GDP is shared between workers and capital

With each scenario, the total economy grows (measured in GDP). But more of the growth goes to capital, compared to the amount people receive in their wages. Some professions see their wages increase significantly, but overall, wages make up a smaller share of the country’s economic growth.

The future is not predetermined.

Ultimately, what the economy looks like in 2030 depends on many factors, like what AI can do, and how companies and workers choose to adopt it. It also depends on how the financial benefit of this technology is shared.

Like any economic model, this one has limits. For example, we did not include scenarios where humanity develops hyper-capable robots. The model draws on our research and external review, and we’ll keep adding to it as the evidence develops.

This model, alongside our full research portfolio, will inform the research Anthropic funds to identify effective interventions for labor market disruptions. It’ll also inform the policy ideas we propose, with the goal of ensuring that the economic benefits of AI are broadly shared across society, both in the US and around the world.

Disclaimer and thanks to reviewers

v1.0 of the Econ Scenario Explorer, September 2026

The economic scenario explorer is currently Version 1.0. Like every model, it is a stark simplification of a complex reality: it isolates a few key forces and omits many others that may become relevant and important in the coming years. For example, it leaves out policy responses, business cycles, potential aggregate demand or financial market disruptions, and possible catastrophic risks. The scenario explorer is a work in progress, and we expect it to evolve both as we invest more time and as economic research itself develops.

We are grateful to the economists who read an early draft of the technical report that lays out the framework behind the explorer, Economic Scenarios for Transformative AI, and gave us detailed comments: Daron Acemoglu, Lukas Althoff, David Autor, Tom Cunningham, Lukas Freund, Joe Hazell, Ben Jones, Pete Klenow, Danial Lashkari, Kurt Mitman, Ben Moll, Emi Nakamura, Pascual Restrepo, David Romer, Jón Steinsson, Chris Tonetti, Ludo Visschers, and David Wiczer. Their feedback was generous and candid, and it has already improved our model. Two examples: several reviewers noted that advances in AI may raise the returns to capital, so that more of the gains flow to the owners of capital; and several noted that the wages of workers in the occupations AI affects most may diverge from those in occupations it barely affects. Both channels are now part of the scenarios. External reviewers were not asked to endorse our conclusions, and any remaining errors are ours.

Other criticisms are still open, and we plan to address many of them in future versions. Reviewers pointed out that the model does not follow individual workers, so it can only paint a very coarse picture of the costs of job displacement. Some questioned whether occupations exposed to AI will shrink at all rather than grow. Some felt the most extreme scenario is better read as a thought experiment than a scenario, while others felt the most modest one understates what is already visible in the data. Several asked us to be clearer that the model does not include the aggregate demand effects driven by the data center buildout. And more than one reviewer argued that we may be underestimating how much AI could accelerate technological progress itself. We agree that many of these are limitations of the current model. The scenario explorer should be viewed as a tool for thinking about what different technological developments would imply for the economy—actual outcomes may differ materially.

Anton Korinek, Chad Jones, Szymon Sacher, Tess Cotter, and Peter McCrory developed the economic model and co-authored the companion technical report. Santi Ruiz wrote this piece with them, with editorial support from Sarah Pollack and Adam Farina. Kelsey Nanan designed and built the interactive experience, with visual design and art direction by Nikki Makagiansar and Monika Tuchowska; Kyle Turman and Szymon Sacher built the scenario explorer and led the translation of the model into interactive form; Fayaz Ashraf and Ryan Heller contributed engineering, and Kim Withee and Maria Gonzalez supported production. Szymon Sacher and Tess Cotter designed and fielded the accompanying surveys with Morning Consult, with support from Ben Fowler. Peter McCrory, Anton Korinek, and Charles Yang coordinated the project, and Jack Clark provided direction throughout. Miriam Chaum, Jack Clark, Saffron Huang, Maxim Massenkoff, and Peter McCrory helped originate this effort. Jim Baker, Shan Carter, Johannes Hermle, Zoë Hitzig, and Eva Lyubich provided feedback.

This week, across Thoughtworks Europe, we took 10 engineers and put them in one room in our Barcelona office. The goal was to see how far and how fast we could go if we really leant into agentic engineering. We called it hyper-agentic. Along the way, we accidentally re-discovered something about coordinating agents.

The 10 engineers were given the goal of building an airline IROps system. This is the system that airlines use in a flight control centre when something goes wrong. When a plane has a technical fault that needs to be repaired. When a crew member gets sick and the crew needs to be replaced. It’s how they decide which flights to cancel, which planes to swap, which passengers get offloaded and put into hotels, etc. etc. They’re doing this over hundreds of aircraft, hundreds of thousands of passengers and many, many crew across multiple stations and airports. It’s a really, really hard problem, hard to solve, hard to execute. An IROps system is complex to build, complex to understand, and complex to use.

We managed to build one in four days. But this post isn’t about how we did that.

We started with a specification and a simulated airline, because this was a practice exercise, not a real client. We tried a couple of things just to see what would and wouldn’t work. We used a monorepo. All the engineers began working at once. We just got started, and after a couple of days we began to see things happening in interesting ways, things emerging.

Emergent behaviour from tuning our approach

With lot of agents working in one repo, build pipelines suffered. To deal with this we introduced a discipline: our agents were to continually commit and rebase from main. At first, we required a rebase after commit and to then push, with all of the build checks and controls in place. We introduced this change to catch build failures locally: integrate early and often. But, there was a side-effect. We were directing the agents to plan, to scope work to sections in the spec and to create plans linked to those sections. These plans were stored in the repo. All agents were working off the same spec using the same numbered and identified sections. As agents worked, plans were updated to record progress. These updates, alongside all others, were swept up with the new commit discipline. Agents were able to see other agents’ progress.

So, one agent was, say, working on the evaluator, the component to determine if a particular plan to restore operations is valid, whether it breaks hard constraints or soft constraints, etc. At the same time another agent was working on the search algorithm that looks for plans that could solve the disruption. The search component depends on the evaluator. Each component can be written together, but there is a shared interface and search depends on the evaluator.

The plans recorded these integration points. One plan said at this point I’m going to need to update the callers to call the real verifier. And on the search side, it said, at this point I need to insert the call to the real verifier when it arrives. Both agents could see each plan, and the progress. We realised that the agents were using the plans to coordinate. One agent would mark a line of the plan as in progress, the other agent would see that and not work on that line. When the first agent finished, the other agent would see not only that the work was complete and thus it was released to proceed, but would also be directly delivered notes on how the line had been implemented.

We started to exploit this. We’d kick off a session and direct it to work on a particular journey. One example was to introduce a cost model alongside the verifier. Knowing that someone else had been working on the cost model and pushing commits continually, we directed the agent working on the verifier to look at plans and source, monitor the repo, and when the work for the cost model lands start to integrate it. And it did.

This was entirely ad hoc. It was an accident of a series of decisions. We saw it happen. And then started to use it.

The repo as an accidental blackboard for agents

There’s a name for the pattern our agents had discovered: a blackboard system. This was something I explored back in my university days. My research thesis was in directing agent behaviour with hierarchical sensors. I was looking at applying modern machine learning techniques of the time, such as reinforcement learning, across large, dynamic datasets. Looking back over the literature, I adopted the blackboard pattern as the core coordination structure. This had previously been discovered in the development of the Hearsay-II system in 1980. It had been subsequently been developed into the more formal tuple space concept by Gelernter et al. in 1986.

A blackboard or tuple space is a shared memory that autonomous agents can read and write from independently. They read and write tuples with a certain minimum structure, and then as many extra fields as you want: no schema. It’s a very effective technique for coordinating autonomous problem solvers towards a single goal. They can each solve a decomposed part of the problem, drop their solution into the shared space, label it, and other autonomous searchers will find it, pick it up, and use it as part of their work.

We had accidentally prompted our agents to start using our repo as a blackboard. But it was an accident. It wasn’t an intentional act. It wasn’t fully structured. It was missing some of the key parts of how blackboards operate. And because it was accidental, I’m not convinced I would be able to reliably prompt our agents into doing it again. I’ve got a pretty good idea what we did, because we did some analysis and identified the single prompt that caused this cascade to start happening. But it was an emergent behaviour. It wasn’t a directed behaviour.

As well as creating this intentionally, rather than accidentally, I believe you want this communication channel to be sitting independently of source control. While we created it by directing a frequent push cycle, we backed-off from that. The frequent commits were overloading our CI pipeline. We switched to only push when a more coherent chunk of change was complete. This deprived the agents of the continuous flow of updates on progress.

A good accidental solution requires a good intentional project. I’ve started working on a project I’m calling Talwrn. That’s Welsh for a threshing pit, an area or space where arguments and conflict get worked out. This is aiming to be a blackboard for agentic engineering. My goal is a very simple to use tool that drops straight into your project and immediately offers a communication channel for agents to coordinate work. The first step is to get Talwrn to a point where it can support its own development. I’m planning to post about it regularly as I’m hoping to use it as a single, evolving example of how pure agentic engineering can proceed.

Illustration by Ariel Davis

For this week’s Infinite Scroll column, Brady Brickner-Wood is filling in for Kyle Chayka.

Garrett Langley, the founder of Flock Safety, frames surveillance as a Faustian bargain. If we want to live in a world where “crime is unsustainable” and “a thing of the past,” in his words, then we must “sacrifice our privacy.” Imagine: no more carjackings or kidnappings, murders or robberies, rapes or vandalism. “Everyone, in my opinion, deserves the right to be safe,” Langley said, during a TED talk last month. “It should not matter the color of your skin, your political affiliation, or your income level.”

Langley, who is thirty-nine, tells the story of creating Flock from the vantage point of a humble vigilante. In 2017, as his neighborhood, in Atlanta, became the target of repeated thefts, he called his local police precinct and pleaded with them to do something. The officer he spoke with lamented a lack of options; the best way to catch the perpetrators was to retrieve their license-plate numbers, but this was possible only if bystanders reported suspicious vehicles on their street, or if officers staked out the neighborhood and caught criminals in the act. Without evidence from such eyewitnesses, or any relevant footage or recognition software to identify the suspects, these carjackers were as good as invisible, stalking the night with impunity. Langley has characterized this conversation as a light-bulb moment. An already accomplished tech entrepreneur with a degree in engineering, he called up two of his former classmates, Matt Feury and Paige Todd, and began creating makeshift license-plate-reading cameras out of Android phones and solar panels that could be affixed to street poles. They sold their first models to homeowners’ associations and private communities but soon attracted the attention of law-enforcement agencies, who were interested not in shutting the project down but in collaborating. By 2020, Flock cameras were popping up on public roads across the country, in partnership with local police departments. Langley describes the company’s rapid growth as a product of organic, word-of-mouth enthusiasm: “That police chief is telling his friends; that mayor is telling her friends.” But, even as Flock was finding its footing, Langley was already envisioning something much larger: a nationwide surveillance network. Early indicators suggested that he was well on his way, and in the intervening years he has grown ever closer to achieving that goal.

Flock was recently reported to have a valuation of more than eight billion dollars, with its biggest investor being the venture-capital firm Andreessen Horowitz. Around forty per cent of the United States’ law-enforcement agencies now have contracts with the company, and a reported hundred and thirty thousand cameras monitor roads in every state but Alaska. Flock cameras have evolved from their initial prototype, and now read more than just license plates: they record the make, model, and color of passing vehicles, data that are then controlled by whatever entity operates the camera. Police can use this information, without obtaining a warrant or judicial approval, to make arrests or to present as evidence in court. For Langley and his legion of law-enforcement allies, such capabilities are proof of Flock’s limitless potential for catching and prosecuting criminals. Just last year, a mass shooter at Brown University was tracked using data from Flock cameras, a triumph touted by both the Providence police chief and the F.B.I. (The shooter was dead when he was found, but still, they said.) The company has claimed that its products have broken up crime rings, intercepted child abductions, and solved missing-persons cases. The International Association of Chiefs of Police has called automated license-plate readers (A.L.P.R.s) like Flock’s a “crime-fighting star.” Politicians across the aisle have expressed support for the technology, though those endorsements are complicated by reports of increased lobbying and political donations from the company.

The surveillance of it all is conveniently ignored by people in power who laud Flock, and by Langley himself. “I’ve never met anyone who doesn’t want to be safer,” Langley told NBC News, last month. “I’ve travelled the country, and I have not met that person yet.” Langley, it seems, has not been meeting the right people. In recent months, Flock cameras and the larger A.L.P.R. industry have faced mounting backlash, as people around the country, and across the political spectrum, denounce the spread of surveillance infrastructure into public life as undemocratic.

Almost half of Americans oppose A.L.P.R.s, according to the Washington Post, and some of them have taken to damaging, stealing, and defacing Flock cameras at parks, street corners, and traffic stops. When one Georgia sheriff posted a video to Facebook offering a thousand-dollar reward for information about the destruction of a Flock camera in a public park—“it was to keep our kids safe,” he said, from “sex offenders”—commenters ridiculed him, and lauded the assailant as a hero. “Looks like a Constitutional wind blew it over,” one user wrote; “Kudos to the citizen that threw tea in the harbor!” another added. The crowdsourced-data app DeFlock has become a leader in the anti-A.L.P.R. movement, providing a nationwide map of Flock cameras, alongside legal and community resources. Langley called the project “terroristic,” a comment he later tried to walk back. But the public tides seem to have turned against Langley’s hollow rhetoric of safety, and his evocation of terrorism rang alarm bells for anyone fluent in post-9/11 propaganda-speak. Telling people that they need to compromise on privacy is a polite way of framing a nonconsensual takeover; calling a community opposition movement “terroristic,” however, gives the game away. In her 2019 book, “The Age of Surveillance Capitalism,” Shoshana Zuboff diagnosed this tension, showing how concessions around privacy expose the paradox within Big Tech’s continual promise of progress: “The precise moment at which our needs are met is also the precise moment at which our lives are plundered for behavioral data, and all for the sake of others’ gain. The result is a perverse amalgam of empowerment inextricably layered with diminishment.”

Even leaders in the country’s most reliably red states, heeding the cries of their constituents, are now cutting ties with Flock. No one wants to get spied on, it seems, even Blue Lives Matter conservatives. Late last month, the Texas governor, Greg Abbott, instructed state agencies to freeze spending on Flock cameras; a few days later, Florida’s governor, Ron DeSantis, revoked permits for A.L.P.R.s on state highways and ordered their removal. Republican officials in tough-on-crime states such as Indiana, Kansas, and Tennessee have also endorsed A.L.P.R. restrictions, invoking the public’s right to privacy amid growing evidence that the technology is being regularly abused. Police officers have been accused of exploiting Flock to stalk ex-girlfriends, track love interests, and monitor family members. A sheriff’s deputy in Texas searched a data pool of more than eighty thousand Flock cameras to incriminate a woman who had allegedly received a medication abortion. ICE has also tapped Flock data with the help of municipal police departments in order to hunt down and detain immigrants. Langley has distanced himself from such deployments of his product by performing the role of the detached inventor. He told Politico that he viewed his “job as building tools and then giving those which we elect democratically the control to decide how they’re implemented.” He “didn’t want to have” the “responsibility” of regulating his products and making difficult ethical determinations, as he admitted to the BBC.

Listen to Langley talk long enough, and a core contradiction in his messaging emerges: he moralizes about safety and societal progress, about how his company is primed to change the course of human history, yet, when faced with genuine concerns about how his product creates less safe conditions for citizens, by repressing their autonomy, freedom, and right to public space, he holds up his hands, shakes his head, and pretends that none of that is any of his business. Flock cameras are meant to be a munificent gift from him to us. All he’s asking for in return is everything.

Privacy has long been a contested subject in American law. The Constitution does not explicitly guarantee a right to privacy—the word itself never appears in the text—and courts have spent decades debating whether such a right can be derived from its provisions and, if so, whom and what it protects. The Fourth Amendment’s prohibition against unreasonable government searches and seizures is considered the foundation of American privacy law, and courts have, over time, extended protections to one’s cellphone and to reproductive and sexual behavior. But the concept of privacy, like that of freedom or of justice, is a supple one. The legal definition of the concept has shrunk or ballooned to meet the needs of political and capitalist forces, proving how malleable the notion could be when needed. As Jeannie Suk Gersen wrote for The New Yorker, in 2022, “The right to privacy should have been understood from the start as a prerogative of the people, establishing a zone where the state cannot readily trespass,” but, instead, became “a prerogative of the privileged, intent on keeping the general public at bay.” This tension—privacy as a right or as a privilege—has been intensified by national-security powers and by revolutions in digital technology. Writing for n+1, in 2013, Jill Priluck noted that “the expanded privacy rights of the 20th century have with the transition to digital data become obsolete in many situations.” She referred to Justice Samuel Alito’s concurrence in the case United States v. Jones, from 2012, which held that the Fourth Amendment applied to G.P.S. surveillance. “New technology may provide increased convenience or security at the expense of privacy,” Alito warned. “Many people may find the tradeoff worthwhile.”

Trade-off, security, safety—this was the language of the post-9/11 surveillance boom, and, lucky us, it never left. The journalist Richard Beck, in his book “Homeland: The War on Terror in American Life,” from 2024, summarizes the ends to which the fear of terrorism drove an unimpeded and irreversible increase in mass surveillance within the U.S. When the Patriot Act was introduced to Congress, a month after 9/11, a “remarkable spirit of bipartisan unity” allowed for “the biggest expansion of law enforcement detention and surveillance powers in history.” Even so, some Democratic senators, such as Pat Leahy, though they broadly supported the bill, were concerned that it granted the intelligence agencies too much power, and lacked sufficient transparency measures and judicial oversight. As the bill reached a logjam in Congress, Attorney General John Ashcroft began publicly to put pressure on the senators holding it up. “We think that there is a very serious threat of additional problems now,” he said during an appearance on “Face the Nation.” Later, he told CNN that he was “deeply concerned about the rather slow pace at which we seem to be making this come true for America. . . . Talk won’t prevent terrorism; tools can help prevent terrorism.” The tools Ashcroft was referring to were being developed in tandem with Silicon Valley executives—biometric data, facial-recognition software, omnipresent CCTV cameras, electronic-communication surveillance. “The kinds of surveillance the government wanted to implement dovetailed nicely” with Silicon Valley’s own goals, Beck observed, noting how these surveillance systems required unfettered digital access to everyone, not just perceived foreign antagonizers or known criminals. Anyone could be a terrorist, the thinking went, and so everyone must be monitored.

Mass surveillance can be difficult to comprehend or think plainly about, ensconced as we are in the digital cocoon of contemporary life. How are we to understand the scope, nature, and threat of systems we cannot see or touch? When we use social media, or accept online privacy-policy updates and agreements, it can be unclear how, exactly, we are being violated. Are we not consenting to these services? Are we not uploading images of ourselves and freely conversing with friends and colleagues, knowing the potential risks of being tracked by our government and tech overlords? How are we to engage with these digital platforms without feeling paranoid, or dissociating from the fact that we are potentially being watched by law-enforcement agencies and large corporations? Zuboff, in her book, described this quandary as “a psychic numbing that inures us to the realities of being tracked, parsed, mined, and codified.” In other words, we must convince ourselves that the digital communication and information systems we use are harmless and, indeed, beneficial, in order to keep using them. Opting out of online life has become nearly impossible given how crucial these tools have become in many professional, academic, and personal contexts. We’ve accepted the compromises of the internet mostly out of submission; there is no visible or viable path for existing in the modern world without in some way resigning ourselves to the realities of surveillance.

Langley, it seems, is hoping that Flock can become similarly indispensable in our increasingly tech-shaped social environment. By threatening us with the violent fantasies of an unsurveilled world, he is pushing something that strikes me as the antithesis of safety: a community ethic founded on suspicion, a closely monitored world with no exit. But, unlike internet data mining or nebulous government intelligence gathering, A.L.P.R. cameras are both visible and tangible, literal black boxes perched high on street poles, designed to watch us. Such tools of mass surveillance no longer feel shrouded in mystery: they are visual reminders of the endangerment of our anonymity and privacy, a monument to our inescapably technocratic society. Langley may have us believe that he is most interested in public safety, but Beck highlights a more obvious impulse behind the perpetuation of technology like Flock: “The motivations behind Silicon Valley’s pursuit of mass surveillance and bulk data collection don’t need investigating,” he writes. “They did it for the money.” As private equity continues to pour resources into Flock, and as its cameras multiply across the country, questions will persist about how much compromise, sacrifice, and trade-off we can withstand before nothing is left of us to give away. ♦

Most people can recognize the Visa and Mastercard brands. Chances are, you use one of their cards to transact every day. You may have some notion that most places (in the US) take both, but some places only take Visa (e.g. Costco), and vice versa.

So what do they do? Here’s Visa’s attempt to answer that question.

A few things they don’t do1:

• They aren’t the company that issues the card. Those are called card issuers.
• They aren’t a bank, though the cards you have are probably issued by one (Chase, Capital One, BofA, etc).
• They don’t distribute point of sale methods or online checkouts, which are done by payment processors.
• They aren’t responsible for onboarding or underwriting stores and merchants, known as merchant acquiring. This is done by banks offering merchant accounts, but increasingly offered by modern payment processors (Stripe, Square, Adyen)2.
• They don’t manufacture or print cards.
• Nor do they manufacture the point of sale hardware.

Instead, Visa and Mastercard are card networks3, facilitating card transactions by connecting the cardholders and issuers to the merchants and acquirers.

This forms a two-sided market of all the participants in a transaction4:

---
config:
  theme: 'neutral'
  fontFamily: system-ui, -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Oxygen, Ubuntu, Cantarell, 'Open Sans', 'Helvetica Neue', sans-serif;
  nodeSpacing: 20
  rankSpacing: 25
  flowchart: 
    subGraphTitleMargin: {bottom: 20}
    useMaxWidth: true
---
%% For some reason arrows are still showing up in the Obsidian preview, but not in Mermaid online editor
flowchart LR
    subgraph Issuing[**Issuing**]
        direction TB
        CI[Card issuer] --- CH[Cardholder]
    end
    subgraph Acquiring[**Acquiring**]
        direction TB
        MA[Merchant acquirer] --- M[Merchant]
    end
    Issuing ---- CN[**Card network**]
    CN ---- Acquiring

The key players in a card transaction.

The card network’s job is to enable card transactions, and also to grow participation in their networks.

This boils down to four key responsibilities:

1. Run the telecommunications network to route transaction messages.
2. Coordinate the banking network to move money and settle transactions.
3. Set incentives to encourage the use of the network.
4. Set and enforce rules of the network, including a mechanism for disputes.

For the rest of the discussion, we’ll focus on Visa, as it’s what I’m most familiar with from my years in the payment industry. Mastercard is more or less the same, with different names for things.

1. Run the telecommunications network

When we talk about networks, we think of the Internet, computers connected together by fiber and deep sea cables.

Card networks, being telecommunication networks, are no different. They maintain data centers and lease fiber optics cables to connect issuers and acquirers electronically. At their most basic technical level, Visa’s responsibility is forwarding transaction messages between its participating issuers and acquirers. Mastercard calls this activity “switching”, seeing itself as a network switch.

Visa takes its data centers very seriously. They are highly secure, redundant, and fitted to survive all kinds of disasters. From Inside Visa’s Data Center (Network Computing, 2013):

“The company’s flagship data center, dubbed Operations Center East, or OCE, is a 140,000-square-foot facility that Visa will only say is located “somewhere along the Eastern seaboard.”

“Not surprisingly, the facility, which is also designed to withstand earthquakes and gale-force winds up to 170 miles per hour, is locked down like a digital Fort Knox. The roads entering the complex have hydraulic bollards that can shoot up fast enough to stop a vehicle traveling up to 50 miles per hour dead in its tracks. (The road is too curvy to drive safely at higher speeds.) Visitors must pass through a security gate, be cleared by roving security teams, and then be subjected to a biometric scan before being admitted.”

And a 2012 headline from USA Today:

Top secret Visa data center banks on security, even has moat

That top secret location? In Ashburn, Virginia, conveniently located by Topgolf and Trader Joe’s.

I think the "moat" is the pool of water on the top center-left.

Authorization and clearing

When a card is used, the card network routes a transaction request, known as an authorization, from the merchant to the issuer. The issuer then approves or declines the request in a response. Card numbers, also known as Primary Account Numbers (PANs), are used like IP addresses5. The first 6 to 8 digits of the PAN identifies the card issuer and is called the Bank Identification Number (BIN)

An approved authorization places a temporary hold on the account for the amount of the transaction. Later, the merchant submits the final transaction amount (for example, adding tips written on receipts or voiding the transaction) to initiate the transfer of money, known as clearing6.

Consider: before this was done by computers, this was done by people via phone calls7 and mail.

2. Coordinate the banking network

In addition to a telecom network, Visa has a financial network of banks.

After a transaction is finalized, money on both ends must move to fulfill the transaction, known as settlement.

Visa’s second job is to route money for settlement by having financial relationships with each party. It can collect money from one and transfer to another.

To be efficient, Visa does net settlement: every day, each network participant’s debits and credits are totalled, and at the end of the day the net money is moved to or from each participant once.

For domestic transactions, moving money is relatively straightforward, thanks to central banks.

Importantly, Visa is also able to settle internationally, even handling currency conversion. Visa acts as an adapter between banking systems with its global banking relationships. This greatly simplifies international money movement for participants in its network — without Visa, each participant would need to manage their own international banking relationships.

---
config:
  theme: 'neutral'
  fontFamily: system-ui, -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Oxygen, Ubuntu, Cantarell, 'Open Sans', 'Helvetica Neue', sans-serif;
  nodeSpacing: 20
  rankSpacing: 60
  flowchart: 
    subGraphTitleMargin: {bottom: 20}
    diagramPadding: 10
    useMaxWidth: true
---
flowchart LR
    CI[Issuer in Country A] --> VisaA["Visa's bank in A"]
    VisaB[Visa's bank in B] --> MA[Acquirer in Country B] 
    subgraph **Visa**
        VisaA -.-VisaB
    end

Visa as an adapter

Shuffling this amount of money around and timing everything right is no easy feat. Visa faces non-payment risk in addition to maintaining a significant balance to cover payouts while waiting to receive settlement payments. From Visa’s 2024 annual SEC report:

Most U.S. dollar settlements are settled within the same day and do not result in a receivable or payable balance, while settlements in currencies other than the U.S. dollar generally remain outstanding for one to two business days, which is consistent with industry practice for such transactions. … As of September 30, 2024, we held $11.2 billion of our total available liquidity to fund daily settlement in the event one or more of our financial institution clients are unable to settle, with the remaining liquidity available to support our working capital and other liquidity needs.

The Company’s settlement exposure is limited to the amount of unsettled Visa payment transactions at any point in time, which vary significantly day to day. For fiscal 2024, the Company’s maximum daily settlement exposure was $137.4 billion and the average daily settlement exposure was $84.3 billion.

3. Set incentives

The key to this whole arrangement are the fees required to participate in the network, largely set by the network.

Let’s walk through an average credit card transaction in the US:

1. A cardholder pays for a product at a merchant for $100.00.
2. The merchant pays 2.5% ($2.50) of the transaction to their payment processor or merchant acquirer. The 2.5% is the merchant discount rate or MDR.
3. The payment processor keeps 0.35% ($0.35), then pays 2% ($2.00) to the cardholder’s issuing bank and 0.15% ($0.15) to Visa. The 2% is the interchange fee, commonly known as interchange. The 0.15% is the network assessment fee. 8
4. The issuing bank keeps 2% ($2.00)!

Surprisingly, the issuing bank keeps most and the network takes the least, by an order of magnitude! This is because for the tranasction, the issuer is traditionally considered to take on most of the risk (although merchants are likely to disagree).

In addition to regulatory requirements, Visa and Mastercard offer zero-liability protection. This means that the issuing bank, not the cardholder, is liable for any charges made on a card if it is lost or stolen.9 The issuing bank also takes on credit risk, and must always pay for an approved transaction even if a cardholder cannot pay off their balance.

Out of these, the network sets the interchange and network assessment fee. Interchange fees vary dramatically based on the kind of card, category of spend, and even the metadata attached to a transaction. The network’s goal is to set fees that incentivize desired behaviors on their network, including using more secure payment methods (lowering interchange fees for merchants), or for companies to do more business spending (higher interchange fees on commercial credit cards).

Because of how much is given to the issuers, there are a lot of incentives for issuers to acquire customers and fund lavish rewards programs to encourage spending. This split also explains the recent rise of issuing processors, which make it easier for neobanks and fintechs to issue cards to access a more lucrative end of the market.

Why are merchants willing to pay this fee?10 The idea is that accepting card payments nets more customers and higher spending, due to convenience, consumer protections, and credit card rewards. More cards means more merchants, more merchants mean more cards, and more of everything is good for the network. In theory, the network benefits through fees, the merchants benefit through more purchases, and the consumer benefits through convenience.

The virtuous cycle of card spend.

In the EU, interchange fees are restricted to 0.3%, which explains the lack of rewards cards and wider acceptance of alternative payment methods like bank payments.

4. Set rules and handle disputes

Aside from incentivizing good behavior, card networks also need to regulate bad behavior on their networks. These rules are detailed in “Visa Core Rules and Visa Product and Service Rules”, a 923 page volume that is publically available. This is a tome detailing things like appropriate use of the Visa brand mark, detailed interchange data requirements, transaction processing timelines requirements, requirements and procedures for specific Visa services, and features issuers/acquirers should support.

For issuers and acquirers, who interact with the network, violating these rules could affect interchange rates, incur non-compliance fines (anywhere from $25k to $1M per month), or risk suspension from the network.

For cardholders and merchants, who interact with each other, networks provide a mechanism for resolving disputes between them. This could mean a fraudulent transaction, the product was not as promised, or a number of other possible reasons detailed in the rules. This is the process that happens behind the scenes when you call your bank to report fraud or request a chargeback.

When a payment method doesn’t have a dedicated dispute mechanism, the legal system is used to settle disputes11. This a problem with cash or bank payments that cards don’t have.

Arbitration

Visa is not actually involved directly in resolving most disputes. The rules essentially impose forced arbitration12. The issuer of the cardholder and acquirer of the merchant send evidence back and forth until one party yields and accepts liability for the transaction.

If both parties refuse to yield, then Visa reviews the dispute, charging a whopping $600 ($1000 for appeals) makes a decision based on the evidence provided (signature, security footage, receipts, etc) and the guidelines listed in the Visa Product and Service Rules.

The losing party pays the original transaction amount plus the review fee, so both parties have a lot of incentive to resolve it between themselves. Issuers often refund the cardholder themselves and write off the loss. Merchants proactively refund dissatisfied customers since they are charged a $15-30 processing fee by the acquirer upon receiving any dispute, even if they win.

Arbitration is not a fair system, but it is an efficient one.13

Conclusion

I hope you have gained some appreciation for the important role of card networks and some reasons for why cards are as popular as they are today. Use this knowledge to topple the V/MC duopoly, design your own payment method, or think about while you spend your money.

Edit: this reached the frontpage of Hacker News — hello, and welcome to town!

---

1. Nowadays they may have product offerings for some of these, or own subsidiaries that do some of these, but these are not core to the business of being a card network. ↩
2. Technically, these are “payment facilitators”. A bank underwites the payment company, and the payment company underwrites their merchants. The different roles in payments have historically been meaningful, but companies are increasingly blurring the lines, so these distinctions and terms are less interesting today. ↩
3. To be precise, we use “card networks” to colloqially refer to the companies operating their own card payment networks / schemes. For example, VisaNet is technically the network, Visa is the company/brand. Mastercard’s network is called Banknet. Both companies also own and operate specialized subsidiary networks for things like debit cards and ATMs, like Visa’s Interlink and Plus, or Mastercard’s Cirrus and Maestro, although technically VisaNet and Banknet can process debit (this is a story for another time). There are further terms to distinguish the telecommunications network with the bank network, and even subsets of each. Visa even thinks of itself as a “network of networks”. Turtles all the way down. ↩
4. Or four or more, depending on how you count. ↩
5. Similar to IPv4, 16 digit PANs are rapidly exhausting due to the use of anonymized “token” PANs used by things like Apple/Google Pay and saved payment details. ↩
6. Clearing includes finalizing/committing the payment and reconciliation against the authorization. “Capturing a payment” is what this is called from the merchant’s perspective. ↩
7. Known as a “voice authorization”. You might be able to get one today, although I don’t know if banks are staffing operators to field calls. ↩
8. Although interchange is short for interchange fee, technically interchange refers to the payment messages being routed by the card networks, and the fee is provided for that data. But, you almost never hear interchange used to mean payment messages except in technical specs. ↩
9. Although this creates trust in card payments, it has opened the doors to friendly fraud, where legitimate purchases are reported as fraudulent. Not to mention a moral hazard. ↩
10. Some merchants add a surcharge on card transactions to pass through the fee to customers. This used to be against Visa’s rules (and California state law) but lawsuits have challenged that. The surcharge amount is also only supposed to be the amount the merchant pays on a card transaction, but I suspect that many small merchants charge more. ↩
11. There are some amusing cases of this at work in large transactions, where the legal fees are much more in proportion with the transaction costs. ↩
12. Not that you asked, but the expansion of forced arbitration and confidential settlements are bad for society. Back in the day, you could sue your bank (for example) for doing bad stuff, but now contracts for everything include a clause for arbitration, waiving your rights as a consumer. The company pays for the arbitration, not you, so you can guess the outcome of that. Famously, Disney tried to use a forced arbitration clause in the Disney+ streaming agreement to prevent a wrongful death suit from a allergic reaction in a Disney restaurant. In the Visa case, the dispute process means the consumer tends to benefit, at the cost of the issuer and the merchant. ↩
13. The system is designed to encourage transactions rather than stopping fraud. It opens the door to fraud, especially friendly fraud, as explained in another footnote. Friendly fraud is a growing problem. ↩

Analytics from the June 2026 attack

In mid-to-late June 2026, Read the Docs experienced the largest and most sophisticated distributed denial-of-service (DDoS) attack in our history. At its peak, our infrastructure was hit with over 5.5 million requests per minute, about 100 times our normal baseline traffic.

The incident lasted for nearly ten days, testing our infrastructure, our edge defenses, and our incident response processes. Unlike simpler traffic floods we've seen in the past, this attack was more distributed, it adapted to our defenses rapidly, and it purposefully attacked areas that bypassed caching.

Now that our small ops team is back to sleeping at normal hours, we wanted to walk through the anatomy of this kind of attack, why our existing rate limiting only partially mitigated it, and what strategies actually helped us (mostly) maintain availability throughout the attack.

Evolution of DDoS attacks

Read the Docs has historically been very tolerant toward spiders and bots scraping documentation we host, and IP-based rate limiting solved most abuse problems. Starting about two years ago, we began seeing a significant uptick as AI crawlers became more prevalent and it seems other members of the dev infrastructure community are seeing similar issues. It became straightforward to plug an AI-generated scraper into a proxy network. Our defenses adapted to that fairly easily, but the June attack was over 10x larger than anything we had faced.

Key characteristics of this attack included:

• Massive volume: At peak, we received 5.5 million requests per minute, compared to our normal daily peak of under 100k requests per minute.
• Global distribution: We saw malicious requests originating from millions of unique IP addresses across hundreds of networks (ASNs) globally. This included residential IP blocks as well as major and minor hosting providers.
• Header & TLS randomization: The attackers systematically randomized HTTP request headers and TLS connection parameters to evade signature-based filters (JA3/JA4).
• Limitations of automated CDN defenses: Read the Docs uses Cloudflare and while Cloudflare's automated DDoS protection mitigated some traffic originating from what they called "known botnets", a big part of the attack passed that first check and got through to our rate limiting and WAF rules.
• Cache evasion: Attackers found and deliberately targeted URLs that resulted in cache misses, such as non-existent pages with unique paths (404s) as well as temporary redirects (302s).
• Adaptive behavior: When we implemented blocks or rate limits, the botnet adjusted its request rates, rotating through different target paths and spreading traffic across broader IP pools to probe our defense boundaries.

Scale and breadth

Previous minor DDoS attacks or large distributed scrapers we'd seen were typically concentrated in some way. The requests either originated from a small set of countries, or a small set of IP blocks, or they had a small set of browser signatures. This attack was truly global. It came from every country all at once, which is a nightmare when rate limiting rules are applied per Cloudflare colo. It's hard to craft rules that can limit a distributed attack while not hitting legitimate bots scraping at a reasonable rate from a single IP or subnet.

At one point when the attackers focused on redirects, they were overwhelming a hardcoded Nginx redirect (a simple rewrite regex directive) with enough traffic to cause dropped requests even on horizontally scaling infrastructure. An Nginx redirect like that can easily handle thousands of requests per second.

In addition to spreading across the globe, it also attacked multiple Read the Docs properties. Public community documentation was attacked as well as our commercially hosted docs. We also saw attackers try to take down our author-facing dashboards that require logins.

While it was an option (Cloudflare's "Under Attack Mode") to simply give every site visitor, legitimate or otherwise, a JavaScript challenge, we didn't want to do that. This would break every API integration and cause lots of friction for the hundreds of thousands of real docs readers. Instead, we relied on rate limiting and targeted challenges combined with more caching and pushing more features out to the edge.

“There's no way we could have handled this attack without Cloudflare.”

Adapting to our defenses

Read the Docs uses Cloudflare heavily for caching and rate limiting, and there's no way we could have handled this attack without Cloudflare. We have dozens of rate limiting rules (managed through Terraform) to protect our infrastructure based on IPs, on the thousands of hostnames and hundreds of thousands of subdomains Read the Docs hosts, on ASNs, browser fingerprints, and on combinations of all of these.

The attack started on a small number of domains where attackers discovered temporary redirects (302) that were not cached at the edge and were served by our Python backend rather than something like Nginx. Within a few minutes, our operations team had been paged due to a short outage (users may not always notice our outages because cached documentation keeps serving), and within half an hour or so we had moved these redirects to be served at the edge by Cloudflare instead of our servers. We thought that might be the end of it, but instead the attackers tried different tactics across various hosts and services for another week and a half.

Analytics showing oscillating "Yo-Yo" traffic levels during the attack.

Attackers would ramp up to discover our rate limit thresholds and then back off to let the rate limit windows expire. This is called a yo-yo pattern, and it's designed to maximize the financial costs of auto-scaled infrastructure and cause intermittent service degradation. The attackers knew we were running a web app firewall (WAF) with rate limits and knew how to cause as much damage as possible in spite of that.

Defending against volumetric DDoS attacks

Defending against multi-million request-per-minute floods requires a defense-in-depth approach with edge caching, web app firewalls, rate limiting, local caches, and request fingerprinting. The fastest request is the one served by the CDN or the web app firewall.

Edge caching

The first line of defense, and one we were already using heavily, is a proper CDN and ensuring that as few requests as possible hit origin servers. Serving docs from a CDN has a lot of benefits. For Read the Docs, where docs sites change infrequently, we cache fairly aggressively but purge the cache for a particular docs site whenever new documentation is pushed to git and rebuilt. CDNs also make fetching documentation much faster for people geographically further away from our origin servers.

However, attackers probing our defenses quickly discovered which requests were cached and which ones weren't by how fast the CDN responded. This means that finding just a few requests that aren't cached gives attackers an angle of attack. We are still finding more paths and endpoints that aren't cached, but even very short-lived cached responses (using the Cache-Control header) for both redirects and normal 200 responses will help with this kind of attack.

A Slack notification when Read the Docs is getting 45k uncached reqs/min. Without caching and rate limiting, auto-scaling infrastructure will just scale out to handle the load at our expense.

Rate limiting and fingerprinting

Since we didn't want to give a JavaScript challenge to all users, we used targeted rate limiting rules combining bot probability scores with per-IP rate limits to challenge suspicious traffic while letting legitimate users and well-behaved bots browse uninterrupted. Everyone who has solved a JavaScript challenge knows they cause a lot of friction. We decided that it was better to let some malicious traffic through and err on the side of not challenging real users. With that said, we still needed rate limits to protect our infrastructure.

Instead of focusing on where the request came from (the IP, the country), defenses must focus on what the request looks like:

• Cipher suite and TLS anomalies: Automated scrapers and bot clients frequently present abnormal TLS connections different from browsers. Cloudflare's bot detection has specific tools to detect these.
• Too many bad requests: Legitimate users and good bots are almost always served successful (200) responses, not redirects or 404s. Since 200s are always cached, they almost never present a problem. When we see too many more expensive requests like redirects or 404s, we begin rate limiting the browser fingerprint, the ASN, or even possibly the specific domain as a whole. Adding these rules, which we call the "penalty box", probably made the biggest difference in automatically mitigating the attack as it changed over time.
• Protocol inconsistencies: Malicious tools often declare modern User-Agent strings while using older HTTP/1.1 connections. Unfortunately, this inconsistency wasn't very useful in this attack, which was entirely HTTP/2 and HTTP/3.
• Client fingerprinting: Everyone using the Golang HTTP client or the Python requests module with the same TLS cipher suites will have the same JA4 fingerprint. This fingerprinting is specific to a browser or tool, not specific to a user. These fingerprints also weren't very helpful in this attack as the attackers were randomizing their TLS parameters.
• IP block classification: One area we are still working on is to classify more IP blocks into different categories with their own limits. For a service like Read the Docs which receives lots of automated traffic and wants to allow bots, we know we're going to get a lot of traffic from major cloud ASNs like Amazon, Google Cloud, and Azure. They should have higher limits than most residential or minor hosting providers.

Give users an escape hatch

One decision we made is to always give real users an escape hatch. Read the Docs very rarely issues outright blocks or bans to specific IPs or user agents. Instead, our "worst" is a JavaScript challenge, and if a user solves a challenge, they are very unlikely to get challenged again for the next day or so.

Lessons learned and key takeaways

The June 2026 DDoS attack reinforced several critical takeaways for running high-traffic infrastructure:

• IP blocking is obsolete for distributed attacks: Botnets or large scrapers use proxy services which make simple IP blocks useless. We already knew that, but this incident underscored it. Defenses need to have broader rate limits across more than just IPs (ASNs, hostnames, etc.).
• Aggressively cache: Cache everything, whether it's a simple static file, a 404, or a temporary redirect. Even setting a short cache window of a few minutes will ensure that these resources can't be used to attack our infrastructure. The default settings on the CDN and in most web frameworks are not what a service like Read the Docs wants.
• Protect cache-miss surfaces: Attackers actively search for non-cacheable paths (e.g. dynamic redirects, search endpoints, and 404s). Cache where possible, and if caching isn't feasible, try to handle as much on the edge as possible.
• Targeted challenges beat blunt instruments: Combining bot management heuristics with rate limits allowed us to mitigate the attack with minimal impact on legitimate users.
• Infrastructure as Code is essential: Managing edge and WAF rules via Terraform enabled us to review, test, version-control, and roll out complex filtering rules quickly and safely.

We're still seeing low levels of background traffic from the attack IP blocks, but our infrastructure is in a much stronger position today than it was before the attack. As AI tooling and proxy networks make attacks like this cheaper and more accessible, they are no longer reserved for big enterprise targets. They are becoming the baseline reality for any high-profile public service. Our ops team is back to getting a full night's sleep, but we're viewing this as more like an extended reprieve rather than attacks being a thing of the past.

Planet Labs' Open Satellite Feed

Planet Labs is a 15-year-old, San Francisco-based satellite manufacturer and constellation operator. They operate or have operated four different constellations throughout their history and capture imagery of the entire Earth's landmasses every day. As of this writing, LinkedIn lists the firm as having 1,751 associated members.

Their CEO, Will Marshall, is one of the firm's three co-founders. In February, he posted images alongside the Secretary General of NATO, Mark Rutte, French President Emmanuel Macron and Kaja Kallas, who was Estonia's Prime Minister between 2021 and 2024 and is now Vice-President of the European Commission.

Their latest Pelican-11 satellite, which will join their Tanager constellation, was deployed by SpaceX's Transporter 17 Mission on July 7th.

This is its release into Lower Low Earth Orbit (LLEO) over Greece.

As of this writing, Planet Labs' Ephemerides service lists 97 of their satellites that are still in orbit. These are their latest details from GCAT.

$ wget https://ephemerides.planet-labs.com/planet_mc.tle

$ grep '^2.*' planet_mc.tle \
    | cut -d' ' -f2 \
    > NORAD_IDs.csv

$ ~/duckdb

.maxrows 4000

SELECT   State,
         Name,
         * EXCLUDE(AltNames,
                   Bus,
                   Dest,
                   Attach_flag,
                   Subtype_flag,
                   Orbit_flag,
                   ID_flag,
                   Status_or_failure_flag,
                   Coarse_type,
                   SDate,
                   SDate_Year,
                   "Primary",
                   DDate,
                   Manufacturer,
                   Motor,
                   JCAT,
                   ODate,
                   Type,
                   OQUAL,
                   PLName,
                   Shape,
                   Name,
                   State,
                   DryMass,
                   TotMass)
FROM     'satcat.parquet'
WHERE    Satcat IN (SELECT column0::VARCHAR
                    FROM   'NORAD_IDs.csv')
AND      Status = 'In orbit'
ORDER BY Launch_Tag,
         Name;

┌─────────┬──────────────┬────────┬──────────┬───────┬────────────┬─────────────┬────────┬───────┬─────────┬─────────┬───────────────┬─────────┬────────────┬─────────┬───────┬──────────┐
│  State  │     Name     │ Apogee │ Diameter │  Inc  │ Launch_Tag │    LDate    │ Length │ Mass  │ OpOrbit │  Owner  │    Parent     │ Perigee │   Piece    │ Satcat  │ Span  │  Status  │
│ varchar │   varchar    │ int32  │  float   │ float │  varchar   │   varchar   │ float  │ float │ varchar │ varchar │    varchar    │ varchar │  varchar   │ varchar │ float │ varchar  │
├─────────┼──────────────┼────────┼──────────┼───────┼────────────┼─────────────┼────────┼───────┼─────────┼─────────┼───────────────┼─────────┼────────────┼─────────┼───────┼──────────┤
│ US      │ SkySat A     │    597 │      0.6 │ 97.81 │ 2013-066   │ 2013 Nov 21 │    0.8 │  90.0 │ LLEO/S  │ SKYBOX  │ S39450        │ 567     │ 2013-066C  │ 39418   │   0.6 │ In orbit │
│ US      │ SkySat C1    │    515 │      0.5 │ 97.51 │ 2016-040   │ 2016 Jun 22 │    1.2 │ 110.0 │ LLEO/S  │ TBELLA  │ S41620        │ 499     │ 2016-040C  │ 41601   │   0.5 │ In orbit │
│ US      │ SkySat C2    │    502 │      0.5 │ 97.42 │ 2016-058   │ 2016 Sep 16 │    1.2 │ 110.0 │ LLEO/S  │ TBELLA  │ S41775        │ 501     │ 2016-058D  │ 41773   │   0.5 │ In orbit │
│ US      │ SkySat C4    │    506 │      0.5 │ 97.43 │ 2016-058   │ 2016 Sep 16 │    1.2 │ 110.0 │ LLEO/S  │ TBELLA  │ S41775        │ 497     │ 2016-058B  │ 41771   │   0.5 │ In orbit │
│ US      │ SkySat C5    │    503 │      0.5 │ 97.42 │ 2016-058   │ 2016 Sep 16 │    1.2 │ 110.0 │ LLEO/S  │ TBELLA  │ S41775        │ 501     │ 2016-058C  │ 41772   │   0.5 │ In orbit │
│ US      │ SkySat C6    │    527 │      0.5 │ 97.35 │ 2017-068   │ 2017 Oct 31 │    1.2 │ 110.0 │ LLEO/S  │ PLABST  │ S42994        │ 500     │ 2017-068F  │ 42992   │   0.5 │ In orbit │
│ US      │ SkySat C7    │    527 │      0.5 │ 97.35 │ 2017-068   │ 2017 Oct 31 │    1.2 │ 110.0 │ LLEO/S  │ PLABST  │ S42994        │ 500     │ 2017-068E  │ 42991   │   0.5 │ In orbit │
│ US      │ SkySat C8    │    527 │      0.5 │ 97.35 │ 2017-068   │ 2017 Oct 31 │    1.2 │ 110.0 │ LLEO/S  │ PLABST  │ S42994        │ 500     │ 2017-068D  │ 42990   │   0.5 │ In orbit │
│ US      │ SkySat C9    │    528 │      0.5 │ 97.35 │ 2017-068   │ 2017 Oct 31 │    1.2 │ 110.0 │ LLEO/S  │ PLABST  │ S42994        │ 500     │ 2017-068C  │ 42989   │   0.5 │ In orbit │
│ US      │ SkySat C12   │    588 │      0.5 │ 97.77 │ 2018-099   │ 2018 Dec  3 │    1.2 │ 110.0 │ LLEO/S  │ PLABST  │ S43763        │ 572     │ 2018-099AR │ 43797   │   0.5 │ In orbit │
│ US      │ Flock 4q-16  │    531 │      0.1 │ 97.48 │ 2023-174   │ 2023 Nov 11 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11079  SD4.4 │ 516     │ 2023-174R  │ 58271   │   0.3 │ In orbit │
│ US      │ Flock 4q-2   │    529 │      0.1 │ 97.48 │ 2023-174   │ 2023 Nov 11 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11079  SD1.2 │ 512     │ 2023-174BS │ 58320   │   0.3 │ In orbit │
│ US      │ Flock 4q-4   │    530 │      0.1 │ 97.48 │ 2023-174   │ 2023 Nov 11 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11079  SD1.4 │ 517     │ 2023-174BZ │ 58327   │   0.3 │ In orbit │
│ US      │ Flock 4q-9   │    529 │      0.1 │ 97.48 │ 2023-174   │ 2023 Nov 11 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11079  SD3.1 │ 512     │ 2023-174BU │ 58322   │   0.3 │ In orbit │
│ US      │ Pelican 1    │    530 │      0.6 │ 97.48 │ 2023-174   │ 2023 Nov 11 │    2.3 │ 160.0 │ LLEO/S  │ PLAN    │ A11079        │ 515     │ 2023-174AS │ 58296   │   3.0 │ In orbit │
│ US      │ Flock 4be-1  │    515 │      0.1 │ 97.45 │ 2024-149   │ 2024 Aug 16 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11288        │ 506     │ 2024-149BC │ 60518   │   0.3 │ In orbit │
│ US      │ Flock 4be-11 │    517 │      0.1 │ 97.44 │ 2024-149   │ 2024 Aug 16 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11288        │ 507     │ 2024-149AK │ 60501   │   0.3 │ In orbit │
│ US      │ Flock 4be-13 │    517 │      0.1 │ 97.44 │ 2024-149   │ 2024 Aug 16 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11288        │ 506     │ 2024-149AW │ 60512   │   0.3 │ In orbit │
│ US      │ Flock 4be-14 │    516 │      0.1 │ 97.44 │ 2024-149   │ 2024 Aug 16 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11288        │ 505     │ 2024-149BB │ 60517   │   0.3 │ In orbit │
│ US      │ Flock 4be-17 │    517 │      0.1 │ 97.45 │ 2024-149   │ 2024 Aug 16 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11288        │ 509     │ 2024-149AA │ 60492   │   0.3 │ In orbit │
│ US      │ Flock 4be-18 │    519 │      0.1 │ 97.44 │ 2024-149   │ 2024 Aug 16 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11288        │ 509     │ 2024-149N  │ 60480   │   0.3 │ In orbit │
│ US      │ Flock 4be-19 │    516 │      0.1 │ 97.44 │ 2024-149   │ 2024 Aug 16 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11288        │ 505     │ 2024-149CZ │ 60563   │   0.3 │ In orbit │
│ US      │ Flock 4be-22 │    518 │      0.1 │ 97.44 │ 2024-149   │ 2024 Aug 16 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11288        │ 508     │ 2024-149AC │ 60494   │   0.3 │ In orbit │
│ US      │ Flock 4be-23 │    518 │      0.1 │ 97.45 │ 2024-149   │ 2024 Aug 16 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11288        │ 509     │ 2024-149Y  │ 60490   │   0.3 │ In orbit │
│ US      │ Flock 4be-24 │    517 │      0.1 │ 97.44 │ 2024-149   │ 2024 Aug 16 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11288        │ 507     │ 2024-149AP │ 60505   │   0.3 │ In orbit │
│ US      │ Flock 4be-25 │    517 │      0.1 │ 97.45 │ 2024-149   │ 2024 Aug 16 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11288        │ 510     │ 2024-149V  │ 60487   │   0.3 │ In orbit │
│ US      │ Flock 4be-28 │    516 │      0.1 │ 97.44 │ 2024-149   │ 2024 Aug 16 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11288        │ 505     │ 2024-149BD │ 60519   │   0.3 │ In orbit │
│ US      │ Flock 4be-29 │    515 │      0.1 │ 97.45 │ 2024-149   │ 2024 Aug 16 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11288        │ 508     │ 2024-149AT │ 60509   │   0.3 │ In orbit │
│ US      │ Flock 4be-3  │    516 │      0.1 │ 97.44 │ 2024-149   │ 2024 Aug 16 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11288        │ 506     │ 2024-149CX │ 60561   │   0.3 │ In orbit │
│ US      │ Flock 4be-31 │    517 │      0.1 │ 97.45 │ 2024-149   │ 2024 Aug 16 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11288        │ 509     │ 2024-149W  │ 60488   │   0.3 │ In orbit │
│ US      │ Flock 4be-32 │    518 │      0.1 │ 97.44 │ 2024-149   │ 2024 Aug 16 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11288        │ 508     │ 2024-149AF │ 60497   │   0.3 │ In orbit │
│ US      │ Flock 4be-34 │    517 │      0.1 │ 97.45 │ 2024-149   │ 2024 Aug 16 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11288        │ 509     │ 2024-149AD │ 60495   │   0.3 │ In orbit │
│ US      │ Flock 4be-5  │    515 │      0.1 │ 97.45 │ 2024-149   │ 2024 Aug 16 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11288        │ 507     │ 2024-149CV │ 60559   │   0.3 │ In orbit │
│ US      │ Flock 4be-6  │    515 │      0.1 │ 97.45 │ 2024-149   │ 2024 Aug 16 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11288        │ 506     │ 2024-149BA │ 60516   │   0.3 │ In orbit │
│ US      │ Flock 4be-7  │    518 │      0.1 │ 97.45 │ 2024 Aug 16 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11288        │ 510     │ 2024-149P  │ 60481   │   0.3 │ In orbit │
│ US      │ Flock 4be-9  │    515 │      0.1 │ 97.45 │ 2024-149   │ 2024 Aug 16 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11288        │ 507     │ 2024-149AY │ 60514   │   0.3 │ In orbit │
│ US      │ Tanager 1    │    516 │      0.6 │ 97.44 │ 2024-149   │ 2024 Aug 16 │    2.3 │ 150.0 │ LLEO/S  │ PLAN    │ A11288        │ 507     │ 2024-149AR │ 60507   │   3.0 │ In orbit │
│ US      │ Flock 4g-1   │    518 │      0.1 │ 97.45 │ 2025-009   │ 2025 Jan 14 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11398        │ 511     │ 2025-009BK │ 62666   │   0.3 │ In orbit │
│ US      │ Flock 4g-10  │    518 │      0.1 │ 97.44 │ 2025-009   │ 2025 Jan 14 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11398        │ 512     │ 2025-009BP │ 62670   │   0.3 │ In orbit │
│ US      │ Flock 4g-11  │    515 │      0.1 │ 97.44 │ 2025-009   │ 2025 Jan 14 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11398        │ 512     │ 2025-009W  │ 62629   │   0.3 │ In orbit │
│ US      │ Flock 4g-12  │    516 │      0.1 │ 97.44 │ 2025-009   │ 2025 Jan 14 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11398        │ 511     │ 2025-009S  │ 62625   │   0.3 │ In orbit │
│ US      │ Flock 4g-14  │    518 │      0.1 │ 97.45 │ 2025-009   │ 2025 Jan 14 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11398        │ 511     │ 2025-009BL │ 62667   │   0.3 │ In orbit │
│ US      │ Flock 4g-15  │    518 │      0.1 │ 97.45 │ 2025-009   │ 2025 Jan 14 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11398        │ 511     │ 2025-009BX │ 62678   │   0.3 │ In orbit │
│ US      │ Flock 4g-17  │    519 │      0.1 │ 97.44 │ 2025-009   │ 2025 Jan 14 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11398        │ 511     │ 2025-009CA │ 62681   │   0.3 │ In orbit │
│ US      │ Flock 4g-2   │    518 │      0.1 │ 97.44 │ 2025-009   │ 2025 Jan 14 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11398        │ 511     │ 2025-009BE │ 62661   │   0.3 │ In orbit │
│ US      │ Flock 4g-22  │    514 │      0.1 │ 97.44 │ 2025-009   │ 2025 Jan 14 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11398        │ 511     │ 2025-009AN │ 62645   │   0.3 │ In orbit │
│ US      │ Flock 4g-24  │    515 │      0.1 │ 97.45 │ 2025-009   │ 2025 Jan 14 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11398        │ 511     │ 2025-009AE │ 62637   │   0.3 │ In orbit │
│ US      │ Flock 4g-25  │    514 │      0.1 │ 97.44 │ 2025-009   │ 2025 Jan 14 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11398        │ 511     │ 2025-009AP │ 62646   │   0.3 │ In orbit │
│ US      │ Flock 4g-26  │    515 │      0.1 │ 97.45 │ 2025-009   │ 2025 Jan 14 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11398        │ 511     │ 2025-009AA │ 62633   │   0.3 │ In orbit │
│ US      │ Flock 4g-27  │    514 │      0.1 │ 97.44 │ 2025-009   │ 2025 Jan 14 │    0.3 │   97.44 │ LLEO/S  │ PLAN    │ A11398        │ 511     │ 2025-009AJ │ 62641   │   0.3 │ In orbit │
│ US      │ Flock 4g-28  │    515 │      0.1 │ 97.44 │ 2025-009   │ 2025 Jan 14 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11398        │ 511     │ 2025-009AB │ 62634   │   0.3 │ In orbit │
│ US      │ Flock 4g-3   │    515 │      0.1 │ 97.45 │ 2025-009   │ 2025 Jan 14 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11398        │ 511     │ 2025-009AG │ 62639   │   0.3 │ In orbit │
│ US      │ Flock 4g-31  │    513 │      0.1 │ 97.44 │ 2025-009   │ 2025 Jan 14 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11398        │ 511     │ 2025-009AV │ 62652   │   0.3 │ In orbit │
│ US      │ Flock 4g-36  │    519 │      0.1 │ 97.45 │ 2025-009   │ 2025 Jan 14 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11398        │ 512     │ 2025-009BU │ 62675   │   0.3 │ In orbit │
│ US      │ Flock 4g-6   │    517 │      0.1 │ 97.44 │ 2025-009   │ 2025 Jan 14 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11398        │ 512     │ 2025-009BD │ 62660   │   0.3 │ In orbit │
│ US      │ Flock 4g-7   │    516 │      0.1 │ 97.44 │ 2025-009   │ 2025 Jan 14 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11398        │ 512     │ 2025-009R  │ 62624   │   0.3 │ In orbit │
│ US      │ Flock 4g-8   │    516 │      0.1 │ 97.44 │ 2025-009   │ 2025 Jan 14 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11398        │ 511     │ 2025-009E  │ 62613   │   0.3 │ In orbit │
│ US      │ Flock 4g-9   │    520 │      0.1 │ 97.44 │ 2025-009   │ 2025 Jan 14 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11398        │ 512     │ 2025-009P  │ 62622   │   0.3 │ In orbit │
│ US      │ Pelican 2    │    516 │      0.6 │ 97.44 │ 2025-009   │ 2025 Jan 14 │    2.3 │ 160.0 │ LLEO/S  │ PLAN    │ A11398        │ 512     │ 2025-009Y  │ 62631   │   3.0 │ In orbit │
│ US      │ Pelican 3    │    464 │      0.6 │ 97.26 │ 2025-188   │ 2025 Aug 26 │    2.3 │ 160.0 │ LLEO/S  │ PLAN    │ A11588        │ 458     │ 2025-188A  │ 65315   │   3.0 │ In orbit │
│ US      │ Pelican 4    │    464 │      0.6 │ 97.26 │ 2025-188   │ 2025 Aug 26 │    2.3 │ 160.0 │ LLEO/S  │ PLAN    │ A11588        │ 458     │ 2025-188B  │ 65316   │   3.0 │ In orbit │
│ US      │ Flock 4h-10  │    518 │      0.1 │ 97.44 │ 2025-276   │ 2025 Nov 28 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11695        │ 512     │ 2025-276AZ │ 66713   │   0.3 │ In orbit │
│ US      │ Flock 4h-12  │    513 │      0.1 │ 97.44 │ 2025-276   │ 2025 Nov 28 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11695        │ 512     │ 2025-276BB │ 66715   │   0.3 │ In orbit │
│ US      │ Flock 4h-14  │    512 │      0.1 │ 97.45 │ 2025-276   │ 2025 Nov 28 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11695        │ 512     │ 2025-276BD │ 66717   │   0.3 │ In orbit │
│ US      │ Flock 4h-15  │    518 │      0.1 │ 97.44 │ 2025-276   │ 2025 Nov 28 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11695        │ 512     │ 2025-276BE │ 66718   │   0.3 │ In orbit │
│ US      │ Flock 4h-16  │    517 │      0.1 │ 97.45 │ 2025-276   │ 2025 Nov 28 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11695        │ 512     │ 2025-276BF │ 66719   │   0.3 │ In orbit │
│ US      │ Flock 4h-18  │    518 │      0.1 │ 97.44 │ 2025-276   │ 2025 Nov 28 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11695        │ 512     │ 2025-276BH │ 66721   │   0.3 │ In orbit │
│ US      │ Flock 4h-19  │    513 │      0.1 │ 97.44 │ 2025-276   │ 2025 Nov 28 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11695        │ 511     │ 2025-276BJ │ 66722   │   0.3 │ In orbit │
│ US      │ Flock 4h-2   │    513 │      0.1 │ 97.44 │ 2025-276   │ 2025 Nov 28 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11695        │ 512     │ 2025-276AR │ 66705   │   0.3 │ In orbit │
│ US      │ Flock 4h-20  │    513 │      0.1 │ 97.45 │ 2025-276   │ 2025 Nov 28 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11695        │ 511     │ 2025-276BK │ 66723   │   0.3 │ In orbit │
│ US      │ Flock 4h-21  │    513 │      0.1 │ 97.44 │ 2025-276   │ 2025 Nov 28 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11695        │ 511     │ 2025-276BL │ 66724   │   0.3 │ In orbit │
│ US      │ Flock 4h-22  │    517 │      0.1 │ 97.44 │ 2025-276   │ 2025 Nov 28 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11695        │ 512     │ 2025-276BM │ 66725   │   0.3 │ In orbit │
│ US      │ Flock 4h-23  │    518 │      0.1 │ 97.45 │ 2025-276   │ 2025 Nov 28 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11695        │ 512     │ 2025-276BN │ 66726   │   0.3 │ In orbit │
│ US      │ Flock 4h-24  │    517 │      0.1 │ 97.45 │ 2025-276   │ 2025 Nov 28 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11695        │ 512     │ 2025-276BP │ 66727   │   0.3 │ In orbit │
│ US      │ Flock 4h-25  │    512 │      0.1 │ 97.44 │ 2025-276   │ 2025 Nov 28 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11695        │ 511     │ 2025-276BQ │ 66728   │   0.3 │ In orbit │
│ US      │ Flock 4h-26  │    518 │      0.1 │ 97.44 │ 2025-276   │ 2025 Nov 28 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11695        │ 512     │ 2025-276BR │ 66729   │   0.3 │ In orbit │
│ US      │ Flock 4h-27  │    513 │      0.1 │ 97.44 │ 2025-276   │ 2025 Nov 28 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11695        │ 512     │ 2025-276BS │ 66730   │   0.3 │ In orbit │
│ US      │ Flock 4h-29  │    517 │      0.1 │ 97.44 │ 2025-276   │ 2025 Nov 28 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11695        │ 512     │ 2025-276BU │ 66732   │   0.3 │ In orbit │
│ US      │ Flock 4h-3   │    517 │      0.1 │ 97.45 │ 2025-276   │ 2025 Nov 28 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11695        │ 512     │ 2025-276AS │ 66706   │   0.3 │ In orbit │
│ US      │ Flock 4h-30  │    518 │      0.1 │ 97.45 │ 2025-276   │ 2025 Nov 28 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11695        │ 512     │ 2025-276BV │ 66733   │   0.3 │ In orbit │
│ US      │ Flock 4h-31  │    517 │      0.1 │ 97.44 │ 2025-276   │ 2025 Nov 28 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11695        │ 512     │ 2025-276BW │ 66734   │   0.3 │ In orbit │
│ US      │ Flock 4h-32  │    518 │      0.1 │ 97.45 │ 2025-276   │ 2025 Nov 28 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11695        │ 512     │ 2025-276BX │ 66735   │   0.3 │ In orbit │
│ US      │ Flock 4h-33  │    513 │      0.1 │ 97.45 │ 2025-276   │ 2025 Nov 28 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11695        │ 511     │ 2025-276BY │ 66736   │   0.3 │ In orbit │
│ US      │ Flock 4h-34  │    513 │      0.1 │ 97.45 │ 2025-276   │ 2025 Nov 28 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11695        │ 511     │ 2025-276BZ │ 66737   │   0.3 │ In orbit │
│ US      │ Flock 4h-35  │    512 │      0.1 │ 97.45 │ 2025-276   │ 2025 Nov 28 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11695        │ 511     │ 2025-276CA │ 66738   │   0.3 │ In orbit │
│ US      │ Flock 4h-36  │    514 │      0.1 │ 97.45 │ 2025-276   │ 2025 Nov 28 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11695        │ 511     │ 2025-276CB │ 66739   │   0.3 │ In orbit │
│ US      │ Flock 4h-4   │    518 │      0.1 │ 97.44 │ 2025-276   │ 2025 Nov 28 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11695        │ 512     │ 2025-276AT │ 66707   │   0.3 │ In orbit │
│ US      │ Flock 4h-5   │    513 │      0.1 │ 97.44 │ 2025-276   │ 2025 Nov 28 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11695        │ 511     │ 2025-276AU │ 66708   │   0.3 │ In orbit │
│ US      │ Flock 4h-7   │    516 │      0.1 │ 97.44 │ 2025-276   │ 2025 Nov 28 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11695        │ 513     │ 2025-276AW │ 66710   │   0.3 │ In orbit │
│ US      │ Flock 4h-8   │    517 │      0.1 │ 97.45 │ 2025-276   │ 2025 Nov 28 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11695        │ 512     │ 2025-276AX │ 66711   │   0.3 │ In orbit │
│ US      │ Flock 4h-9   │    515 │      0.1 │ 97.45 │ 2025-276   │ 2025 Nov 28 │    0.3 │   5.7 │ LLEO/S  │ PLAN    │ A11695        │ 511     │ 2025-276AY │ 66712   │   0.3 │ In orbit │
│ US      │ Pelican 5    │    515 │      0.6 │ 97.44 │ 2025-276   │ 2025 Nov 28 │    2.3 │ 160.0 │ LLEO/S  │ PLAN    │ A11695        │ 512     │ 2025-276B  │ 66667   │   3.0 │ In orbit │
│ US      │ Pelican 6    │    516 │      0.6 │ 97.44 │ 2025-276   │ 2025 Nov 28 │    2.3 │ 160.0 │ LLEO/S  │ PLAN    │ A11695        │ 513     │ 2025-276AP │ 66703   │   3.0 │ In orbit │
│ US      │ Edda 1       │    509 │      0.6 │ 97.41 │ 2026-100   │ 2026 May  3 │    2.3 │ 160.0 │ LLEO/S  │ PLAN    │ A11839        │ 500     │ 2026-100AP │ 69017   │   3.0 │ In orbit │
│ S       │ Pelican 8    │    518 │      0.6 │ 97.72 │ 2026-100   │ 2026 May  3 │    2.3 │ 160.0 │ LLEO/S  │ FVM     │ A11839        │ 482     │ 2026-100AQ │ 69018   │   3.0 │ In orbit │
│ US      │ Pelican 9    │    508 │      0.6 │ 97.41 │ 2026-100   │ 2026 May  3 │    2.3 │ 160.0 │ LLEO/S  │ PLAN    │ A11839        │ 500     │ 2026-100AR │ 69019   │   3.0 │ In orbit │
│ US      │ Pelican 11   │    597 │      0.6 │ 97.76 │ 2026-156   │ 2026 Jul  7 │    2.3 │ 160.0 │ LLEO/S  │ PLAN    │ A11917        │ 587     │ 2026-156C  │ 69871   │   3.0 │ In orbit │
└─────────┴──────────────┴────────┴──────────┴───────┴────────────┴─────────────┴────────┴───────┴─────────┴─────────┴───────────────┴─────────┴────────────┴─────────┴───────┴──────────┘

These are the spectrum response levels of their SuperDove and SkySat satellites compared alongside RapidEye, Landsat-8 and Sentinel-2.

The Hyperspectral Pelican constellation differs from their other constellations as its satellites capture 426 bands of light. I reviewed some of its free imagery last year.

Last year, Planet Labs announced a new Disaster Data feed. This is a Cloudflare-hosted, DuckDB & QGIS-friendly feed with select imagery they've captured from various natural and man-made disasters shortly after they've occurred. The metadata describing the images is shipped as Parquet and JSON files.

In this post, I'll analyse the imagery they've captured and published openly this year.

My Workstation

I'm using a 5.7 GHz AMD Ryzen 9 9950X CPU. It has 16 cores and 32 threads and 1.2 MB of L1, 16 MB of L2 and 64 MB of L3 cache. It has a liquid cooler attached and is housed in a spacious, full-sized Cooler Master HAF 700 computer case.

The system has 96 GB of DDR5 RAM clocked at 4,800 MT/s and a 5th-generation, Crucial T700 4 TB NVMe M.2 SSD which can read at speeds up to 12,400 MB/s. There is a heatsink on the SSD to help keep its temperature down. This is my system's C drive.

The system is powered by a 1,200-watt, fully modular Corsair Power Supply and is sat on an ASRock X870E Nova 90 Motherboard.

I'm running Ubuntu 24 LTS via Microsoft's Ubuntu for Windows on Windows 11 Pro. In case you're wondering why I don't run a Linux-based desktop as my primary work environment, I'm still using an Nvidia GTX 1080 GPU which has better driver support on Windows and ArcGIS Pro only supports Windows natively.

Installing Prerequisites

I'll use GDAL 3.9.3 to manipulate imagery and jq to help format data in this post.

Note: Last year, GDAL 3.12 was released and made a substantial overhaul to its CLI. It's likely the commands in this post would need to be adjusted to fit this or newer versions of GDAL.

$ sudo apt update
$ sudo apt install \
    gdal-bin \
    jq

I'll use DuckDB, along with its H3, JSON, Lindel, Parquet and Spatial extensions in this post.

$ cd ~
$ wget -c https://github.com/duckdb/duckdb/releases/download/v1.5.4/duckdb_cli-linux-amd64.zip
$ unzip -j duckdb_cli-linux-amd64.zip
$ chmod +x duckdb
$ ~/duckdb

INSTALL h3 FROM community;
INSTALL lindel FROM community;
INSTALL json;
INSTALL parquet;
INSTALL spatial;

I'll set up DuckDB to load every installed extension each time it launches.

$ vi ~/.duckdbrc

.timer on
.width 180
LOAD h3;
LOAD lindel;
LOAD json;
LOAD parquet;
LOAD spatial;

The maps in this post were rendered with QGIS version 4.2.1. QGIS is a desktop application that runs on Windows, macOS and Linux. The application has grown in popularity in recent years and has ~22M application launches from users all around the world each month.

The boundaries and place names were sourced from Natural Earth. The earthquake data is from the USGS. Maritime Boundaries were sourced from Marine Regions. I used QGIS' HCMGIS plugin to add satellite imagery basemaps from Bing to this post.

The List of Disasters

This is a programmatic way of getting the list of disasters. Each line represents the URL slug where the assets and event-specific documentation can be found.

$ touch catalog.json; rm catalog.json
$ wget https://data.source.coop/planet/disasterdata/catalog.json

$ jq -r '.links[]|.href' catalog.json \
    | grep '.*[0-9]/catalog.json' \
    | cut -d/ -f2 \
    | sort

colombia-earthquake-2026-08-10
gironde-wildfire-2026
hurricane-melissa-2025
nepal-flash-flood-2026-08-26
philippines-earthquake-2026-06-08
venezuela-earthquake-2026-06-24

Pre-event Basemaps

Each event's folder will contain a variety of assets. The types of assets available vary between the events. Two of the events, Colombia and Venezuela, have basemaps. These are mosaics of several satellite images captured over a recent period of time and aim to have a very low amount of cloud cover obscuring the Earth's surface.

The basemap for Venezuela covers 280 KM of coastline and up to 52 KM inland.

The basemaps are released as tiles and, in the case of Colombia, are shipped as 93 GeoTIFFs with corresponding JSON files for their metadata.

I'll download the list of basemap assets for Colombia below.

$ mkdir -p colombia_basemap
$ cd colombia_basemap

$ wget https://data.source.coop/planet/disasterdata/Colombia-earthquake-2026-08-10/pre-event/basemap-2026q2/items.parquet

This is an example record for one of the tiles.

$ echo "FROM  'items.parquet'
        LIMIT 1" \
    | ~/duckdb -json \
    | jq -S .

[
  {
    "assets": {
      "thumbnail": {
        "href": "https://data.source.coop/planet/disasterdata/colombia-earthquake-2026-08-10/pre-event/basemap-2026q2/items/585-1059/585-1059_thumbnail.png",
        "roles": [
          "thumbnail"
        ],
        "title": "Thumbnail",
        "type": "image/png"
      },
      "visual": {
        "eo:bands": [
          {
            "common_name": "red",
            "name": "Red"
          },
          {
            "common_name": "green",
            "name": "Green"
          },
          {
            "common_name": "blue",
            "name": "Blue"
          },
          {
            "common_name": "alpha",
            "name": "Alpha"
          }
        ],
        "href": "https://data.source.coop/planet/disasterdata/colombia-earthquake-2026-08-10/pre-event/basemap-2026q2/items/585-1059/585-1059_visual.tif",
        "roles": [
          "data",
          "visual"
        ],
        "title": "Visual (true-colour RGB) basemap quad COG",
        "type": "image/tiff; application=geotiff; profile=cloud-optimized"
      }
    },
    "bbox": {
      "xmax": -76.99218748925043,
      "xmin": -77.16796873922601,
      "ymax": 6.315298537512351,
      "ymin": 6.140554781656619
    },
    "collection": "pre-event-basemap-2026q2",
    "constellation": "planetscope",
    "datetime": "2026-05-16 15:00:00+03",
    "end_datetime": "2026-07-01 03:00:00+03",
    "geometry": "POLYGON ((-77.16796873922601 6.140554781656619, -76.99218748925043 6.140554781656619, -76.99218748925043 6.315298537512351, -77.16796873922601 6.315298537512351, -77.16796873922601 6.140554781656619))",
    "gsd": 4.77731426716,
    "id": "585-1059",
    "links": [
      {
        "href": "../../../../catalog.json",
        "rel": "root",
        "title": "Planet Crisis Response — Colombia Earthquake (2026)",
        "type": "application/json"
      },
      {
        "href": "../../collection.json",
        "rel": "parent",
        "title": null,
        "type": "application/json"
      },
      {
        "href": "../../collection.json",
        "rel": "collection",
        "title": null,
        "type": "application/json"
      },
      {
        "href": "https://data.source.coop/planet/disasterdata/colombia-earthquake-2026-08-10/pre-event/basemap-2026q2/items/585-1059/585-1059.json",
        "rel": "self",
        "title": null,
        "type": "application/json"
      },
      {
        "href": "./585-1059_thumbnail.png",
        "rel": "thumbnail",
        "title": null,
        "type": "image/png"
      }
    ],
    "pl:mosaic": "global_quarterly_2026q2_mosaic",
    "pl:mosaic_id": "649cf917-0aab-40eb-94df-c222535f2cf7",
    "pl:percent_covered": 100,
    "pl:quad_id": "585-1059",
    "proj:epsg": 3857,
    "proj:shape": [
      4096,
      4096
    ],
    "stac_extensions": [
      "https://stac-extensions.github.io/eo/v1.0.0/schema.json",
      "https://stac-extensions.github.io/view/v1.0.0/schema.json",
      "https://stac-extensions.github.io/raster/v1.1.0/schema.json",
      "https://stac-extensions.github.io/projection/v1.1.0/schema.json"
    ],
    "stac_version": "1.0.0",
    "start_datetime": "2026-04-01 03:00:00+03",
    "type": "Feature"
  }
]

I'll build a list of every tile's GeoTIFF URL and download them with eight parallel threads. The resulting GeoTIFFs take up 1.7 GB of space.

$ echo "WITH a AS (
            SELECT DISTINCT
                UNNEST([assets.visual.href]) AS url
            FROM READ_PARQUET('items.parquet')
        )
        FROM   a
        WHERE  url IS NOT NULL" \
    | ~/duckdb -csv -noheader \
    | xargs \
        -I% \
        -P8 \
        wget -c "%"

I'll then build a single GeoTIFF from those tiles and use WEBP compression. This can help reduce the file size 80%+ with minimal changes in appearance. The resulting GeoTIFF is 327 MB.

$ gdalbuildvrt mosaic.vrt *visual.tif

$ gdal_translate \
    -co NUM_THREADS=ALL_CPUS \
    -of COG \
    -co COMPRESS=WEBP \
    mosaic.vrt \
    mosaic.tif

I will then convert the JSON metadata files into a single Parquet file.

$ mkdir -p metadata

$ echo "SELECT href
        FROM   (
            SELECT UNNEST(links, recursive := true)
            FROM   READ_PARQUET('items.parquet')
        )
        WHERE  href LIKE 'https%'" \
    | ~/duckdb -csv -noheader \
    | xargs \
        -I% \
        -P8 \
        wget -P metadata/ -c "%"

$ touch footprints.json; rm footprints.json

$ for FILENAME in metadata/*.json; do
    jq -c . $FILENAME >> footprints.json
  done

$ ~/duckdb

COPY(
    SELECT * EXCLUDE(geometry),
           geometry: ST_GEOMFROMGEOJSON(geometry)
    FROM   'footprints.json'
) TO 'footprints.parquet' (
      FORMAT 'PARQUET',
      CODEC  'ZSTD',
      COMPRESSION_LEVEL 22,
      ROW_GROUP_SIZE 15000);

These are the numbers of unique values for each field in the properties structure. I've excluded any nested structures and lists.

SELECT   column_name,
         column_type,
         approx_unique,
         min,
         max
FROM     (SUMMARIZE
          SELECT properties.*
          FROM   'footprints.parquet')
WHERE    column_type::TEXT NOT LIKE 'STRUCT%'
AND      column_type::TEXT NOT LIKE '%[]%'
ORDER BY LOWER(column_name);

The datetime field appears to be the timestamp of the oldest image in the mosaic and the end_datetime appears to be when the mosaic was either published or when the newest image within the mosaic was captured.

┌────────────────────┬─────────────┬───────────────┬──────────────────────────────────────┬──────────────────────────────────────┐
│    column_name     │ column_type │ approx_unique │                 min                  │                  max                 │
│      varchar       │   varchar   │     int64     │               varchar                │               varchar                │
├────────────────────┼─────────────┼───────────────┼──────────────────────────────────────┼──────────────────────────────────────┤
│ constellation      │ VARCHAR     │             1 │ planetscope                          │ planetscope                          │
│ datetime           │ TIMESTAMP   │             1 │ 2026-05-16 12:00:00                  │ 2026-05-16 12:00:00                  │
│ end_datetime       │ TIMESTAMP   │             1 │ 2026-07-01 00:00:00                  │ 2026-07-01 00:00:00                  │
│ gsd                │ DOUBLE      │             1 │ 4.77731426716                        │ 4.77731426716                        │
│ pl:mosaic          │ VARCHAR     │             1 │ global_quarterly_2026q2_mosaic       │ global_quarterly_2026q2_mosaic       │
│ pl:mosaic_id       │ UUID        │             1 │ 649cf917-0aab-40eb-94df-c222535f2cf7 │ 649cf917-0aab-40eb-94df-c222535f2cf7 │
│ pl:percent_covered │ BIGINT      │             1 │ 100                                  │ 100                                  │
│ pl:quad_id         │ VARCHAR     │            86 │ 585-1059                             │ 596-1054                             │
│ proj:code          │ VARCHAR     │             1 │ EPSG:3857                            │ EPSG:3857                            │
│ proj:shape         │ VARCHAR     │             1 │ [4096, 4096]                         │ [4096, 4096]                         │
│ start_datetime     │ TIMESTAMP   │             1 │ 2026-04-01 00:00:00                  │ 2026-04-01 00:00:00                  │
└────────────────────┴─────────────┴───────────────┴──────────────────────────────────────┴──────────────────────────────────────┘

These are the basemap tiles' footprints in relation to Colombia and the wider region.

Below, I've zoomed in on Pereira.

Planet Labs Open Satellite Feed

Pre- and Post-Event Imagery

There are two high-level categories of assets in any event: pre- and post-event imagery. I'll use the post-event imagery for Nepal for this walk-through.

In the folder listings for either pre- or post-event imagery, there will be an items.parquet file. Clicking the filename will bring up a webpage describing its contents, but next to the filename is a link that you can access from the command line and/or DuckDB, without Cloudflare blocking your client.

Planet Labs Open Satellite Feed

The following will download the 83 KB Parquet-formatted, post-event metadata file for the flash-flood disaster in Nepal last month.

$ wget -O nepal.post.parquet \
       https://data.source.coop/planet/disasterdata/nepal-flash-flood-2026-08-26/post-event/items.parquet

There are 24 records in this file.

$ ~/duckdb

SELECT COUNT(*)
FROM   'nepal.post.parquet';

24

This is an example record.

$ echo "FROM  'nepal.post.parquet'
        LIMIT 1" \
    | ~/duckdb -json \
    | jq -S .

[
  {
    "assets": {
      "analytic": null,
      "pansharpened": {
        "eo:bands": null,
        "href": "https://data.source.coop/planet/disasterdata/nepal-flash-flood-2026-08-26/post-event/pelican-2026-08-27/items/20260827_060956_98_3009/20260827_060956_98_3009_pansharpened.tif",
        "pl:asset_type": "ortho_pansharpened",
        "pl:bundle_type": "pansharpened_udm2",
        "proj:bbox": [
          335132.5,
          3125818.0,
          346500.5,
          3137230.5
        ],
        "proj:epsg": 32645,
        "proj:shape": [
          22825,
          22736
        ],
        "proj:transform": [
          0.5,
          0.0,
          335132.5,
          0.0,
          -0.5,
          3137230.5,
          0.0,
          0.0,
          1.0
        ],
        "raster:bands": [
          {
            "data_type": "uint16",
            "nodata": 0.0,
            "spatial_resolution": 0.5,
            "statistics": {
              "maximum": 58588.0,
              "minimum": 11968.0
            }
          },
          {
            "data_type": "uint16",
            "nodata": 0.0,
            "spatial_resolution": 0.5,
            "statistics": {
              "maximum": 33961.0,
              "minimum": 10257.0
            }
          },
          {
            "data_type": "uint16",
            "nodata": 0.0,
            "spatial_resolution": 0.5,
            "statistics": {
              "maximum": 34764.0,
              "minimum": 7934.0
            }
          },
          {
            "data_type": "uint16",
            "nodata": 0.0,
            "spatial_resolution": 0.5,
            "statistics": {
              "maximum": 29368.0,
              "minimum": 6403.0
            }
          },
          {
            "data_type": "uint16",
            "nodata": 0.0,
            "spatial_resolution": 0.5,
            "statistics": {
              "maximum": 42197.0,
              "minimum": 7554.0
            }
          },
          {
            "data_type": "uint16",
            "nodata": 0.0,
            "spatial_resolution": 0.5,
            "statistics": {
              "maximum": 39092.0,
              "minimum": 7926.0
            }
          }
        ],
        "roles": [
          "data"
        ],
        "title": "Pansharpened multiband ortho COG",
        "type": "image/tiff; application=geotiff; profile=cloud-optimized"
      },
      "thumbnail": {
        "href": "https://data.source.coop/planet/disasterdata/nepal-flash-flood-2026-08-26/post-event/pelican-2026-08-27/items/20260827_060956_98_3009/20260827_060956_98_3009_thumbnail.png",
        "roles": [
          "thumbnail"
        ],
        "title": "Thumbnail",
        "type": "image/png"
      },
      "udm": null,
      "udm2": {
        "eo:bands": null,
        "href": "https://data.source.coop/planet/disasterdata/nepal-flash-flood-2026-08-26/post-event/pelican-2026-08-27/items/20260827_060956_98_3009/20260827_060956_98_3009_udm2.tif",
        "pl:asset_type": "ortho_pansharpened_udm2",
        "pl:bundle_type": "pansharpened_udm2",
        "proj:bbox": [
          335132.5,
          3125818.0,
          346500.5,
          3137230.5
        ],
        "proj:epsg": 32645,
        "proj:shape": [
          22825,
          22736
        ],
        "proj:transform": [
          0.5,
          0.0,
          335132.5,
          0.0,
          -0.5,
          3137230.5,
          0.0,
          0.0,
          1.0
        ],
        "raster:bands": [
          {
            "data_type": "uint8",
            "spatial_resolution": 0.5,
            "statistics": {
              "maximum": 1.0,
              "minimum": 0.0
            }
          },
          {
            "data_type": "uint8",
            "spatial_resolution": 0.5,
            "statistics": {
              "maximum": 0.0,
              "minimum": 0.0
            }
          },
          {
            "data_type": "uint8",
            "spatial_resolution": 0.5,
            "statistics": {
              "maximum": 1.0,
              "minimum": 0.0
            }
          },
          {
            "data_type": "uint8",
            "spatial_resolution": 0.5,
            "statistics": {
              "maximum": 1.0,
              "minimum": 0.0
            }
          },
          {
            "data_type": "uint8",
            "spatial_resolution": 0.5,
            "statistics": {
              "maximum": 0.0,
              "minimum": 0.0
            }
          },
          {
            "data_type": "uint8",
            "spatial_resolution": 0.5,
            "statistics": {
              "maximum": 1.0,
              "minimum": 0.0
            }
          },
          {
            "data_type": "uint8",
            "spatial_resolution": 0.5,
            "statistics": {
              "maximum": 100.0,
              "minimum": 0.0
            }
          },
          {
            "data_type": "uint8",
            "spatial_resolution": 0.5,
            "statistics": {
              "maximum": 2.0,
              "minimum": 0.0
            }
          }
        ],
        "roles": [
          "data",
          "snow-ice",
          "cloud",
          "cloud-shadow"
        ],
        "title": "Usable Data Mask (UDM2) — per-pixel quality",
        "type": "image/tiff; application=geotiff; profile=cloud-optimized"
      },
      "visual": {
        "eo:bands": null,
        "href": "https://data.source.coop/planet/disasterdata/nepal-flash-flood-2026-08-26/post-event/pelican-2026-08-27/items/20260827_060956_98_3009/20260827_060956_98_3009_visual.tif",
        "pl:asset_type": "ortho_visual",
        "pl:bundle_type": "visual",
        "proj:bbox": [
          335132.5,
          3125818.0,
          346500.5,
          3137230.5
        ],
        "proj:epsg": 32645,
        "proj:shape": [
          22825,
          22736
        ],
        "proj:transform": [
          0.5,
          0.0,
          335132.5,
          0.0,
          -0.5,
          3137230.5,
          0.0,
          0.0,
          1.0
        ],
        "raster:bands": [
          {
            "data_type": "uint8",
            "spatial_resolution": 0.5,
            "statistics": {
              "maximum": 255.0,
              "minimum": 0.0
            }
          },
          {
            "data_type": "uint8",
            "spatial_resolution": 0.5,
            "statistics": {
              "maximum": 255.0,
              "minimum": 0.0
            }
          },
          {
            "data_type": "uint8",
            "spatial_resolution": 0.5,
            "statistics": {
              "maximum": 255.0,
              "minimum": 0.0
            }
          },
          {
            "data_type": "uint8",
            "spatial_resolution": 0.5,
            "statistics": {
              "maximum": 255.0,
              "minimum": 0.0
            }
          }
        ],
        "roles": [
          "data",
          "visual"
        ],
        "title": "Visual (true-colour RGB) ortho COG",
        "type": "image/tiff; application=geotiff; profile=cloud-optimized"
      }
    },
    "bbox": {
      "xmax": 85.43406669026984,
      "xmin": 85.31918550951734,
      "ymax": 28.35121763114424,
      "ymin": 28.24911357422872
    },
    "collection": "post-event-pelican-2026-08-27",
    "constellation": "pelican",
    "created": "2026-08-27 10:33:45+03",
    "datetime": "2026-08-27 09:09:56.99105+03",
    "eo:cloud_cover": 85,
    "geometry": "POLYGON ((85.33889798257499 28.35121763114424, 85.31918550951734 28.2661375282364, 85.41382376956273 28.24911357422872, 85.43406669026984 28.334115698528574, 85.33889798257499 28.35121763114424))",
    "gsd": 0.55,
    "id": "20260827_060956_98_3009",
    "instruments": null,
    "links": [
      {
        "href": "../../../../catalog.json",
        "rel": "root",
        "title": "Planet Crisis Response — Bhote Koshi–Trishuli Outburst Flood, Nepal (2026)",
        "type": "application/json"
      },
      {
        "href": "../../collection.json",
        "rel": "parent",
        "title": null,
        "type": "application/json"
      },
      {
        "href": "../../collection.json",
        "rel": "collection",
        "title": null,
        "type": "application/json"
      },
      {
        "href": "https://data.source.coop/planet/disasterdata/nepal-flash-flood-2026-08-26/post-event/pelican-2026-08-27/items/20260827_060956_98_3009/20260827_060956_98_3009.json",
        "rel": "self",
        "title": null,
        "type": "application/json"
      },
      {
        "href": "./20260827_060956_98_3009_thumbnail.png",
        "rel": "thumbnail",
        "title": null,
        "type": "image/png"
      }
    ],
    "pl:data_version": null,
    "pl:ground_control": true,
    "pl:ground_control_ratio": null,
    "pl:item_type": "PelicanScene",
    "pl:pixel_resolution": 0.5,
    "pl:publishing_stage": "finalized",
    "pl:quality_category": "standard",
    "pl:strip_id": "20260827_060956_98_3009_strip",
    "platform": "3009",
    "published": "2026-08-27 10:33:45+03",
    "stac_extensions": [
      "https://stac-extensions.github.io/eo/v1.0.0/schema.json",
      "https://stac-extensions.github.io/view/v1.0.0/schema.json",
      "https://stac-extensions.github.io/raster/v1.1.0/schema.json",
      "https://stac-extensions.github.io/projection/v1.1.0/schema.json"
    ],
    "stac_version": "1.0.0",
    "type": "Feature",
    "updated": "2026-09-02 08:47:29+03",
    "view:azimuth": 255.9,
    "view:off_nadir": 3.8,
    "view:sun_azimuth": 171.2,
    "view:sun_elevation": 71.6
  }
]

These are the numbers of unique values for each field in the properties structure. I've excluded structures and lists.

$ ~/duckdb --nullvalue ""

SELECT   column_name,
         column_type[:15],
         approx_unique,
         min[:25],
         max[:25]
FROM     (SUMMARIZE
          FROM   'nepal.post.parquet')
WHERE    column_type::TEXT NOT LIKE 'STRUCT%'
AND      column_type::TEXT NOT LIKE '%[]%'
ORDER BY LOWER(column_name);

┌─────────────────────────┬──────────────────┬───────────────┬───────────────────────────┬───────────────────────────┐
│       column_name       │ column_type[:15] │ approx_unique │         min[:25]          │          max[:25]          │
│         varchar         │     varchar      │     int64     │          varchar          │          varchar          │
├─────────────────────────┼──────────────────┼───────────────┼───────────────────────────┼───────────────────────────┤
│ collection              │ VARCHAR          │             6 │ post-event-pelican-2026-0 │ post-event-skysat-2026-08 │
│ constellation           │ VARCHAR          │             3 │ pelican                   │ skysat                    │
│ created                 │ TIMESTAMP WITH   │            18 │ 2026-08-26 09:32:01+03    │ 2026-09-01 09:45:14+03    │
│ datetime                │ TIMESTAMP WITH   │            26 │ 2026-08-26 08:01:25.99322 │ 2026-09-01 08:06:35.68372 │
│ eo:cloud_cover          │ BIGINT           │            15 │ 47                        │ 93                        │
│ geometry                │ GEOMETRY('EPSG:  │            25 │ POLYGON ((85.339966733816 │ POLYGON ((85.321140858017 │
│ gsd                     │ DOUBLE           │            11 │ 0.55                      │ 3.9                       │
│ id                      │ VARCHAR          │            24 │ 20260826_050125_99_255f   │ 20260901_050635_67_300b   │
│ pl:data_version         │ VARCHAR          │             7 │ 20260826T100156Z          │ 20260828T092936Z          │
│ pl:ground_control       │ BOOLEAN          │             1 │ true                      │ true                      │
│ pl:ground_control_ratio │ BIGINT           │             1 │ 1                         │ 1                         │
│ pl:item_type             │ VARCHAR          │             2 │ PSScene                   │ SkySatCollect             │
│ pl:pixel_resolution     │ DOUBLE           │             2 │ 0.5                       │ 3.0                       │
│ pl:publishing_stage      │ VARCHAR          │             1 │ finalized                 │ finalized                 │
│ pl:quality_category     │ VARCHAR          │             2 │ standard                  │ test                      │
│ pl:strip_id              │ VARCHAR          │             7 │ 20260827_060956_98_3009_s │ s3_20260827T020055Z       │
│ platform                │ VARCHAR          │             7 │ 251f                      │ SSC9                      │
│ published               │ TIMESTAMP WITH   │            18 │ 2026-08-26 09:32:01+03    │ 2026-09-01 09:45:14+03    │
│ stac_version             │ VARCHAR          │             1 │ 1.0.0                     │ 1.0.0                     │
│ type                    │ VARCHAR          │             1 │ Feature                   │ Feature                   │
│ updated                 │ TIMESTAMP WITH   │            12 │ 2026-08-26 17:32:46+03    │ 2026-09-02 08:47:29+03    │
│ view:azimuth             │ DOUBLE           │            14 │ 3.7                       │ 276.1                     │
│ view:off_nadir           │ DOUBLE           │            12 │ 2.1                       │ 28.9                      │
│ view:sun_azimuth         │ DOUBLE           │            24 │ 92.7                      │ 255.4                     │
│ view:sun_elevation       │ DOUBLE           │            21 │ 26.9                      │ 71.8                      │
└─────────────────────────┴──────────────────┴───────────────┴───────────────────────────┴───────────────────────────┘

These are the numbers of images captured by each constellation on any given day.

PIVOT (
    SELECT   constellation,
             taken_on: datetime::DATE,
             cnt:      COUNT(*)
    FROM     'nepal.post.parquet'
    GROUP BY 1, 2
)
ON       constellation
USING    SUM(cnt)
GROUP BY taken_on
ORDER BY taken_on;

┌────────────┬─────────┬─────────────┬────────┐
│  taken_on  │ pelican │ planetscope │ skysat │
│    date    │ int128  │    int128    │ int128 │
├────────────┼─────────┼─────────────┼────────┤
│ 2026-08-26 │         │           9 │        │
│ 2026-08-27 │       3 │             │      2 │
│ 2026-08-28 │         │           5 │        │
│ 2026-08-31 │         │             │      2 │
│ 2026-09-01 │       3 │             │        │
└────────────┴─────────┴─────────────┴────────┘

These are the highest and lowest resolutions of the imagery captured, broken down by constellation.

SELECT   constellation,
         min_: MIN("pl:pixel_resolution"),
         max_: MAX("pl:pixel_resolution")
FROM     'nepal.post.parquet'
GROUP BY 1
ORDER BY 2;

┌───────────────┬────────┬────────┐
│ constellation │  min_  │  max_  │
│    varchar    │ double │ double │
├───────────────┼────────┼────────┤
│ pelican       │    0.5 │    0.5 │
│ skysat        │    0.5 │    0.5 │
│ planetscope   │    3.0 │    3.0 │
└───────────────┴────────┴────────┘

The images have been published as quickly as 1.4 hours after being captured.

SELECT   hours_:     ROUND(EPOCH(created - datetime) / 3600, 1),
         num_images: COUNT(*)
FROM     'nepal.post.parquet'
GROUP BY 1
ORDER BY 1;

┌────────┬────────────┐
│ hours_ │ num_images │
│ double │   int64    │
├────────┼────────────┤
│    1.4 │          3 │
│    1.5 │          4 │
│    1.6 │          4 │
│    1.7 │          1 │
│    2.2 │          4 │
│    3.1 │          4 │
│    4.0 │          2 │
│    4.1 │          2 │
└────────┴────────────┘

These are the locations of the footprints in relation to the whole of Nepal and southern China.

Planet Labs Open Satellite Feed

The imagery has been captured at different times and, in some cases, overlaps with other imagery that has already been collected.

Planet Labs Open Satellite Feed

Downloading Imagery

I'll build a list of the SkySat GeoTIFFs and download them with four parallel threads.

$ mkdir -p nepal_skysat
$ cd nepal_skysat

$ echo "SELECT href
        FROM (
            SELECT UNNEST(assets.visual, recursive := true)
            FROM   '../nepal.post.parquet'
            WHERE  constellation = 'skysat'
        )" \
    | ~/duckdb -csv -noheader \
    | xargs \
        -I% \
        -P4 \
        wget -c "%"

The resulting images are 1.9 GB in total.

$ du -hsc *_visual.tif

749M    20260827_020055_ssc1_u0001_visual.tif
416M    20260827_020055_ssc1_u0002_visual.tif
359M    20260831_092523_ssc9_u0002_visual.tif
342M    20260831_092523_ssc9_u0003_visual.tif
1.9G    total

These are the satellite images.

Planet Labs Open Satellite Feed

There was a lot of cloud cover when these images were captured, but some of the settlements affected by the flash flood can be seen.

Planet Labs Open Satellite Feed

Usable Data Masks

The above images have usable data mask (UDM) counterparts which pinpoint areas where the imagery is covered by clouds.

$ echo "SELECT href
        FROM (
            SELECT UNNEST(assets.udm, recursive := true)
            FROM   '../nepal.post.parquet'
            WHERE  constellation = 'skysat'
        )" \
    | ~/duckdb -csv -noheader \
    | xargs \
        -I% \
        -P4 \
        wget -c "%"

The UDMs for the above images are 49 MB in total.

$ du -hsc *_udm.tif

22M     20260827_020055_ssc1_u0001_udm.tif
8.8M    20260827_020055_ssc1_u0002_udm.tif
12M     20260831_092523_ssc9_u0002_udm.tif
7.5M    20260831_092523_ssc9_u0003_udm.tif
49M     total

The areas coloured in black will have a clear view of the Earth's surface, white will be obstructed by clouds, and the grey areas don't contain any imagery.

Planet Labs Open Satellite Feed

Imagery from the PlanetScope and Tanager constellations will be paired with UDM2s, which, in addition to clouds, can identify shadows, haze and snow.

Thank you for taking the time to read this post. I offer both consulting and hands-on development services to clients in North America and Europe. If you'd like to discuss how my offerings can help your business please contact me via LinkedIn.

IN THE UNITED STATES DISTRICT COURT, SOUTHWESTERN DISTRICT, TEMPE, ARIZONA

CASE NO. B19294, JUDGE JOAN KUJAVA, PRESIDING

WILE E. COYOTE, Plaintiff

-v.-

ACME COMPANY, Defendant

Opening Statement of Mr. Harold Schoff, attorney for Mr. Coyote: My client, Mr. Wile E. Coyote, a resident of Arizona and contiguous states, does hereby bring suit for damages against the Acme Company, manufacturer and retail distributor of assorted merchandise, incorporated in Delaware and doing business in every state, district, and territory. Mr. Coyote seeks compensation for personal injuries, loss of business income, and mental suffering caused as a direct result of the actions and/or gross negligence of said company, under Title 15 of the United States Code, Chapter 47, section 2072, subsection (a), relating to product liability.

Mr. Coyote states that on eighty-five separate occasions he has purchased of the Acme Company (hereinafter, “Defendant”), through that company’s mail-order department, certain products which did cause him bodily injury due to defects in manufacture or improper cautionary labelling. Sales slips made out to Mr. Coyote as proof of purchase are at present in the possession of the Court, marked Exhibit A. Such injuries sustained by Mr. Coyote have temporarily restricted his ability to make a living in his profession of predator. Mr. Coyote is self-employed and thus not eligible for Workmen’s Compensation.

Mr. Coyote states that on December 13th he received of Defendant via parcel post one Acme Rocket Sled. The intention of Mr. Coyote was to use the Rocket Sled to aid him in pursuit of his prey. Upon receipt of the Rocket Sled Mr. Coyote removed it from its wooden shipping crate and, sighting his prey in the distance, activated the ignition. As Mr. Coyote gripped the handlebars, the Rocket Sled accelerated with such sudden and precipitate force as to stretch Mr. Coyote’s forelimbs to a length of fifty feet. Subsequently, the rest of Mr. Coyote’s body shot forward with a violent jolt, causing severe strain to his back and neck and placing him unexpectedly astride the Rocket Sled. Disappearing over the horizon at such speed as to leave a diminishing jet trail along its path, the Rocket Sled soon brought Mr. Coyote abreast of his prey. At that moment the animal he was pursuing veered sharply to the right. Mr. Coyote vigorously attempted to follow this maneuver but was unable to, due to poorly designed steering on the Rocket Sled and a faulty or nonexistent braking system. Shortly thereafter, the unchecked progress of the Rocket Sled brought it and Mr. Coyote into collision with the side of a mesa.

Paragraph One of the Report of Attending Physician (Exhibit B), prepared by Dr. Ernest Grosscup, M.D., D.O., details the multiple fractures, contusions, and tissue damage suffered by Mr. Coyote as a result of this collision. Repair of the injuries required a full bandage around the head (excluding the ears), a neck brace, and full or partial casts on all four legs.

Hampered by these injuries, Mr. Coyote was nevertheless obliged to support himself. With this in mind, he purchased of Defendant as an aid to mobility one pair of Acme Rocket Skates. When he attempted to use this product, however, he became involved in an accident remarkably similar to that which occurred with the Rocket Sled. Again, Defendant sold over the counter, without caveat, a product which attached powerful jet engines (in this case, two) to inadequate vehicles, with little or no provision for passenger safety. Encumbered by his heavy casts, Mr. Coyote lost control of the Rocket Skates soon after strapping them on, and collided with a roadside billboard so violently as to leave a hole in the shape of his full silhouette.

Mr. Coyote states that on occasions too numerous to list in this document he has suffered mishaps with explosives purchased of Defendant: the Acme “Little Giant” Firecracker, the Acme Self-Guided Aerial Bomb, etc. (For a full listing, see the Acme Mail Order Explosives Catalogue and attached deposition, entered in evidence as Exhibit C.) Indeed, it is safe to say that not once has an explosive purchased of Defendant by Mr. Coyote performed in an expected manner. To cite just one example: At the expense of much time and personal effort, Mr. Coyote constructed around the outer rim of a butte a wooden trough beginning at the top of the butte and spiralling downward around it to some few feet above a black X painted on the desert floor. The trough was designed in such a way that a spherical explosive of the type sold by Defendant would roll easily and swiftly down to the point of detonation indicated by the X. Mr. Coyote placed a generous pile of birdseed directly on the X, and then, carrying the spherical Acme Bomb (Catalogue # 78-832), climbed to the top of the butte. Mr. Coyote’s prey, seeing the birdseed, approached, and Mr. Coyote proceeded to light the fuse. In an instant, the fuse burned down to the stem, causing the bomb to detonate.

In addition to reducing all Mr. Coyote’s careful preparations to naught, the premature detonation of Defendant’s product resulted in the following disfigurements to Mr. Coyote:

1. Severe singeing of the hair on the head, neck, and muzzle.

2. Sooty discoloration.

3. Fracture of the left ear at the stem, causing the ear to dangle in the aftershock with a creaking noise.

4. Full or partial combustion of whiskers, producing kinking, frazzling, and ashy disintegration.

5. Radical widening of the eyes, due to brow and lid charring.

We come now to the Acme Spring-Powered Shoes. The remains of a pair of these purchased by Mr. Coyote on June 23rd are Plaintiff’s Exhibit D. Selected fragments have been shipped to the metallurgical laboratories of the University of California at Santa Barbara for analysis, but to date no explanation has been found for this product’s sudden and extreme malfunction. As advertised by Defendant, this product is simplicity itself: two wood-and-metal sandals, each attached to milled-steel springs of high tensile strength and compressed in a tightly coiled position by a cocking device with a lanyard release. Mr. Coyote believed that this product would enable him to pounce upon his prey in the initial moments of the chase, when swift reflexes are at a premium.

To increase the shoes’ thrusting power still further, Mr. Coyote affixed them by their bottoms to the side of a large boulder. Adjacent to the boulder was a path which Mr. Coyote’s prey was known to frequent. Mr. Coyote put his hind feet in the wood-and-metal sandals and crouched in readiness, his right forepaw holding firmly to the lanyard release. Within a short time Mr. Coyote’s prey did indeed appear on the path coming toward him. Unsuspecting, the prey stopped near Mr. Coyote, well within range of the springs at full extension. Mr. Coyote gauged the distance with care and proceeded to pull the lanyard release.

At this point, Defendant’s product should have thrust Mr. Coyote forward and away from the boulder. Instead, for reasons yet unknown, the Acme Spring-Powered Shoes thrust the boulder away from Mr. Coyote. As the intended prey looked on unharmed, Mr. Coyote hung suspended in air. Then the twin springs recoiled, bringing Mr. Coyote to a violent feet-first collision with the boulder, the full weight of his head and forequarters falling upon his lower extremities.

The force of this impact then caused the springs to rebound, whereupon Mr. Coyote was thrust skyward. A second recoil and collision followed. The boulder, meanwhile, which was roughly ovoid in shape, had begun to bounce down a hillside, the coiling and recoiling of the springs adding to its velocity. At each bounce, Mr. Coyote came into contact with the boulder, or the boulder came into contact with Mr. Coyote, or both came into contact with the ground. As the grade was a long one, this process continued for some time.

The sequence of collisions resulted in systemic physical damage to Mr. Coyote, viz., flattening of the cranium, sideways displacement of the tongue, reduction of length of legs and upper body, and compression of vertebrae from base of tail to head. Repetition of blows along a vertical axis produced a series of regular horizontal folds in Mr. Coyote’s body tissues—a rare and painful condition which caused Mr. Coyote to expand upward and contract downward alternately as he walked, and to emit an off-key, accordionlike wheezing with every step. The distracting and embarrassing nature of this symptom has been a major impediment to Mr. Coyote’s pursuit of a normal social life.

As the Court is no doubt aware, Defendant has a virtual monopoly of manufacture and sale of goods required by Mr. Coyote’s work. It is our contention that Defendant has used its market advantage to the detriment of the consumer of such specialized products as itching powder, giant kites, Burmese tiger traps, anvils, and two-hundred-foot-long rubber bands. Much as he has come to mistrust Defendant’s products, Mr. Coyote has no other domestic source of supply to which to turn. One can only wonder what our trading partners in Western Europe and Japan would make of such a situation, where a giant company is allowed to victimize the consumer in the most reckless and wrongful manner over and over again.

Mr. Coyote respectfully requests that the Court regard these larger economic implications and assess punitive damages in the amount of seventeen million dollars. In addition, Mr. Coyote seeks actual damages (missed meals, medical expenses, days lost from professional occupation) of one million dollars; general damages (mental suffering, injury to reputation) of twenty million dollars; and attorney’s fees of seven hundred and fifty thousand dollars. Total damages: thirty-eight million seven hundred and fifty thousand dollars. By awarding Mr. Coyote the full amount, this Court will censure Defendant, its directors, officers, shareholders, successors, and assigns, in the only language they understand, and reaffirm the right of the individual predator to equal protection under the law. ♦

Forest in Alps. Credit: INRAE—Hervé Cochard

A research team from INRAE and the University Clermont Auvergne has shown that trees are capable of correcting a curvature they detect in their stems through a specific biological process. In the study, young trees with bent stems were placed in an experimental set-up that prevented them from sensing their orientation relative to light and gravity. The only sense remaining to the trees was the perception of their own curvature. Under these conditions, the scientists observed the formation of a particular type of wood, known as tension wood, which acts like a muscle to correct the curvature of the stem, allowing it to realign within a few weeks.

Published in New Phytologist, the findings show how plants—under natural conditions—finely perceive their own shape and combine this information with signals relating to their orientation to adjust their posture. This ability plays an important role in their resilience when faced with extreme events such as storms or landslides.

Proprioception is the sense through which living organisms perceive the position of their body parts, and it was long believed to be specific to animals. In 2012, a research team involving INRAE demonstrated that plants also possess this ability. This enables them to control their posture and remain as straight as possible. However, the biological mechanism governed by this proprioception remained unknown.

An experimental setup to deprive plants of all senses except proprioception

Plants orient their growth in response to factors they perceive, including gravity, the direction of incoming light and their own shape, that is, proprioception. To investigate proprioception on its own, all other sensory inputs must be eliminated. To remove the sense of gravity and the influence of light direction in trees, the scientists placed the specimens under study in a specially designed experimental setup: a horizontal platform rotating around its own axis inside a sphere flooded with light coming simultaneously from all directions.

Clinostat : specially designed experimental set-up to study trees' proprioception. Credit: INRAE—Bruno Moulia

Tension wood: A 'muscle' that can both bend and straighten plants

The researchers first placed young poplar trees in a horizontal position. Over time, the trees curved upward, bringing their tops back to a vertical orientation. This movement is driven by a particular type of wood: tension wood. It acts like a muscle that, by contracting, exerts a pulling force on the upper side of the stem, causing it to bend upward. After around 10 days, once the trees had reached a sufficient degree of curvature, they were transferred to the experimental device.

Over the following weeks, the stems gradually straightened and returned to a rectilinear shape. The scientists examined the anatomy of the wood formed during this straightening movement. The formation of tension wood on the upper side, which had caused the stem to bend upward, ceased when the device was activated, while wood identical in every respect formed on the opposite side.

The latter appears to function as an antagonistic muscle, generating a pulling force in the opposite direction and progressively restoring the stem to a straight form. Tension wood formation is a complex biological process that is regulated at the cellular level and unfolds through several successive stages. The process is also governed by the plant's proprioception.

Until now, tension wood was thought to form only on the upper side of a stem, causing it to bend upward. It can be observed, for example, at the base of trees growing on mountain slopes. This study shows that tension wood can play antagonistic roles, much like the muscles in animals that maintain their posture and enable their movements.

These results show that plants combine fine perceptions of their environment (such as light and gravity) with an awareness of their own shape. They integrate and process this information to activate tension wood in different directions, thereby achieving or maintaining the most appropriate posture. These key abilities contribute to the resilience of trees when faced with extreme events that can alter their position, such as storms or landslides, a trait of particular importance in the context of climate change.

The findings also open new avenues for the selection of cultivated plants based on their proprioception by promoting plants that remain upright and, for example, helping to combat lodging in cereal crops.

"What we have uncovered is a genuine sensorimotor loop operating in the woody parts of trees! Poor coordination in the successive activation of tension wood results in excessive internal tension, which can affect wood quality. These findings therefore reshape applied research aimed at improving wood quality ... and at obtaining trees that are as straight and as relaxed as possible, whatever life throws at them," said Bruno Moulia, INRAE research director.

"Revealing the remarkable capabilities of trees requires a great deal of ingenuity. In this project, we achieved it by bringing together researchers from different disciplines, with complementary skills and perspectives. This requires time and perseverance, but these interdisciplinary discoveries show that the effort is worthwhile," said Félix Hartman, INRAE research engineer.

Publication details

Alexandre Caulus et al, Proprioception drives tension wood formation for autotropic straightening and postural control in trees, New Phytologist (2026). DOI: 10.1111/nph.71238

Journal information: New Phytologist

Cloudflare's rapid growth as a cloud service is due in part to its many unique technologies. Technologies like Anycast and same-IP access are just a few examples, and it was CTO and co-founder Lee Holloway who created these mechanisms. However, he was forced to step down due to a serious brain disease (FTD). This article explores the technological achievements he left behind and the tragedy of the illness that came to light. Text by Masaichi Urabe

Born into an Apple Family

Lee Holloway was born in California, USA, in 1981 (estimated) and grew up in Cupertino, the heart of Silicon Valley. Thanks to his father, who worked at Apple, he was blessed with opportunities to interact with computers from a young age and naturally became familiar with technology.

He displayed exceptional strategic thinking in the video games he played with his younger brother, and he was known for his legendary skill among his friends. There is even a story about him entering a middle school chess club tournament on a whim and winning it. It is clear that he possessed an outstanding intellect and a natural instinct for competition from an early age.

Studying Computer Science

Lee graduated from Monte Vista High School, a prestigious public school in his hometown of Cupertino. He demonstrated his programming talent while still in high school, and after graduation, he worked as an engineer at HomeWarehouse.com, a startup during the early days of the internet.

The company was acquired by retail giant Walmart in 2000. For the young engineer Lee, the experience here was a valuable asset. He later went on to the University of California, Santa Cruz (UC Santa Cruz), where he majored in computer science. While studying computers in earnest at university, he cultivated the spatial awareness and logical thinking skills to visualize large-scale system structures in his mind.

Through Professor Arthur Keller at the university, Lee met a young entrepreneur named Matthew Prince. Matthew was pursuing an idea for anti-spam software, and Lee's lab was working on a project with a similar concept. Matthew took notice of Lee's work, formed a collaborative relationship involving joint patent ownership, and hired Lee on the spot.

Living in Matthew's Basement

Unspam Technologies, the anti-spam company Matthew founded in 2004, was based in Park City, Utah, and Lee became a key member. He lived in the basement of Matthew's home, receiving food and housing in lieu of a salary while working on software development. Lee became the Chief Software Architect, literally leading the design and implementation of the company's software foundation. During his tenure, Lee and his colleagues co-founded Project Honey Pot as a side project. This is an open-source community project that tracks and collects data on spammers (abusers) on the web. Project Honey Pot still exists today as an initiative to track online fraud and abuse, and it is one of the achievements that highlights Lee's creativity and technical prowess.

In 2007, when Prince left Utah to attend Harvard Business School, Lee also returned to California and began living with Alexandra Carey, a friend from his university days.

As Matthew Prince's MBA program neared its end, Lee began receiving job offers from other companies. Hearing this, Matthew made a bold proposal to Lee regarding a new business plan he had been developing with his Harvard classmate Michelle Zatlyn. That idea, which would later develop into Cloudflare, was the concept of "utilizing the spammer information collected by Project Honey Pot to build a service that not only 'monitors' threats but also 'blocks' threats on the internet."

It is said that Matthew spoke passionately about the business concept for an hour at this time. After listening, Lee was silent for a moment before saying, "That's interesting, let's do it!" and immediately decided to join the new venture.

Founding Cloudflare and the Supporting Technical Infrastructure

In the early days of Cloudflare, Lee Holloway was called the "resident genius" and was known for his ability to write code for long hours with exceptional focus.

2011. Lee Holloway flanked by Cloudflare co-founders Michelle Zatlyn (left) and Matthew Prince (right)

The platform he designed continued to support Cloudflare's growth, and his technical vision has been passed down as the foundation of the company's services to this day. Lee Holloway co-founded Cloudflare in 2009 with Matthew and Michelle, building the company's foundation as co-founder and lead engineer. Based on the business plan the two had refined at Harvard, Lee developed Cloudflare's first prototype and built the software infrastructure that would become the core of the service. Examples include the implementation of the Anycast network and the design of the Cell architecture.

In 2010, the three of them took on the emerging business contest TechCrunch Disrupt to showcase Cloudflare to the world. Although Lee did not stand on stage, he worked frantically to fix software bugs until the day of the presentation, leading the demo to success.

After that, Lee was active at the company as Lead Engineer and Architect, and also took on the role of hiring and mentoring the early engineering team. The system built under his command was extremely scalable, serving as the cornerstone that allows Cloudflare to process over 10% of all internet requests and block billions of cyber threats every day.

In 2014, an event occurred that demonstrated Lee's excellence as an engineer. During a project to provide free encrypted communication (SSL/TLS) to all websites, he entered a state of deep focus from the day before the deadline, wearing a hoodie and listening to music with headphones, refusing to speak to anyone, and wrote the code in a single night. Thanks to this, free encrypted communication was provided to all Cloudflare customers as scheduled. It was also reported that the amount of encrypted web traffic on the internet doubled overnight.

Lee's Transformation: The Onset of Frontotemporal Dementia

Lee's career appeared to be sailing smoothly, but starting in the early 2010s, things began to change. Around 2011, when he was around 30 years old, he became extremely prone to fatigue, and days where he would collapse on the floor as soon as he got home from work became more frequent. His once-sociable personality began to fade, and he stopped showing up to invitations from colleagues or events. At home, he began to show signs of apathy, such as no longer interacting with his young son.

Initially, his wife Alexandra (his first wife from his college days) thought they were drifting apart emotionally and suggested counseling, but the situation did not improve.

In 2012, when Alexandra decided to temporarily separate and take their young son with her, Lee asked for divorce papers without hesitation. Their marriage ended this way, and his colleagues assumed that "he had become selfish because he had achieved success and financial freedom." However, this inexplicable behavioral change was actually a symptom of the illness that would later be discovered.

Afterward, Lee became close with Kristin Tarr, who was in charge of communications at Cloudflare, and they proposed to each other during a trip to Rome in 2014 and remarried. Their married life seemed to be going well, but in early 2015, a turning point in the treatment of his underlying condition arrived. Lee had been born with a heart condition called aortic valve regurgitation, and it was suggested that this might be the cause of his headaches. Recommended by a doctor at Stanford University Hospital, he underwent a six-hour heart surgery in January 2015. The surgery was successful and his heart condition improved, but Holloway's mental state changed significantly after this surgery.

Even after returning to work, Lee was listless throughout, and the passion and charisma he once had were gone. His wife, Kristin, felt that he seemed to have lost the rich color of his personality, and he was somewhat absent-minded even at an event in Hawaii just before their wedding. Later, at work, he began to take on an attitude that was like a different person than before, such as starting to play games on his smartphone during meetings, not responding to people, and stubbornly opposing his colleagues' suggestions. Matthew and Michelle held repeated meetings to urge him to improve, but it had no effect.

Matthew recalled feeling a "tremendous anxiety, like losing a best friend" as he watched his co-founder change right before his eyes. Eventually, in 2016, it became difficult to keep Lee at the workplace, and a decision for him to leave the company was made. It is said that while Matthew was giving a tearful farewell speech at an all-hands meeting, Lee stood by with a smirk on his face, holding a beer bottle. People inside and outside the company were shocked by the outcome that he, who had dedicated his life to the company's business, was leaving Cloudflare at the age of 36.

After leaving the company, Lee planned to become a stay-at-home father for the second child he was expecting (his child with Kristin). However, his strange behavior gradually escalated at home as well. While his wife was in labor and suffering from contractions at the hospital, Holloway reportedly slept for a long time next to her, and when he woke up, he argued with the doctor against the epidural procedure.

He showed no interest in the newborn baby, and clearly abnormal behaviors occurred frequently, such as watching the movie "Home Alone" over and over again by himself in the middle of the night, or walking around the house all day with a beanie hat pulled down low over his eyes.
Kristin grew increasingly alarmed by her husband's condition and dragged him to several hospitals. Despite these desperate efforts, Lee himself remained expressionless, only repeating "I'll get better," and their conversations gradually stopped making sense. Those around him initially suspected the aftereffects of heart surgery, burnout, or depression.

Then, in March 2017, the results of an examination by a specialized neurologist showed an unbelievable finding: atrophy of Lee's brain. Holloway's brain, as seen in the MRI images, showed atrophy disproportionate to his age, and the doctor told his family that "the possibility of some kind of neurodegenerative disease is high."
Further detailed examinations and interviews were conducted at the UCSF Memory and Aging Center, where he was referred, and the medical team diagnosed him with "behavioral variant frontotemporal dementia (bvFTD)."
His wife Kristin, his parents, and his younger brother were devastated by the shocking news of a type of early-onset dementia at just 36 years old.

Unlike Alzheimer's disease, frontotemporal dementia is a disease in which changes in personality and behavior appear more prominently than memory impairment, and it is a disease that often develops in people in their 50s and 60s. There is no fundamental cure, and the doctor explained that the symptoms would progress further, and eventually he would be unable to speak, have difficulty walking, lose his swallowing function, and his life would eventually be in danger due to infections or accidents.
The cross-section of Lee's brain projected on the screen showed scattered areas where the gray matter of the frontal lobe, which should have been there, had been replaced by black collapse, and it is said that the family present was speechless.

It is reported that only Holloway himself accepted this diagnosis, which was equivalent to a "death sentence," calmly, and while his family was crying, he acted as if it were someone else's problem, such as by complimenting the doctor's wedding ring. That figure was no longer the person the family knew, but a painful reminder that he was a different person created by the disease.

After the diagnosis, Kristin quit her job to spend as much time as possible with her husband. However, the progression of the disease was relentless, and by the summer of 2017, Holloway's judgment had further declined, and his family could no longer keep an eye on him alone. He could no longer control behaviors such as playing music videos at high volume in the middle of the night or walking around the house all night long.

His spoken language was also gradually lost, and by around 2018, he was in a state of mechanically repeating events from his life, such as "met at Cloudflare," "engaged in Rome," and "held a wedding in Maui," and the words he uttered gradually turned into strings of numbers and letters that made no sense.

By 2019, meaningful conversation had finally become impossible, his emotional expression was lost, and he was unable to even make a single phone call. Searching for the shadow of her son who once lovingly hugged her and said "I love you, Mom," his mother Kathy said she cried, saying, "I will never hear those words from my son again."

Reaction from the US Media

The US media also took a great interest in Lee's dramatic life, his technological achievements, and the tragedy that suddenly struck him. In 2020, a feature article titled "The Devastating Decline of a Brilliant Young Coder" was published in the US edition of Wired magazine, reporting in detail on the process by which Holloway, who built the technical foundation as a "genius" within Cloudflare, gradually began to show listless and unpredictable behavior, and no one knew the reason why.

Wired senior editor Sandra Upson wrote, "Lee's personality had been consistent for decades—until that day," and vividly depicts the personality transformation that occurred to him, the years his family spent trying to find the cause, and how the disease called FTD toyed with the lives of him and those around him.

This feature attracted a great response, and Lee's name became widely known outside the technology industry. Also, through the same article, the awareness of the disease FTD itself increased, and "dementia that can occur in the prime of life" was received with surprise by many readers.

On the other hand, praise and gratitude for the technical contributions Lee made were also expressed from various quarters. Cloudflare went public (IPO) in 2019. In the published IPO prospectus, co-founders Matthew and Michelle stated that "third co-founder Lee Holloway was a genius who designed our platform and led the early technical team" and that "his technical decisions and the engineering team he nurtured are the foundation of our current business." When deciding on the codename for the IPO preparation within the company, they named it "Project Holloway" to pay tribute to his contributions.

Furthermore, the tech news site TechCrunch also introduced him at the time of the IPO with the headline, "Cloudflare has a third co-founder," reporting that "Lee Holloway, who left the company in 2016 due to symptoms of the neurological disorder FTD, was the key figure who built the company into what it is today." In this way, the media widely reported on his achievements and his battle with the disease, and many readers responded with comments such as "I was moved" and "My heart aches for the tragedy of this genius programmer."

Lee and his family have also begun contributing to the community fighting this disease. In 2019, his wife, Kristin, and Lee's parents established the The Holloway Fund for Help and Hope in Honor of Lee Holloway within the FTD support organization AFTD (Association for Frontotemporal Degeneration), deciding to allocate 50% of donations to support programs for patients and their families, and the remaining 50% to FTD research.

Kristin Holloway, who also serves as a board member for the AFTD

Due to the calls made by this fund and Kristin's own energetic activities as a board member of the AFTD, Lee's case is being featured in various media and expert conferences as a symbolic case for deepening understanding of FTD.

Since 2020, an annual conference for FTD researchers called the "Holloway Summit," named after Holloway, has also been held, where discussions toward elucidating the causes and developing treatments are taking place. These social movements ensure that his personal history is not just introduced as a tragedy, but is connected to hope for saving future patients.

To be continued in Part 2

In the second part, we will also explain the following technologies:
The technical achievements Lee Holloway left at Cloudflare

1. Implementation and thorough optimization of the Anycast network
2. Design of the "Cell" architecture
3. Integrated platform for performance and security
4. Code quality and system building capabilities

Sources:
Wired
AFTD
TechCrunch
Longreads