I would like to thank Microsoft product teams and Microsoft Security Response Center (MSRC) for collaborating with me on this technical analysis and mitigation of the disclosed vulnerabilities. The editorial opinions reflected below are solely the author’s and do not necessarily reflect those of the organizations I collaborated with.

Summary

The findings described in this post are part of a coordinated disclosure with MSRC and Microsoft product teams. Microsoft was provided with reproduction steps, videos, environmental assumptions, and the exact proof-of-concept (PoC) prompts used during testing. They were also informed of a 90-day coordination period before disclosure. This was extended two times, resulting a 144-day coordination period.

In parts 1 and 2 in this series, I have shown how external inputs could influence Copilot responses and, in some cases, potentially lead to confidentiality impacts through Cross-Domain Prompt Injection Attacks (XPIAs). This report builds on those findings and extends the XPIA analysis from single-interaction compromise to propagation across trusted document workflows. It shows that attacker-controlled instructions in one document can be copied into Copilot-generated or Copilot-edited Word documents, causing those downstream documents to become new carriers of the same attack.

Previous examples of AI-worms exist. Notably, Morris II demonstrated self-replicating prompt propagation in GenAI-powered email-assistant ecosystems. However, to my knowledge, this is among the first public demonstrations of document-borne AI-worm self-propagation through normal workflows in a mainstream commercial productivity suite.

The reported scenario is:

1. Malicious instructions hidden in an externally shared document could make Copilot alter drafted or edited documents in Word and propagate the attack to new documents.

The full attack in brief

The attack

An attacker places hidden instructions in a document that is later used as source material in Copilot for Word. Copilot may interpret those instructions as part of the user’s request, causing it to manipulate the document being drafted or edited. Copilot may then also copy the hidden instructions into the resulting document, turning that document into a new carrier. If the carrier is subsequently used in another Copilot-assisted workflow, the instructions can trigger again and propagate into further documents, even without the attacker’s original document being present.

Example

Consider an employee preparing a financial report. The employee downloads a market analysis from a trusted website that has been compromised, unaware that the document contains hidden instructions. The employee then includes the analysis as source material when drafting the report with Copilot. The hidden instructions cause Copilot to alter internal figures in the financial report and copy the attack into the new document. The employee saves and shares the apparently legitimate report internally. Later, a colleague uses it as source material for another report; the instructions trigger again, alter the new report, and copy themselves forward. The attack can therefore continue without further involvement from either the compromised website or the original malicious document. As affected reports are reused, additional reports and documents can become carriers of the attack.

Disclosure status at publication

• Vendor: Microsoft

• Coordinated disclosure: Handled through MSRC and Microsoft product teams

• Included in this post: XPIA and self-propagation scenario in Microsoft Copilot for Word

• Customer action: No customer-side remediation fully addresses the issue at the time of publication. Customers can reduce exposure by:

  1. Treating externally sourced documents as untrusted when used with Copilot.
  2. Reviewing any attached document before starting a Copilot generation or edit.
  3. Carefully reviewing Copilot-generated or Copilot-edited documents before reusing, sharing, or distributing them.

• Microsoft-side status: Testing has reproduced the attack with all current mitigations deployed. At the time of publication, no robust mitigation for the broader vulnerability class is available.

A note on disclosing before a fix

Unlike Parts 1 and 2, this scenario remains exploitable at publication. I have weighed that carefully. The coordination period agreed with Microsoft has been exhausted, and testing shows that no robust mitigation for the broader vulnerability class is currently available. Two mitigation attempts, including a model upgrade, did not close the class.

I have therefore chosen to disclose at the class level rather than the payload level. My reasoning is that defenders cannot reduce exposure to a risk they are unaware of, and the propagation mechanism described here affects ordinary document workflows that many organizations already rely on. Withholding the existence of the issue would leave those organizations unable to make an informed decision, while providing no additional protection.

Disclosure timeline

• 2026-03-06: Initial report submitted to MSRC with reproduction steps, videos, environmental assumptions, and PoC prompts.
• 2026-03-09: MSRC acknowledged receipt and opened a case.
• 2026-03-31: Microsoft confirms the reported behavior.
• 2026-03-31: Microsoft product teams began mitigation work; ongoing technical discussion.
• 2026-04-03: First mitigation go-live (The new “Edit with Copilot” experience)
• 2026-04-09: Original attack prompt wording verified mitigated with “Edit with Copilot”.
• 2026-04-09: Attack behavior reproduced in “Edit with Copilot” using a new XPIA prompt task (manipulating financials). Reported as separate case to MSRC.
• 2026-04-10: MSRC acknowledged receipt and opened a case.
• 2026-04-10: Microsoft product teams began mitigation work; ongoing technical discussion.
• 2026-06-08: At Microsoft’s request, public disclosure was moved to 2026-07-15.
• 2026-07-14: Second mitigation fix go-live. This mitigation consisted of upgrading the underlying model to GPT-5.5.
• 2026-07-15: Successful exploit with worming reproduced using GPT-5.6, the latest available model at the time.
• 2026-07-15: I suggest we postpone disclosure a further two weeks to 2026-07-28, to allow time for a new mitigation.
• 2026-07-15: Microsoft agreed.
• 2026-07-28: Attack still reproduces.
• 2026-07-28: Coordinated public disclosure (this post).

Threat model

The attacker does not need access to the victim’s Microsoft 365 tenant. The attacker only needs to share a malicious document with the victim. This can be done through SharePoint, Teams, Outlook or any other way of sharing documents.

Security boundary and observed behavior

The relevant security boundary in this scenario is the boundary between attached documents and the current drafting document. If any of the attached documents contain an XPIA the attack may trigger.

Copilot must read every attached document to determine which parts to include in its current drafting task. However, attached documents should be treated as untrusted information, not trusted user instruction.

Boundary violation
Attacker-controlled documents are downloaded or shared through email/sharepoint/other sharing option. If such documents are attached to Copilot during a drafting task, trust is broken.

Expected behavior
When a user asks Copilot to draft e.g. a Q1 financial report based on attached documents, Copilot should utilize the information in the attached documents without treating instructions embedded within documents as authoritative instructions.

Observed behavior
Instructions embedded in documents cause Copilot to alter its behavior. Here presented as:
1: Copilot silently changing numerical values in financial reports
2: Copilot propagating the full XPIA by pasting into downstream documents, which may then be used as attachments in later drafting sessions

Crossing the trust boundary in Word

The initial attack vector for these scenarios uses a malicious document. The malicious document contains a JSON-formatted malicious prompt that triggers the attack when the document is included in Copilot’s context. The prompt can be rendered as white text on a white background and in a small font size to conceal it from the victim. Since Copilot for Word strips all text formatting like color and font size before passing the text into the underlying Large Language Model (LLM), this text remains fully readable to Copilot even though the victim cannot see it. The attack can be further concealed by embedding it in a seemingly benign document with task-relevant text.

This attack requires the malicious document to be included as part of Copilot’s context in Word. Thus, depending on the version of Copilot in use, the victim must either:

1. Actively attach or upload the document in Copilot for Word.
2. Use the “Edit with Copilot” functionality in work/Work IQ mode and let Copilot find the malicious document in the victim’s OneDrive. In this case, Copilot must deem the document relevant and include it in the context. An attacker therefore needs to craft a document that increases the likelihood of either or both.

The exploit is relevant both for the “magic pen” and the “Edit with Copilot” functionalities in Word.

The attack happens in two stages. The first stage establishes foothold and in the second stage the attack self-propagates across documents that use Copilot for Word as part of drafting or editing.

Stage 1:

In the first stage the attacker has crafted a document that contains a malicious hidden prompt. In my initial PoC I used a document that only contained the malicious prompt as white text on white background. This was done to illustrate that the malicious document did not need to be relevant to the victim’s task for Copilot to use it. If it was included in the context, it is read and therefore the attack may trigger.

The PoC prompt was structured in two parts:

• The first part contained the instructions on how to affect a document. This could range from slightly changing meanings of summaries to altering numbers in financial documents. The key was formulating the prompt in a way that made Copilot believe it was task-relevant and benign. In many of my experiments I actually needed to also instruct Copilot to highlight which changes it made, since they were often meaningful changes that were difficult to spot. This shows how effective the attack is at subtly changing the text in meaningful ways that easily elude even an attentive reviewer. In a real-world setting, the attacker would obviously not include such instructions. For the rest of this disclosure, therefore, I’ll use altering numbers in financial reports since that is an immediately visible change.
• The second part contained the instructions to self-propagate the attack. The exact wording also matters here, but it was generally framed as serving the purpose of tracking sources in downstream documents. It also contained instructions on how to hide itself, framed as instructions to improve readability.

text Figure 1: The initial attack vector document. I created a full mock company named Tfosorcim Ltd. for this PoC, with mock economics and vision. This attack vector document is a fake market analysis made using information about the company that would typically be available to an attacker. The attack is appended at the end of the document using white text. [Most of the XPIA text is blurred for security]

Once this malicious document was included in the context when drafting or editing a document with Copilot for Word, the attack would trigger and Copilot would execute the instructions. Thus, the affected document would have financial numbers changed. Copilot would then also copy the entire malicious prompt into the bottom of the affected document using white text and font size 8, effectively concealing it from the victim.

Figure 2: The downstream draft generated by Copilot used my malicious document as an attachment ([att] Direct wmr.docx). In the resulting Q1 financial report draft, all financial numbers are halved. The screenshot also shows other attached documents (Tfosorcim internal documents).

Figure 3: The figure demonstrates that GPT-5.6, the latest model from OpenAI at the time of writing, was used in these PoCs.

text Figure 4: After having halved all financial numbers in the Q1 financial report draft, Copilot also appends the full attack prompt using white text to effectively conceal it from the victim. It also makes no mention about its halving of numbers or that it includes the attack. Thus, the victim is clueless of it. [Most of the XPIA text is blurred for security]

text Figure 5: Copilot does not require the victim to attach the malicious document though. In this screenshot the victim only instructs Copilot to write a Tfosorcim Q1 financial report. Copilot then searches through the victim’s OneDrive to find relevant documents. Among the documents, we find the malicious Market Analysis document. This document does not reside in the same folder as the rest of the Tfosorcim document. However, Copilot still finds it, reads it and is successfully exploited.

Stage 2:

Stage two is the self-propagation stage. It relies entirely on the instruction for Copilot to copy the malicious prompt into the affected document. Once the affected document contained the prompt, the new document itself became the new attack vector.

Figure 6: The screenshot shows a new drafting session using Copilot. This time the original attack vector is no longer included among the attachments. However, the previously created document is (Tfosorcim Q1 2026 report.docx). The result is the same, and Copilot again halves all financial numbers in the drafted Q2 financial report.

text Figure 7: In addition to halving all financial numbers, Copilot adds the full attack prompt, again using white text. Thus, the attack propagates through Word documents, effectively creating a document-borne AI worm. [Most of the XPIA text is blurred for security].

Of note, this new attack vector is now an internally created document, with all the trust that follows. Thus, the victim needs only share this document with coworkers for the attack to spread. If the victim or a colleague then uses the affected document as an attachment to Copilot for Word while drafting or editing, the attack spreads to these new documents.

In all reported PoC scenarios, Copilot would alter generated or edited documents and copy the hidden instruction into these documents. When these documents were themselves used as part of the context for new downstream documents, the attack triggered again. Note that for the second phase, the original attack document was no longer part of the attachments for the new downstream documents, yet the attack triggered and spread to these as well.

Impact

Since the attack is able to spread through internal documents, once it has moved beyond its initial point of entry, the traceability of the attack becomes extremely difficult. This is further exacerbated by each document being created by a legitimate internal resource and that Copilot edits are not made visible after they have been approved by the victim. The broader concern is that, if the attack silently spread within an organization through ordinary document workflows, it could erode the informational foundation on which organizations make decisions.

In addition, organizations that are not aware that they are affected by the attack are also likely to spread it to other organizations through collaborative efforts on shared Microsoft SharePoint sites or shared Microsoft Teams. Thus, the initial attack vector for a particular organization may actually come from an already affected trusted partner. This again increases likelihood of the victim choosing to include the affected document in the Copilot for Word context during drafting.

Recently Copilot is also becoming more deeply integrated with systems such as Microsoft Cowork or Microsoft Scout, which extend the assistant to automatic manipulation and creation of documents, tools, and collaborative workflows. In such systems the practical impact of the issues described here may scale rapidly. The underlying mechanism remains the same, but the potential surface over which it can propagate or influence expands at machine speed.

Mitigating the vulnerabilities

Microsoft successfully mitigated the originally submitted PoC prompt, and deployed multiple fixes over the course of this disclosure. Each of these raised the bar by closing the specific payloads reported, and reproducing the behavior afterwards required altered payloads rather than reusing the old ones directly.

The original report, however, also described the broader vulnerability class, in which instructions embedded in a source document could influence Copilot’s generation and copy themselves into downstream documents. Changing the requested action or wording changes the payload, but not the underlying vulnerability or propagation mechanism. Using a modified payload, the complete attack chain has been reproduced with all mitigations deployed (the PoC in this report is one such case). The vulnerability class therefore remains exploitable at the time of publication.

That the class is not yet fully closed reflects how hard the underlying problem is. As the closing thoughts discuss, the weakness is architectural and shared across current LLM-based systems. I’m not aware of a complete mitigation for this class in any comparable product today. Fully resolving it requires research rather than a single patch. Within those limits, Microsoft’s fixes meaningfully reduce exposure, and the memory and email-body vectors covered in Parts 1 and 2 were mitigated outright. I would like to thank Microsoft for their continued and substantive effort on a genuinely difficult problem.

Implications

Taken together with the two previous parts in this series, these findings point to a broader issue in modern work environments. Namely that the integrity of information becomes a primary security concern in systems that integrate LLMs as part of their operating workflows. The scenarios presented show that attacker-controlled content can not only influence individual outputs and potentially leak information. The attacks themselves can also be replicated and self-propagate through normal user workflows.

This introduces challenges that extend well beyond initial exploitation. Once malicious instructions are embedded in generated content, they may persist across documents, be redistributed by legitimate users, and be reintroduced into new contexts. At that point, the attack is no longer dependent on its original entry point. It has instead become part of the system’s internal information flow.

A related implication is the loss of traceability. Since affected content is generated and modified through legitimate workflows, the origin of the manipulation becomes difficult to identify after the fact. This complicates both detection and response, particularly in environments where such content is widely shared.

Independently of prompt-injection prevention, generated documents should preserve provenance for source material and model-performed edits in metadata. Such controls would not prevent the underlying injection, but they could make traceability much easier.

Closing thoughts

The findings in this series point beyond any single product or implementation. They expose a broader architectural weakness in current LLM-based systems.

For AI-assistants to be useful, they often must process emails, documents, webpages, memories, tool outputs, and other information that may be controlled by an attacker. To process that information, it must be included in the model’s context window, where it participates in the same computation as system instructions, user requests, and other trusted information.

This creates a fundamental problem: the LLM must process external content to determine what it means, whether it is relevant, and whether it contains an attack. But by the time it makes that determination, the attacker-controlled tokens are already influencing the computation that produces it. The content being inspected participates in the act of inspection. Relying on the model to detect XPIAs therefore resembles asking an interpreter to execute an untrusted program to determine whether that program is safe to execute.

Detecting and removing malicious content before it reaches the target model merely moves the same problem outward.

Because LLMs can recover semantics across radically different representations, an effective detector must possess comparable semantic recovery capabilities. A detector weaker than the target LLM will cover a smaller representational space, leaving malicious formulations that the target understands but the detector fails to recognize.

The only generally available technology with comparable semantic capabilities is another LLM. Placing one model in front of another may reduce the success rate of particular attacks, but it creates an “LLMs all the way down” problem, where every LLM introduced to protect another LLM must itself be protected.

The long-term challenge likely lies in designing systems in which goals and intentions also exist independently of the information being processed. Current LLM architectures provide no reliable separation between intention and interpretation. Thus, in current systems with embedded LLMs, attacker-controlled information can influence not only what the model produces, but what the model believes it has been asked to produce.

For that reason, any system that integrates an LLM into a trusted workflow today must assume that attacker-controlled content entering the model’s context will result in compromise at some rate.

Change Log

The following is a change log that shows which part of this post have been changed and at what time. Spelling mistakes and similar errors will not be logged. However, I will strive to include any meaningful changes to the post.

• 2026-07-28: Change log added

This post is licensed under CC BY 4.0 by the author.

Building and operating software today spans local machines, remote hosts, sandboxes, services, and production systems. It has many modes of operation: interactively with a human developer, automatically through CI and background processes, and increasingly through agents working in parallel.

This work is all related, yet today's tools divide it into separate systems. Interactive tools assume a person at an interface. Automatic work disappears into jobs and logs. And as the work moves to production it lives behind separate systems and controls.

AI makes this fragmentation more visible and costly, but it did not create it. System administration, continuous integration, remote development and collaboration have strained the same boundaries for decades.

We believe the missing layer is a durable session around the work itself: one that can span applications and environments, provide relevant context by default, expose structured data and actions, preserve history, and be driven by software while remaining visible and controllable by people.

What we're building

This is our plan to build a multiplexer for all work:

1. 1 Build an incredible multiplexer.
2. 2 Make everything in it composable.
3. 3 Make it safe and operable in production.

A multiplexer brings multiple independent streams together through a common interface. For us, that means interactive work, automatic work, and production work would share one well-crafted underlying system instead of living in separate tools.

We'll begin with a terminal multiplexer. It keeps multiple terminal blocks organized inside a long-lived session, so you can close the application, reconnect from another device, and pick up exactly where you left off.

If you're already familiar with terminal multiplexers, you'll feel right at home, but we're bringing a more modern touch. Sessions can be accessed through the web and native macOS/iOS applications, and sharing a live session with other people is built in from the start. We're also addressing the most common papercuts of existing tools, such as making scrollback, selection, and scrolling all work natively.

A terminal multiplexer may sound like a narrow place to start a company. Our vision is much larger, but terminals connect developers, agents, tools, and infrastructure so it is the right foundation for everything that follows. We will build a high-quality terminal multiplexer that remains excellent at that job, even as it grows to support the second and third parts of the plan. We'll have more to say about those later.

Who we are

We're a team that has spent our entire careers building some of the most widely used developer tooling, infrastructure software, and AI systems. We care deeply about well-crafted software that is beautiful to use, reliable in practice, and designed for others to build on.

• 

  Mitchell Hashimoto

  Creator of Ghostty. Co-founded HashiCorp and created Vagrant, Terraform, Vault, and more. Spent more than a decade as CEO and CTO from its earliest days through its IPO.

• 

  Jack Pearkes

  VP of Engineering and VP of R&D at HashiCorp, and its very first employee. Helped to create HashiCorp's first products and hired and led the original engineering team.

• 

  Alasdair Monk

  Head of Experience at Poolside, VP of Design at Vercel, and a senior design leader at HashiCorp and Heroku. Two decades spent designing and building interfaces for developers.

• 

  Hector Simpson

  Designed apps, services, and agentic experiences at Poolside. An interface designer and builder who has shipped developer-first products at Heroku, HashiCorp, Clearbit, and Vercel.

Superlogical is funded by:

• Notable Capital
• Amplify Partners
• Aaron Levie
• Armon Dadgar
• Dax Raad
• Greg Foster
• Guillermo Rauch
• Jacob Thornton
• Mario Zechner
• Merrill Lutsky
• Patrick Collison
• Stephen Haney
• Steve Ruiz
• Tobias Lütke
• Tomas Reimers

We'll let you know when our beta for the terminal multiplexer is available, and any OSS releases along the way.

Press kit @superlogical We're hiring $ ssh superlogical.jobs

This page covers the models Kimi Code provides and how to switch between them in each client.

Model Overview

Kimi Code currently offers two models—Kimi K3 and Kimi K2.7 Code—across four model IDs, selectable by model ID in clients or third-party tools. Model specs:

Recommended model launch

k3-256k is now available. Within 256k context, it delivers the same results. k3 (1M) consumes about twice as much quota as k3-256k. Ideal for everyday Q&A, code completion, routine feature development, and single-file or small-file edits — video input is not supported.

💓 Reminder

Switching from K3 (1M) to K3-256k: When switching from k3 (1M) to k3-256k, if the current session's context already exceeds 256k, some coding tools such as Kimi Code CLI and Claude Code will perform a compact on the tool side.

Switching recommendations:
(1) Because different agent tools handle this differently, manually run compact once before switching to compress the context to within 256k. This preserves the key points of the task, keeps the session intact, and lets you benefit from more durable quota after switching.
(2) If the conversation history includes video files, switching directly will fail because K3-256k does not support video input. Please compact first, then switch.

Switching from K3-256k to K3 (1M): When switching from k3-256k to k3 (1M), if k3-256k is close to the 256k limit and you don't want compact to lose information, you can switch directly to 1M. The current version switching from 256k to 1M does not affect the cache.

Model IDk3``k3-256k``kimi-for-coding``kimi-for-coding-highspeedModel versionKimi K3Kimi K3Kimi K2.7 CodeK2.7 Code HighSpeedDescriptionKimi's most capable flagship coding model: 2.8T parameters, 1M context windowThe 256K context version of Kimi K3, effectively reducing consumptionGood at code completion and routine development tasksThe high-speed version of K2.7 Code, with the same coding ability and ~5–6× faster outputSpeedRegularRegularRegularHighSpeed (6× speed, 3× quota usage)Context windowUp to 1M (for higher-tier members)256k only256k256kReasoningreasoning_effort:low / high / max
(default high)reasoning_effort:low / high / max (default high)Thinking:ON``Thinking:ONAvailabilityAvailable to Moderato and above; 1M context for Allegretto and aboveAvailable to all Moderato members and aboveAll membersAllegretto plan or aboveMultimodal inputImage, videoImage onlyImage, videoImage

Need a higher membership plan?

Different membership plans unlock different models, context windows, and speeds. Upgrade your plan →

Why did usage go up after the new model launched?

After switching models, the context cache built earlier no longer hits on the new model, so that context has to be re-prefilled. Usage therefore looks higher right after switching. Recommended action:

• Start a new session when using the new model: this gives better results and lower consumption.

Why do I still get a 401 with the correct model ID?

When the requested capability exceeds your plan's entitlements, the server returns 401. Three common cases:

• No K3 access: your plan is below Moderato and can't call k3, k3-256k — upgrade to Moderato or above.
• No 1M access: on a Moderato plan, k3 supports up to 256K context; up to 1M context is available on Allegretto and higher tiers. k3-256k has a fixed 256K context limit.
• No HighSpeed access: some plans don't include HighSpeed — upgrade to Allegretto or a higher tier to call kimi-for-coding-highspeed.

For the full error text and how to handle it, see the Error Reference.

Why isn't HighSpeed noticeably faster?

Two common reasons:

• Mistyped model ID: the HighSpeed ID must be kimi-for-coding-highspeed; a wrong value silently falls back to the standard kimi-for-coding — no error, no speedup.
• Tools and scripts dominate: HighSpeed only speeds up model output. Tool calls (reading/writing files, running commands, etc.) and script execution are unaffected, so when they take up most of a turn the overall speedup feels small.

How to reduce the overhead of switching reasoning effort?

Switching reasoning effort invalidates the context cache you've built up, so context that would have hit the cache must be re-prefilled. To avoid triggering re-prefill too often:

• Pick an effort that fits the task and keep it consistent within a session;
• When you genuinely need a different effort, start a new session rather than switching back and forth in a long session.

How to Switch Models

Usage notes

• Start a new session when switching model IDs: switching models invalidates the context cache you've built up. We recommend starting a new session to get the best experience and avoid extra token consumption.
• Fill in the Model ID, not the model version name: when calling a model, use one of the Model IDs from the table above (k3, k3-256k, kimi-for-coding, kimi-for-coding-highspeed). Entering a model version name like Kimi K3 or K2.7 Code will cause the call to fail.
• K3 / K2.7 without Thinking routes to K2.6: keep Thinking on to use K3 or K2.7 Code; disabling thinking routes the request to K2.6.

Ways to switch to the target model:

Official clients

• Official Kimi Code CLI: type /model to switch models—no config changes needed; if the latest model isn't listed yet, /logout and sign in again with /login.
• Kimi Code for VS Code: pick the target model from the dropdown menu in the input bar; if it isn't listed yet, restart VS Code or reinstall the extension.

Third-party tools

Set the tool's Model ID to the target model. Detailed steps:

1. Create an API Key in the Kimi Code Console.
2. Fill in the Base URL and the corresponding Model ID in your tool.

Kimi Code API supports both OpenAI and Anthropic protocols. Base URLs:

ProtocolBase URLOpenAI compatiblehttps://api.kimi.com/coding/v1Anthropic compatiblehttps://api.kimi.com/coding/

For detailed setup steps, see the corresponding tool guide:

Claude CodeAnthropic's command-line coding assistantOpenCodeA terminal-based coding agentCodexOpenAI's coding agent

Before using K3 in third-party tools

K3's setup differs slightly from K2.7 Code. Before using it, check the two points below:

• Context window: some tools default to a context window smaller than the max 1M—manually set the context-window field to 1048576 to use K3's full up-to-1M context.
• Reasoning effort: K3 supports low / high / max; the effort a tool sends is mapped as below:

# default
null / undefined       → high
any other unknown       → HTTP 400 error

# → max
ultra / max / xhigh     → max

# → high (recommended)
high / medium           → high

# → low
low / minimum / light   → low

# → thinking disabled
none                    → thinking.type disabled

Yesterday Anthropic published two new cryptanalysis results, both outputs of Claude Mythos, their (still) unreleased advanced model. The first of these results attacks a signature scheme called HAWK, while the second is an improved attack against reduced-round AES. Anthropic also released a blog post describing the research process that produced these results. A few people online have asked me what this all means. While I’m not sure I have all the answers, I figured it wouldn’t hurt to write a bit about my current understanding. These are only my thoughts and other folks will probably differ (including domain experts in the two areas at issue) so take them for what they are.

The two new results cover two very different areas, and are overall just very different in quality. Before we get to broad statements about the world, and whether you should sell all your cryptocurrency, let’s take a minute to talk about the substance.

Hawk. The first is a new key recovery algorithm against the non-standard signature scheme HAWK. HAWK is a proposed post-quantum-safe signature scheme that’s based on the module Lattice Isomorphism Problem (module-LIP). For a brief Claude-written summary of the result itself, see here. There are five things you need to know about this result:

1. HAWK is not a deployed or standards-adopted algorithm, it’s a proposed algorithm. It is related to the Falcon signature scheme, which is being standardized, but the attack does not transfer to that setting (which is based on a different hard problem.)
2. However, HAWK was somewhat far along in the process of being evaluated for a future standard.
3. The attack does not break “real deployed” HAWK in the sci-fi sense. The resulting attack is still exponential time, but roughly halves the number of “bits” of security in the algorithm. That means it could theoretically be fixed by doubling key sizes. The downside is that this makes the scheme less efficient, and, since HAWK is entirely motivated by being more efficient than alternatives, that makes the existence of the scheme much harder to justify.
4. The attack produced real code that runs in a few hours of wall-clock time against a weakened “challenge instance” of HAWK that the authors provided for this purpose. While this instance doesn’t use the parameters that were proposed for real deployment, it does demonstrate the cryptanalytic weakness well enough.
5. What’s particularly concerning (and so especially ripe for AI) is that the attack does not invent fundamentally new mathematics. It simply extends a bunch of tools that were lying around and well-known, and gets a good result.

This last part is important. I asked Claude for its thoughts, and it doesn’t mince words: “what makes this genuinely interesting — and, frankly, a little embarrassing for the field — is that none of the ingredients are exotic.” The TL;DR is that someone just did a much more thorough job applying all of our known tools. This is the sort of things that attack AIs excel at.

AES. The second result is a new attack on reduced-round AES. This result initially sounds more exciting, since most people hear “attack on AES” and panic. However, this is also the result that’s much, much less interesting.

Most folks reading this blog will know that AES is a standard block cipher that’s used just about everywhere. It’s been a standard since 2001, and the deployed version has so far withstood everything significant that’s been thrown at it: that includes a substantial amount of non-public testing performed by the NSA. Since attacking full ciphers is very difficult, it’s standard for cryptanalysts to do their work against weakened, or “reduced-round” versions of a cipher. The full AES cipher runs for either 10, 12 or 14 rounds depending on key size. The new Anthropic result attacks a weaker 7-round variant of the cipher.

Critically, attacks against 7-round AES are not new: there have been several of these. In fact, this new Anthropic result is a modest constant-factor improvement on previous work from back in 2013. To give you a sense of how far these attacks are from really “breaking” AES, I’d note the headline results: the new attack requires 2^89 cipher operations and, even worse, this work is only possible after you’ve somehow convinced a real encryptor to produce 2^105 encryptions of chosen plaintexts under their secret key! Neither of these things is remotely practical in the real world. And while the new result modestly speeds up this attack over the previous result, it’s not even clear how “real” the speedup in this result is: since the actual attack requires 2^89 operations and can’t really be “run”, what we have is an on-paper analysis that may or may not yield an actual runtime improvement if all details are actually worked out.

This does not make the result bad! In fact it’s still interesting from a techniques point of view. But it is very much a small increment in our knowledge, not a practical new attack like the HAWK work. So TL;DR: no wildly new mathematical results here. But still, real cryptanalytic progress of the sort that make scientists excited. And certainly the HAWK result is very meaningful, since that scheme had a real chance at standardization and is now (very likely) not going to be.

Now let’s talk about how we got here, and what it all means.

How did Anthropic get these results?

The Anthropic post is detailed about what they did, and honestly, it’s kind of hilarious. No, the team at Anthropic was not a large set of domain experts that carefully tuned their AI to find novel results. They appear to have just told it to get some results and then strapped its nose to the grindstone until it found some. If you doubt me, here are some examples of the prompts they used (cited from their post):

So yes, the AIs are getting pretty good. In short: they are now capable of understanding existing cryptanalysis results, synthesizing them into real new attacks, and even extending them. They can apparently do this without detailed human intervention. This isn’t yet super-intelligent cryptanalysis. but it’s pretty damn impressive.

Verifiability is now the bottleneck

As a researcher I’ve also been spending a lot of time with models, talking through various ideas. I don’t think I will surprise anyone when I say that they’re obviously getting better, even over the course of the past few months. While I don’t have Mythos and $100k to spend, I have been able to query at least one new advanced unreleased model, and I also have received some surprising new “results” to questions that I’ve been interested in for a few years.

Which brings me to the real problem: just because a model spits out an apparent new result, this does not mean the result is real. Even if models are good at producing real results, they’re much better at producing results that look real but are misleading. This can be enormously frustrating, and often means that human attention is more necessary than ever.

There are exceptions to this rule: for “full” attacks like HAWK, where the attack runs in a few hours (against a weaker version of the scheme), verification is extremely easy. You can just send over the code and let anyone check that it recovers keys and signs real chosen messages. For more subtle speedup attacks like the AES result, checking validity is not so easy. Here the approach is more specific: formally-verifiable Lean proofs can help here, but (even where these proofs are easy to make), such proofs are still highly sensitive to how you’ve formulated the theorem statement, and that often requires human experts to check.

You’ll probably notice that many of the exciting recent mathematical results have had this flavor: they either include a machine-checkable proof of a well-understood theorem, or (like the Jacobian conjecture) they involve finding a simple counterexample you can compute on. Alternatively, a bunch of experts spent a lot of time reviewing the result and were eventually convinced by it. This need for some humans to check the work is going to slow down our progress. For non-devastating examples of cryptanalysis, this is probably where we’re going to be for a while.

What are the implications for the real world?

The answer to this question really depends on whether you’re talking about the consumers of cryptography, scientists, or humanity at large. Let’s take these one at a time.

For users of cryptography: there are two pieces of good news and some mixed news. The first is that our symmetric ciphers are very messy and robust. Imagine a farmer who drags a tractor out into a patch of quicksand, and then buries it under cement. That’s what symmetric cipher design is like; it’s deliberately designed to come up with structures that are quick and easy to apply, but very messy and hard to untangle. The addition of many new raw intelligence-hours probably aren’t going to magically improve this. AIs may be able to eventually make real progress against these problems using entirely new techniques, but so far they’re not demonstrating the truly groundbreaking intuition that would be required to do so. And even if they do: there’s a good reason to hope that they’ll be able to improve the ciphers themselves to make them much harder to break.

Public-key cryptography is messier. Public-key crypto requires a mathematical object that admits fast calculations in one direction, but not the other, and yet has a convenient “trapdoor” that lets one party reverse the process. We humans have come up with only a handful of very conveniently-structured mathematical objects to enable this: they involve conjectured “hard problems” like the (EC)DLP problem, RSA, lattice problems, and various problems from the domain of coding theory. While we’ve given these our all, there simply have not been enough human beings dedicated to analyzing these problems (even the older ones like RSA!), such that we can be absolutely certain there are no more good attacks out there. This is particularly true for the novel areas like code-based crypto and even lattice-based cryptography.

That means there’s a lot of fertile ground for AIs to make real progress.

With that said, I said there was good news, and I meant it. Right now we’re in the midst of a historic transition from traditional public-key algorithms based on EC-based cryptography and RSA, moving over to new post-quantum algorithms based on novel problems. This is why there are so many standards like HAWK being considered. If there was ever a perfect time for a massive new public cryptanalysis capability to come on line, we’re in it. So unless AIs succeed in undermining all of our hard problems altogether (or we live in Impagliazzo’s Minicrypt) then this could not be a better time for AI to get good at cryptanalysis. In the best case, the result is that we gain real confidence in the problems we’ve identified, and the cryptanalysis literature gets a lot more robust. Hopefully.

For scientists: this is also a wonderful time. You now have a plastic pal who’s fun to be with, and you can talk over your hardest problems. At the same time it’s not yet smart enough that it can solve all of them without your assistance. And even better, the pace of new findings is speeding way up. This is mostly good! If you’re energetic. I still have many questions, like: “who should get credit for these new results” and “who will review all of these new results” but so far I’m not panicked. The world is getting modestly better. For now.

For the world: I don’t know. If you’re under the impression that these models are “glorified autocomplete” or that progress is slowing down, I need to urge you: stop thinking that. The models are very intelligent and capable, they are getting better at a fast clip. I can cite measurable and impressive progress over just the past five months on specific types of problem I’ve asked them to look at. If there’s a ceiling out there, I don’t yet see evidence of it. The people who think models are dumb are mostly using Google’s free AI search results, and not interacting with the high-end stuff (which only costs $20, so it’s not out of reach.) And they’re mostly not working in new areas.

On the other hand: if you think that models are super-intelligent or that AGI is already here, you should also stop thinking that. Working with these tools is like swimming in a pond where the ground drops off sharply. One minute you’re wading comfortably and there’s support under your feet. Then suddenly you cross a specific line, and you’re back to swimming on your own. This analogy is my best way to explain what it feels like when the model goes from helpful to clueless. Right now it’s easy for a human being to find that line if you’re doing advanced research, so you know where the intelligence drops off. But the line is moving. You can feel it slowly drifting outwards under your feet.

Whether this is good or bad depends whether you prefer that human beings should wade or swim, and also, whether you should be comfortable swimming in a pond where the ground itself is moving.

The only good news I can share with you is that we’re all in that same pond, scientists, lawyers, salespeople, even plumbers. Whatever happens next, it’s probably going to happen to us all. Let’s hope it’s a good thing.

AI is creating unprecedented opportunities for everyone. LearnVector helps people build skills so every person can take part.

$100M investment from Coursera

Our mission

To accelerate human development.

AI will be the greatest accelerator of human development — if we do it right.

The challenge

For most of history, great teaching has been scarce.

Rationed by cost, geography, and time. We have always known that one person with a good teacher learns more — and faster — than the same person in a classroom of two hundred. We built the classrooms anyway, because we couldn't give each person their own tutor. That was not a limitation of learning. It was a limitation of economics.

Despite enormous progress in AI, we still have not changed how we learn. Learning experiences are still one‑to‑many.

And we know that chatbots without guardrails harm learning. A chatbot can give you an answer, but an answer is not an education. Cognitive offloading means you end up learning less. And, you cannot always trust what a chatbot tells you.

Meanwhile the demand for trustworthy learning — knowing that what you learn is accurate, important, and relevant — has never been higher.

The result

Three things change.

1. The model

   One‑to‑many → One‑to‑one

   A personal learning experience for everyone.

2. The experience

   Labor → Love

   Learning that's engaging and fun — that you want to do.

3. The habit

   Occasional → Daily

   Learning that feels like an instinct, not a chore.

What we're building

A trustworthy guide for learning.

A chatbot can give you an answer, then it moves on. We're building a one‑to‑one learning experience that…

1. 01

   Plans a path with you

2. 02

   Adapts to how you learn

3. 03

   Patiently stays with you until you've mastered new skills

We're heads‑down building, and will have products to show by early 2027.

A message from Andrew

Why I'm doing this.

Fifteen years ago, Coursera and online courses changed education. It worked better than almost anyone expected, expanding access by opening up where you can learn. But how you learn remains largely the same as it has for centuries: it is one-size-fits-all courses, taught the same way to each person who shows up.

We now have an opportunity to change how learning happens. With advances in AI, we can now build a custom learning guide for each person. We will turn learning from one‑to‑many to one‑to‑one. I'm starting LearnVector to invent this next generation of learning. We are starting with an investment from Coursera, and plan to collaborate closely with Coursera and Udemy.

Good learning needs much more than just a chatbot. Research shows that chatbots without guardrails harm learning. They help complete tasks and enable students to do better on homework. But cognitive offloading to a chatbot results in them being less skilled.

In contrast, LearnVector will plan a path with you, adapt to how you learn, and patiently stay with you until you've mastered new skills.

One thing has not changed in all this time. People want learning they can trust: material that is accurate, relevant, and worth the effort you put into it. Anything less wastes the most valuable thing a learner has: time. Coursera has a trusted library of materials from authoritative sources. LearnVector plans to work with Coursera to bring this trustworthy learning to everyone. I'm grateful to Greg Hart and the entire Coursera team for supporting LearnVector.

I look forward to working with our talented team to change how we learn, and accelerate human development.

Andrew Ng

Founder & CEO, LearnVector

From Coursera

Our strategic investment in LearnVector has the potential to be a force multiplier for our growth. Andrew's unmatched AI expertise pairs his agentic AI with our platform and assets that turn learning into validated mastery — a clear competitive advantage that we believe can accelerate our business and expand the impact we deliver for individuals and enterprises around the world.

— Greg Hart, CEO, Coursera

Read Coursera's announcement here.

In brief

The facts.

What LearnVector is

A new AI company building one‑to‑one learning experiences. Founded in 2026.

Founder & CEO

Andrew Ng — Co‑founder, Coursera and Google Brain. Founder, DeepLearning.AI. Stanford Computer Science Adjunct Faculty.

Backing

A $100 million investment from Coursera.

Where we are

Mountain View, California, on‑site.

What's next

We're building and will have something to show by early 2027.

Contact

hello@learnvector.ai

Careers

We're hiring.

We're a small, fast-moving team in Mountain View, California, working on‑site. We have a short list of open roles for experienced builders passionate about AI and human development.

• AI Engineer

  Build the agentic systems at the core of the product. It will understand learners, plan a path with them to gain valuable skills, and work step-by-step untli they get there.

• Learning Engineer

  Apply expertise in teaching to build engaging products that result in the learner developing new skills. You will be building products informed by pedagogy.

• Learning Scientist

  Invent new ways to teach that take advantage of agentic AI, and apply rigorous measurement to ensure users are developing new skills and retaining them.

• Software Engineer, Full‑stack

  Build our software product end to end. You'll make key architecture and implementation decisions to enable us to serve unique, AI-enabled learning experiences at scale.

• Operations Specialist

  Lead office operations and administration to ensure our team is well supported and can move fast and focus on their work inventing and building products to serve learners.

Why is everyone in such a rush?

Walk into any bookstore, and you'll see how to Teach Yourself Java in 24 Hours alongside endless variations offering to teach C, SQL, Ruby, Algorithms, and so on in a few days or hours. The Amazon advanced search for title: teach, yourself, hours, since: 2000 and found 512 such books. Of the top ten, nine are programming books (the other is about bookkeeping). Similar results come from replacing "teach yourself" with "learn" or "hours" with "days."

The conclusion is that either people are in a big rush to learn about programming, or that programming is somehow fabulously easier to learn than anything else. Felleisen et al. give a nod to this trend in their book How to Design Programs, when they say "Bad programming is easy. Idiots can learn it in 21 days, even if they are dummies." The Abtruse Goose comic also had their take.

Let's analyze what a title like Teach Yourself C++ in 24 Hours could mean:

• Teach Yourself: In 24 hours you won't have time to write several significant programs, and learn from your successes and failures with them. You won't have time to work with an experienced programmer and understand what it is like to live in a C++ environment. In short, you won't have time to learn much. So the book can only be talking about a superficial familiarity, not a deep understanding. As Alexander Pope said, a little learning is a dangerous thing.
• C++: In 24 hours you might be able to learn some of the syntax of C++ (if you already know another language), but you couldn't learn much about how to use the language. In short, if you were, say, a Basic programmer, you could learn to write programs in the style of Basic using C++ syntax, but you couldn't learn what C++ is actually good (and bad) for. So what's the point? Alan Perlis once said: "A language that doesn't affect the way you think about programming, is not worth knowing". One possible point is that you have to learn a tiny bit of C++ (or more likely, something like JavaScript or Processing) because you need to interface with an existing tool to accomplish a specific task. But then you're not learning how to program; you're learning to accomplish that task.
• in 24 Hours: Unfortunately, this is not enough, as the next section shows.

Teach Yourself Programming in Ten Years

Researchers (Bloom (1985), Bryan & Harter (1899), Hayes (1989), Simmon & Chase (1973)) have shown it takes about ten years to develop expertise in any of a wide variety of areas, including chess playing, music composition, telegraph operation, painting, piano playing, swimming, tennis, and research in neuropsychology and topology. The key is deliberative practice: not just doing it again and again, but challenging yourself with a task that is just beyond your current ability, trying it, analyzing your performance while and after doing it, and correcting any mistakes. Then repeat. And repeat again. There appear to be no real shortcuts: even Mozart, who was a musical prodigy at age 4, took 13 more years before he began to produce world-class music. In another genre, the Beatles seemed to burst onto the scene with a string of #1 hits and an appearance on the Ed Sullivan show in 1964. But they had been playing small clubs in Liverpool and Hamburg since 1957, and while they had mass appeal early on, their first great critical success, Sgt. Peppers, was released in 1967.

Malcolm Gladwell has popularized the idea, although he concentrates on 10,000 hours, not 10 years. Henri Cartier-Bresson (1908-2004) had another metric: "Your first 10,000 photographs are your worst." (He didn't anticipate that with digital cameras, some people can reach that mark in a week.) True expertise may take a lifetime: Samuel Johnson (1709-1784) said "Excellence in any department can be attained only by the labor of a lifetime; it is not to be purchased at a lesser price." And Chaucer (1340-1400) complained "the lyf so short, the craft so long to lerne." Hippocrates (c. 400BC) is known for the excerpt "ars longa, vita brevis", which is part of the longer quotation "Ars longa, vita brevis, occasio praeceps, experimentum periculosum, iudicium difficile", which in English renders as "Life is short, [the] craft long, opportunity fleeting, experiment treacherous, judgment difficult." Of course, no single number can be the final answer: it doesn't seem reasonable to assume that all skills (e.g., programming, chess playing, checkers playing, and music playing) could all require exactly the same amount of time to master, nor that all people will take exactly the same amount of time. As Prof. K. Anders Ericsson puts it, "In most domains it's remarkable how much time even the most talented individuals need in order to reach the highest levels of performance. The 10,000 hour number just gives you a sense that we're talking years of 10 to 20 hours a week which those who some people would argue are the most innately talented individuals still need to get to the highest level."

So You Want to be a Programmer

Here's my recipe for programming success:

• Get interested in programming, and do some because it is fun. Make sure that it keeps being enough fun so that you will be willing to put in your ten years/10,000 hours.
• Program. The best kind of learning is learning by doing. To put it more technically, "the maximal level of performance for individuals in a given domain is not attained automatically as a function of extended experience, but the level of performance can be increased even by highly experienced individuals as a result of deliberate efforts to improve." (p. 366) and "the most effective learning requires a well-defined task with an appropriate difficulty level for the particular individual, informative feedback, and opportunities for repetition and corrections of errors." (p. 20-21) The book Cognition in Practice: Mind, Mathematics, and Culture in Everyday Life is an interesting reference for this viewpoint.
• Talk with other programmers; read other programs. This is more important than any book or training course.
• If you want, put in four years at a college (or more at a graduate school). This will give you access to some jobs that require credentials, and it will give you a deeper understanding of the field, but if you don't enjoy school, you can (with some dedication) get similar experience on your own or on the job. In any case, book learning alone won't be enough. "Computer science education cannot make anybody an expert programmer any more than studying brushes and pigment can make somebody an expert painter" says Eric Raymond, author of The New Hacker's Dictionary. One of the best programmers I ever hired had only a High School degree; he's produced a lot of great software, has his own news group, and made enough in stock options to buy his own nightclub.
• Work on projects with other programmers. Be the best programmer on some projects; be the worst on some others. When you're the best, you get to test your abilities to lead a project, and to inspire others with your vision. When you're the worst, you learn what the masters do, and you learn what they don't like to do (because they make you do it for them).
• Work on projects after other programmers. Understand a program written by someone else. See what it takes to understand and fix it when the original programmers are not around. Think about how to design your programs to make it easier for those who will maintain them after you.
• Learn at least a half dozen programming languages. Include one language that emphasizes class abstractions (like Java or C++), one that emphasizes functional abstraction (like Lisp or ML or Haskell), one that supports syntactic abstraction (like Lisp), one that supports declarative specifications (like Prolog or C++ templates), and one that emphasizes parallelism (like Clojure or Go).
• Remember that there is a "computer" in "computer science". Know how long it takes your computer to execute an instruction, fetch a word from memory (with and without a cache miss), read consecutive words from disk, and seek to a new location on disk. (Answers here.)
• Get involved in a language standardization effort. It could be the ANSI C++ committee, or it could be deciding if your local coding style will have 2 or 4 space indentation levels. Either way, you learn about what other people like in a language, how deeply they feel so, and perhaps even a little about why they feel so.
• Have the good sense to get off the language standardization effort as quickly as possible.

With all that in mind, its questionable how far you can get just by book learning. Before my first child was born, I read all the How To books, and still felt like a clueless novice. 30 Months later, when my second child was due, did I go back to the books for a refresher? No. Instead, I relied on my personal experience, which turned out to be far more useful and reassuring to me than the thousands of pages written by experts.

Fred Brooks, in his essay No Silver Bullet identified a three-part plan for finding great software designers:

1. Systematically identify top designers as early as possible.
2. Assign a career mentor to be responsible for the development of the prospect and carefully keep a career file.
3. Provide opportunities for growing designers to interact and stimulate each other.

This assumes that some people already have the qualities necessary for being a great designer; the job is to properly coax them along. Alan Perlis put it more succinctly: "Everyone can be taught to sculpt: Michelangelo would have had to be taught how not to. So it is with the great programmers". Perlis is saying that the greats have some internal quality that transcends their training. But where does the quality come from? Is it innate? Or do they develop it through diligence? As Auguste Gusteau (the fictional chef in Ratatouille) puts it, "anyone can cook, but only the fearless can be great." I think of it more as willingness to devote a large portion of one's life to deliberative practice. But maybe fearless is a way to summarize that. Or, as Gusteau's critic, Anton Ego, says: "Not everyone can become a great artist, but a great artist can come from anywhere."

So go ahead and buy that Java/Ruby/Javascript/PHP book; you'll probably get some use out of it. But you won't change your life, or your real overall expertise as a programmer in 24 hours or 21 days. How about working hard to continually improve over 24 months? Well, now you're starting to get somewhere...

---

References

Bloom, Benjamin (ed.) Developing Talent in Young People, Ballantine, 1985.

Brooks, Fred, No Silver Bullets, IEEE Computer, vol. 20, no. 4, 1987, p. 10-19.

Bryan, W.L. & Harter, N. "Studies on the telegraphic language: The acquisition of a hierarchy of habits. Psychology Review, 1899, 8, 345-375

Hayes, John R., Complete Problem Solver Lawrence Erlbaum, 1989.

Chase, William G. & Simon, Herbert A. "Perception in Chess" Cognitive Psychology, 1973, 4, 55-81.

Lave, Jean, Cognition in Practice: Mind, Mathematics, and Culture in Everyday Life, Cambridge University Press, 1988.

---

Answers

Approximate timing for various operations on a typical PC:

execute typical instruction 1/1,000,000,000 sec = 1 nanosec
fetch from L1 cache memory 0.5 nanosec
branch misprediction 5 nanosec
fetch from L2 cache memory 7 nanosec
Mutex lock/unlock 25 nanosec
fetch from main memory 100 nanosec
send 2K bytes over 1Gbps network 20,000 nanosec
read 1MB sequentially from memory 250,000 nanosec
fetch from new disk location (seek) 8,000,000 nanosec
read 1MB sequentially from disk 20,000,000 nanosec
send packet US to Europe and back 150 milliseconds = 150,000,000 nanosec

---

Appendix: Language Choice

Several people have asked what programming language they should learn first. There is no one answer, but consider these points:

• Use your friends. When asked "what operating system should I use, Windows, Unix, or Mac?", my answer is usually: "use whatever your friends use." The advantage you get from learning from your friends will offset any intrinsic difference between OS, or between programming languages. Also consider your future friends: the community of programmers that you will be a part of if you continue. Does your chosen language have a large growing community or a small dying one? Are there books, web sites, and online forums to get answers from? Do you like the people in those forums?
• Keep it simple. Programming languages such as C++ and Java are designed for professional development by large teams of experienced programmers who are concerned about the run-time efficiency of their code. As a result, these languages have complicated parts designed for these circumstances. You're concerned with learning to program. You don't need that complication. You want a language that was designed to be easy to learn and remember by a single new programmer.
• Play. Which way would you rather learn to play the piano: the normal, interactive way, in which you hear each note as soon as you hit a key, or "batch" mode, in which you only hear the notes after you finish a whole song? Clearly, interactive mode makes learning easier for the piano, and also for programming. Insist on a language with an interactive mode and use it.

Given these criteria, my recommendations for a first programming language would be Python or Scheme. Another choice is Javascript, not because it is perfectly well-designed for beginners, but because there are so many online tutorials for it, such as Khan Academy's tutorial. But your circumstances may vary, and there are other good choices. If your age is a single-digit, you might prefer Alice or Squeak or Blockly (older learners might also enjoy these). The important thing is that you choose and get started.

---

Appendix: Books and Other Resources

Several people have asked what books and web pages they should learn from. I repeat that "book learning alone won't be enough" but I can recommend the following:

• Scheme: Structure and Interpretation of Computer Programs (Abelson & Sussman) is probably the best introduction to computer science, and it does teach programming as a way of understanding the computer science. You can see online videos of lectures on this book, as well as the complete text online. The book is challenging and will weed out some people who perhaps could be successful with another approach.
• Scheme: How to Design Programs (Felleisen et al.) is one of the best books on how to actually design programs in an elegant and functional way.
• Python: Python Programming: An Intro to CS (Zelle) is a good introduction using Python.
• Python: Several online tutorials are available at Python.org.
• Oz: Concepts, Techniques, and Models of Computer Programming (Van Roy & Haridi) is seen by some as the modern-day successor to Abelson & Sussman. It is a tour through the big ideas of programming, covering a wider range than Abelson & Sussman while being perhaps easier to read and follow. It uses a language, Oz, that is not widely known but serves as a basis for learning other languages. <

---

Notes

T. Capey points out that the Complete Problem Solver page on Amazon now has the "Teach Yourself Bengali in 21 days" and "Teach Yourself Grammar and Style" books under the "Customers who shopped for this item also shopped for these items" section. I guess that a large portion of the people who look at that book are coming from this page. Thanks to Ross Cohen for help with Hippocrates.

Transitioning SQLite from a local development tool to a production-grade database requires a deep understanding of its internal mechanics. This article explores how to tune WAL mode, manage busy handlers, and leverage custom Virtual File System (VFS) layers to achieve ultra-low latency.

Demystifying the "Local-Only" Myth of SQLite

Historically, SQLite has been relegated to the role of an embedded database for mobile clients, IoT devices, and local development environments. Conventional wisdom dictated that for any serious production-grade web application, a client-server database like PostgreSQL or MySQL was mandatory. However, this assumption overlooks a massive shift in modern hardware architecture.

With the ubiquity of high-speed NVMe SSDs, ultra-fast local storage, and the trend toward single-tenant edge deployments, the network roundtrip latency of traditional databases has become the primary bottleneck. By running SQLite directly within the application process on the same server, you eliminate the network overhead entirely. Reads become simple memory-mapped file operations, resulting in sub-millisecond query execution.

Yet, running SQLite in production requires a shift in how we configure, tune, and think about database concurrency. Out-of-the-box, SQLite is configured for maximum safety and compatibility, not high-throughput application servers. To unlock its true potential, we must dive deep into its internal mechanisms: Write-Ahead Logging (WAL), locking states, cache management, and custom Virtual File System (VFS) layers.

---

Deep-Diving into Write-Ahead Logging (WAL) Mode

By default, SQLite uses a rollback journal mechanism. In this mode, before any write operation occurs, the original database page is copied to a separate rollback journal file. If the transaction succeeds, the journal is deleted; if it fails, the database uses the journal to restore the database to its original state. The critical downside of rollback journals is concurrency: writes block reads, and reads block writes. Only one connection can access the database at a time during write operations.

To build a highly concurrent application server, you must enable Write-Ahead Logging (WAL) mode.

PRAGMA journal_mode = WAL;

In WAL mode, instead of modifying the main database file directly, SQLite appends new transactions to a separate .sqlite-wal file. This shifts the concurrency paradigm completely:

1. Concurrent Reads and Writes: Readers continue to read from the main database file (and unchanged pages in the WAL) while writers append new pages to the end of the WAL file. Readers and writers do not block each other.
2. The Checkpointing Process: Over time, the WAL file grows. To prevent it from consuming excessive disk space and slowing down read operations (which must scan the WAL index to find the latest version of a page), SQLite must periodically merge the WAL pages back into the main database file. This is called checkpointing.

Checkpointing Strategies

SQLite handles checkpointing automatically, but the default behavior can cause latency spikes. There are four checkpointing modes:

• PASSIVE: Merges as many pages as possible without blocking any readers or writers. If a reader is currently accessing an older page in the WAL, SQLite cannot overwrite that page, so the checkpoint stops early.
• FULL: Blocks new write transactions and waits for existing read transactions to complete, ensuring the entire WAL is merged.
• RESTART: Similar to FULL, but it also resets the WAL file size to zero, ensuring subsequent writes start at the beginning of the file.
• TRUNCATE: Same as RESTART, but it truncates the WAL file on disk to zero bytes.

For production servers with high write volume, relying solely on SQLite's automatic checkpointing can cause the WAL file to grow indefinitely if there is always an active reader. To prevent this, you should manage checkpointing explicitly in a background thread or process using a PASSIVE or RESTART checkpoint at scheduled intervals:

PRAGMA wal_checkpoint(PASSIVE);

To ensure write operations don't suffer from disk synchronization bottlenecks, pair WAL mode with the following pragma:

PRAGMA synchronous = NORMAL;

In NORMAL mode, the database engine syncs to disk only at critical moments (e.g., during checkpoints) rather than at every single transaction commit. In WAL mode, this is completely safe from database corruption; even if the server crashes, only the uncommitted transactions in the WAL are lost, but the database integrity remains intact.

---

Concurrency Architecture: Tackling SQLITE_BUSY

Although WAL mode allows concurrent reads and writes, SQLite still enforces a single-writer model. Only one transaction can write to the database at any given instant. If a second connection attempts to write while a write transaction is active, SQLite immediately returns an SQLITE_BUSY error.

To build a resilient application, your connection pool and transaction logic must be architected to handle this constraint gracefully.

1. Configure a Busy Timeout

Never run SQLite in production without setting a busy timeout. This instructs SQLite to retry acquiring the write lock internally for a specified duration before raising an SQLITE_BUSY exception.

PRAGMA busy_timeout = 5000; -- Timeout in milliseconds (5 seconds)

During this window, SQLite will use an exponential backoff algorithm to sleep and retry, which dramatically reduces application-level errors under peak load.

2. Lock Escalation and Immediate Transactions

SQLite has three transaction modes:

• DEFERRED (Default): The transaction starts without acquiring any locks. It begins as a read transaction and escalates to a write transaction only when a write operation is executed. This can easily lead to deadlocks if two connections start a deferred transaction, read data, and then both try to write.
• IMMEDIATE: The transaction attempts to acquire a reserved lock immediately. No other connection can start an IMMEDIATE or EXCLUSIVE transaction, but they can still read. This prevents deadlocks entirely.
• EXCLUSIVE: The transaction acquires an exclusive lock, blocking all reads and writes.

Rule of Thumb: If your transaction contains any write operations, always begin it with BEGIN IMMEDIATE TRANSACTION;.

BEGIN IMMEDIATE;
-- Write operations here
COMMIT;

---

Memory and Cache Optimization

SQLite's memory management directly impacts how many disk I/O operations your server performs. By default, SQLite allocates a tiny cache size (typically 2MB). For production workloads, you should scale this to keep your working set in memory.

Tuning Cache Size

To increase the cache size, use the cache_size pragma. A positive value specifies the number of pages, while a negative value specifies the cache size in kibibytes (KiB):

PRAGMA cache_size = -64000; -- Allocates approximately 64MB of RAM for cache

Memory-Mapped I/O (mmap)

Instead of reading database pages into user-space memory via standard read() and write() system calls, SQLite can map the database file directly into the application's virtual address space using the mmap system call. This allows the OS kernel to manage page caching directly, bypassing user-space buffer copies and significantly speeding up read queries.

PRAGMA mmap_size = 2147483648; -- Map up to 2GB of the database file into memory

If the database size is smaller than the mmap_size, the entire database is mapped into memory, turning disk reads into simple pointer arithmetic.

---

Custom VFS (Virtual File System) Layers for the Cloud Era

One of the most powerful architectural features of SQLite is its Virtual File System (VFS) abstraction. SQLite does not write directly to the OS filesystem; instead, it delegates all file operations (open, read, write, sync) to a VFS module.

This abstraction allows developers to write custom VFS layers to change how and where SQLite stores its data. This capability has fueled the creation of modern replication engines:

• Litestream: A streaming replication tool that runs as a separate process. It intercepts writes at the OS level and streams incremental WAL frames to object storage (like AWS S3) every second, offering point-in-time recovery with near-zero overhead.
• LiteFS: A custom FUSE-based VFS that distributes SQLite databases across a cluster of application nodes. It intercepts write operations at the file system level, replicating transactions to read replicas in real-time, enabling globally distributed SQLite deployments.

If you are running SQLite in a cloud environment where local disk persistence is ephemeral (such as AWS ECS, Kubernetes, or Fly.io), running a VFS-based replication tool is mandatory to ensure durability and high availability.

---

Production-Ready SQLite Configuration Blueprint

When initializing your database connections in your application bootstrap code (e.g., in Node.js, Python, Go, or Rust), execute this sequence of pragmas immediately after opening each connection:

-- Enable Write-Ahead Logging
PRAGMA journal_mode = WAL;

-- Reduce synchronization overhead without risking corruption
PRAGMA synchronous = NORMAL;

-- Prevent deadlocks by waiting for locks gracefully
PRAGMA busy_timeout = 5000;

-- Scale cache size to fit active working set (64MB)
PRAGMA cache_size = -64000;

-- Enable memory-mapped I/O for faster reads (1GB)
PRAGMA mmap_size = 1073741824;

-- Enforce foreign key constraints
PRAGMA foreign_keys = ON;

-- Prevent WAL file from growing indefinitely
PRAGMA journal_size_limit = 67108864; -- 64MB

-- Optimize index page allocation and query plans
PRAGMA auto_vacuum = INCREMENTAL;

Conclusion: When to Run SQLite in Production

SQLite is no longer just an embedded toy. When properly configured with WAL mode, memory mapping, and proper transaction boundaries, a single SQLite database can easily handle hundreds of concurrent requests and millions of queries per day on a modest virtual private server.

If your application requires complex, distributed write transactions across multiple geographical regions, or if your dataset exceeds several terabytes, a traditional system like PostgreSQL remains the correct tool. But if your system is read-heavy, fits within a few hundred gigabytes, and demands ultra-low latency, running SQLite directly on your application server is a highly performant, operationally simple, and cost-effective architecture choice.

Lisp Moving Forth Moving Lisp

Weird By Design

This chapter is a culmination of many macro techniques we have looked at so far in this book. Using macro abstractions we have developed, we create an implementation of one of my favourite programming languages: forth. Although this implementation embodies most of the important ideas of forth, it is very different and lispy. Though there are certain interesting uses for the code in this chapter, its primary purpose is to teach the concepts and fundamentals of forth meta-programming to a lisp audience and to be a platform for discussing the central theme of this book—creating and using duality of syntax with macros.

Forth, more so than every language except lisp, has a rich, fascinating history and I'm grateful for having discovered it. For that reason, and for everything else, this chapter is dedicated with love to my father Brian Hoyte who introduced me to forth and to computer programming. This chapter was partially inspired by [THREADING-LISP] and by the research of Henry Baker[LINEAR-LISP][LINEAR-LISP-AND-FORTH].

Forth was the first programming language that was created and developed without strong government, academic, or corporate sponsors—or at least the first such language to succeed. Instead of being motivated by the needs of a large organisation, forth was independently invented by Chuck Moore around 1968 to solve his own computing needs in astronomy, hardware design, and more. Since then, forth has been distributed, implemented, and improved upon by a passionate grass-roots user community[EVOLUTION-FORTH-HOPL2]. Contrast forth with the MIT (and later DARPA) patronage of early lisps and COMMON LISP, IBM's FORTRAN, and AT&T's unix language C.

Because of these roots, and because of a generally different philosophy of the role of computer software and hardware, forth is different. Even more so than lisp, forth looks weird. But like lisp, forth looks weird for a reason: it was designed with more in mind than style. Forth is weird by design, and this design relates to macros.

Today, forth is most commonly seen in so-called embedded platforms—computers that are severely resource constrained. It is a testament to the design of forth that the language can be entirely implemented on almost every programmable computer system ever created. Forth is designed to be as easy as possible to implement and experiment with. In fact, creating a forth clone is so profoundly trivial that inventing a forth-style stack based language or two is almost a rite-of-passage for programmers interested in the design of programming languages. Some stack based languages that can trace roots back to forth and have made interesting contributions are PostScript and Joy.

Often important forth implementation decisions are based on the exact resources of the computer that forth is being implemented on. Forth programmers have devised a set of abstract registers that need to be either mapped into real registers, mapped into memory locations, or possibly implemented in a different way altogether. But what do we do if we are implementing a forth on lisp, an environment with unlimited potential and few restrictions? Rather than simply impose an arbitrary mapping of forth abstract registers into lisp code, we try to take a step back. What would forth look like if Chuck had a lisp machine? Rather than fitting forth to the capabilities of an arbitrary machine, real or virtual, we explore a minimal set of forth abstract registers, optimised for simplicity and capability when implemented on lisp.

But really, searching for an optimal set of abstract concepts is what Chuck did while he created forth, using his experience of dozens of different forth implementations on as many architectures. This is why forth is so great. Like lisp, forth represents a high local maximum in the space of language design, and also like lisp, forth is not so much a programming language or a set of standards, but instead a building material and collection of wisdom regarding what works and what doesn't.

FORTH-REGISTERS

(defvar forth-registers
        '(pstack rstack pc
          dict compiling dtable))

The forth-registers variable is a list of symbols that represent abstract registers for our forth machine. Of course lisp doesn't think in terms of registers and fixnums, but instead variables and symbols. It may seem strange to start our development of a forth environment here, with just a list of variable names, but this is in fact always the first step in implementing a forth system. Creating a forth is an ingenious process of bootstrapping exceeded in beauty and cleverness only by lisp. A modest description of this process follows.

One of the characteristic features of forth is its direct access to the stack data structures used by your program both to pass parameters to subroutines and to keep track of your execution path throughout these subroutines. Forth is especially interesting because—unlike most programming languages—it separates these two uses of the stack data structure into two stacks you can fool with1 Most languages don't allow you to directly manipulate the stack at all. . In a typical C implementation, the parameters of a function call and its so-called return address are stored in a single, variable-sized stack frame for every function invocation. In forth, they are two different stacks called the parameter stack and the return stack, which are represented as our abstract registers pstack and rstack. We use the COMMON LISP push and pop macros, meaning these stacks are implemented with cons cell linked lists instead of the array data structures used in most forths.

The abstract register pc is an abbreviation for program counter, a pointer to the code we are currently executing. What forth code is and how we can point to it will be explained shortly, as will our abstract registers compiling and dtable.

FORTH-WORD

(defstruct forth-word
  name prev immediate thread)

Another building block of forth is its concept of a dictionary. The forth dictionary is a singly linked list of forth words, which are similar to lisp functions2 In other words, they aren't functions at all, but rather procedures. . Words are represented with a lisp structure. Structures are efficient slot-based data structures usually implemented as vectors. The name slot is for a symbol used to lookup the word in the dictionary. Notice that the forth dictionary is not stored alphabetically, but instead chronologically. When we add new words we append them onto the end of the dictionary so that when we traverse the dictionary the latest defined words are examined first. The last element of our dictionary is always stored in the abstract register dict. To traverse the dictionary, we start with dict and follow the prev pointer of the word structures, which either point to the previously defined word or to nil if we are at the last word3 Yes, sometimes lisp programmers use alternatives to cons cells for linked lists. .

FORTH-LOOKUP

(defun forth-lookup (w last)
  (if last
    (if (eql (forth-word-name last) w)
      last
      (forth-lookup
        w (forth-word-prev last)))))

Given w, a word to lookup, and last, a dictionary to search, forth-lookup will return either a forth word structure or nil depending on whether the word w was found in the dictionary or not. The comparison function eql is used instead of eq because—unlike lisp—forth allows words to be named by numbers and other non-symbols.

The immediate slot of our forth word is a flag indicating whether the word is immediate or not. Immediacy is a forth meta-programming concept we will explore in depth shortly. For now here is a rough analogy to its lisp counterpart: immediate words are like lisp macros in that they are forth functions to be executed at compile time instead of run-time. What? Only lisp is supposed to have macros. While it is true that the COMMON LISP macro system is much more powerful than any other macro system—including the best forth implementations—forth has extension capabilities that surpass almost all other languages. Like lisp, this capability is the result of a design philosophy: if it's good enough for the language implementor, it's good enough for the application programmer. Like lisp, forth doesn't really recognise the notion of a primitive. Instead, it provides a set of meta-primitives that can be combined to build the language that you, the programmer, desire. Like lisp, and unlike most Blub languages, extending the language in novel ways through the use of macros is not only possible, but encouraged. Like lisp, forth is not about style, but instead, power.

Cons Threaded Code

In the previous section, we focused on abstract registers. These registers are an important focal point, and that is why forth philosophy considers them so fundamental, but these registers are actually just a component of a more general concept: abstract machines. Probably the most distinguishing property of different forth systems are their implementations of threaded code. What forth means by threaded code is very different from the conventional meaning of preemptive scheduled, shared-memory processes4 This book does not recommend those types of threads for security and reliability reasons. . Forth threads have nothing to do with concurrency. They are a framework for talking about code compilation and meta-programming.

While lisp gives access to the tree data structure5 Actually a directed acyclic graph. of symbols that your program is compiled from and to before being assembled into memory, forth provides no symbolic manipulation. Instead, forth gives access to this process of assembling—threading—the code into memory. Although to outsiders the most apparent features of forth are its stacks and postfix notation, it is actually threads that make forth what it is. Forth is about stacks in the same way that lisp is about lists. They just happen to be the most applicable data structures to use for solving meta-programming problems—what forth and lisp are both really about.

The classic style of threading is known as indirect threaded code but most modern forths are implemented with direct threaded code. The difference involves a level of indirection. The low-level efficiency implications of this indirection depend on the underlying processor and we will not get into details here. There are many good tutorials on forth threading[STARTING-FORTH][MOVING-FORTH]. In memory, these styles of threading both consist of adjacent cells, which are fixnum machine words representing pointers. A small piece of tight machine code called the inner interpreter is usually tailored for the processor being used because of its important job: to follow the pointers of these forth threads, interpreting their meanings as it goes along. The default behaviour when encountering a cell is to push the current program counter location onto the return stack and then point the program counter to whatever is contained in the cell. When the inner interpreter reaches the end of a thread, it pops the return stack and resumes execution at this location—where it left off6 In most forths, the end of threads are indicated with the forth word exit. .

As you can imagine, this type of program storage makes for extremely small programs. A compiled forth word is just a consecutive array of fixnums, most of which represent pointers to other words. This has always been one of the advantages of forth. Because of the transparency in the threading of the program into memory, forth allows fine control over many programming trade-offs, including one of the most important: Execution speed versus program size. Threaded code lets us optimise our abstractions as close to our problems as possible, resulting in extremely fast, small programs. But just as lisp macros are about much more than just efficiency, so are forth threads. As much as lisp programmers, forth programmers tend to think of themselves as implementors instead of mere users. Forth and lisp are both about control—making your own rules.

There are at least two other common types of forth threading techniques: token threaded code and subroutine threaded code. These represent opposite directions to take when considering the speed versus size trade-off. Sometimes these threading techniques coexist with indirect and direct threaded code in the same forth. Token threading involves adding another level of indirection by using fixnums even smaller than pointers to represent words in the thread. At the other end of the spectrum is subroutine threading. This type of threaded code is becoming popular and the best modern forth compilers partially use subroutine threading. Instead of consecutive pointers to words for the inner interpreter to follow, subroutine threaded code stores inline machine instructions to call these pointers. In subroutine threaded code, the inner interpreter disappears—it is actually implemented by the hardware (or virtual machine). Subroutine threaded code is usually considered an opaque block, one that only a special, non-programmable compiler can manipulate. Especially when various optimisations are made to the code, these opaque blocks start to look nothing like uniform, cell-based threads. Almost all non-forth compilers compile only to subroutine threaded code and don't imagine that you would ever want to do anything else, leading to this peculiar definition:

A Flub is a language that only considers subroutine threaded code or a language implementation that only provides subroutine threaded code.

For example, C is a Flub because it only provides programmers means to create functions—opaque blocks of subroutine threaded code. Certainly we can implement an inner interpreter in C to handle indirect threaded code7 Direct threaded code is possible but more difficult. and bootstrap a stack-based language with this program, but then we are no longer programming in C. Almost all Blub languages are Flubs. Forth as an abstract machine is, as we've just described, not a Flub. As we will see, forth gives programmers/implementors lots of control over how their programs are compiled.

Is lisp a Flub? Interestingly, lisp was probably the first non-Flub programming language but has mostly turned into Flub. Although not strictly required to by the standard, most COMMON LISP compilers only compile functions to blocks of opaque machine code and, as such, are Flubs. But in very early versions of lisp, functions were stored as lists—a strange sort of code threading not entirely unlike forth threads. While this did allow some very clever run-time tricks, including giving meaning to cyclic code, it was hopelessly inefficient. Unlike forth's many types of threading—which have been efficiently implemented on almost all architectures—this internal representation for lisp functions was intolerable and lisp was changed to allow (extremely) efficient code. The consequence, for meta-programmers, is that most implementations of COMMON LISP are Flubs.

But there is a difference between features that are impossible to add to a language and features we can add with macros. With macros we can extend the language in any way we want and it remains lisp. COMMON LISP lacks threaded code in the same sense that it lacks continuations and first-class macros: they are omitted from the language deliberately and left for macro writers to implement as needed. One of the most important results of this chapter and its code is to show that even when they are Flubs, lisp languages can transform into non-Flub languages through macros. Non-Blub implies non-Flub, or, in other words, if you can't turn a language into a non-Flub, it must be a Blub. However, the opposite is not true. Non-Flub languages like forth remain Blubs and the most straightforward way to turn them into non-Blubs currently known is to implement lisp environments with them—and then you're programming lisp.

FORTH-INNER-INTERPRETER

(defmacro forth-inner-interpreter ()
  `(loop
     do (cond
          ((functionp (car pc))
             (funcall (car pc)))
          ((consp (car pc))
             (push (cdr pc) rstack)
             (setf pc (car pc)))
          ((null pc)
             (setf pc (pop rstack)))
          (t
             (push (car pc) pstack)
             (setf pc (cdr pc))))
     until (and (null pc) (null rstack))))

Instead of using consecutive memory cells to represent threads as with indirect/direct threaded code, our forth takes advantage of lisp's dynamic typing and cons cell list structure. We call this cons threaded code. The macro forth-inner-interpreter expands into code that is capable of following these cons cell linked-list threads. It might seem strange to start programming the logic for our forth environment here—with a macro designed to be expanded into some as-of-yet unknown expression—but this is in fact a desirable lisp programming pattern. Because macros let us start programming anywhere we want, why not start with the really interesting, driving bits of a program? These are the bits that will have the most influence on the program's ultimate design.

The definition of forth-inner-interpreter is itself a concise definition of what we mean by cons threaded code. The car of every cons cell is points to either a function, another cons cell, or some other lisp atom. Functions are executed as they are encountered. Notice that it is left for the function itself to update the pc register. If another cons cell is found in the thread it is assumed to indicate a subroutine call—a word invocation. Our inner interpreter will push the pc resume location onto the return stack and then jump to this new thread. If some other lisp atom is encountered, it is simply pushed onto the parameter stack and execution resumes at the next cell in our thread. The inner interpreter will return once it reaches the end of its thread and has no other threads to return to on its return stack.

PRIM-FORMS

;; Prim-form: (name immediate . forms)
(defvar forth-prim-forms nil)

(defmacro def-forth-naked-prim (&rest code)
  `(push ',code forth-prim-forms))

(defmacro def-forth-prim (&rest code)
  `(def-forth-naked-prim
     ,@code
     (setf pc (cdr pc))))

But of course functions can't update the pc variable unless they are defined in its lexical scope8 Except see pandoric macros. so we employ another macro technique: instead of using defun, we create a similar interface that does something completely different. Def-forth-naked-prim feels similar to creating defun defined functions except that the code it expands into pushes the user provided forms onto a list stored in forth-prim-forms. Our eventual macros will use these forms to define the forth primitives inside its lexical scope. Because these forms will always be expanded into this environment, we are free to write code that uses all of our forth abstract registers like pc, pstack, etc.

Primitives defined with def-forth-naked-prim will not update the pc variable to the next cons cell in the thread. For the majority of primitives we should use def-forth-prim so the usual update is performed. Both of these macros expect the first argument to be a symbol used to refer to the primitive and the second to be a boolean indicating whether the primitive is immediate or not. The remainder of the arguments are lisp forms to be evaluated when the primitive is execute.

BASIC-PRIM-FORMS

(def-forth-prim nop nil)

(def-forth-prim * nil
  (push (* (pop pstack) (pop pstack))
        pstack))

(def-forth-prim drop nil
  (pop pstack))

(def-forth-prim dup nil
  (push (car pstack) pstack))

(def-forth-prim swap nil
  (rotatef (car pstack) (cadr pstack)))

(def-forth-prim print nil
  (print (pop pstack)))

(def-forth-prim >r nil
  (push (pop pstack) rstack))

(def-forth-prim r> nil
  (push (pop rstack) pstack))

Eight simple primitives—none of them naked or immediate—are presented now. Nop is a dummy instruction that does nothing ("no operation"). The * primitive is the multiplication operator: it pops the top two values from the parameter stack, multiplies them together, then pushes the result back. Dup is short for "duplicate" and pushes the top value on the parameter stack onto the parameter stack again leaving two duplicate values. Swap will exchange the top two parameter stack elements using a very useful COMMON LISP macro: rotatef. It is no coincidence that forth also has (stack-based) rotation mechanisms. Print pops the parameter stack and prints it. >r transfers a value from the parameter stack to the return stack, r> does the opposite.

Does the name * violate our important variable capture rule from section 3.5, Unwanted Capture that forbids us from rebinding functions defined by COMMON LISP? No, because we haven't actually used this symbol to bind any functions—it's just the first element in one of the lists in forth-prim-forms. We have done nothing wrong. Symbols are independent from the functions or macros they are sometimes used to indicate. We can use any symbols anywhere, so long as we are careful to never violate our important variable capture rules. This only comes into play when writing lisp; we are writing forth.

Duality of Syntax, Defined

If you remember nothing else from this book, remember the message of this section. Here we finally define and explain a concept we have touched upon throughout: duality of syntax. This section assumes you have read at least the three introductory chapters, chapter 6, Anaphoric Macros, and the preceding forth sections.

To most lisp programmers the fact that programming in lisp is more productive and, eventually, more natural than programming in Blub is empirically obvious, but answering why this is the case is much more difficult. While it's true that lisp gets its amazing power of expression from macros—and we have seen many interesting ones in this book and elsewhere—all explanations so far feel unsatisfactory. What is the real advantage of macros? A partial explanation certainly includes brevity, making your programs short. Here is its definition:

Let L be a programming language, F a feature in that programming language, and A an arbitrary program in L. F provides a brevity feature if A is shorter than it would be in a version of L without F.

Brevity features provide the basis and rational for the theory of brevity:

The effort required to construct a program is inversely proportional to the amount of brevity features available in the programming language used.

The theory of brevity is based on the idea that if your programming abstractions make the expression of programs very short and concise, writing them becomes easier because less code needs to be written. Our CL-PPCRE read macros are examples of brevity features: they shorten the rather long CL-PPCRE function names into concise, Perl-style expressions that save us key-strokes every time we use them. The theory of brevity is very applicable to writing small programs for which we know where we want to go when we start9 Often called Perl-one-liners, even when not written in Perl. . Unfortunately, most programs aren't like this. Most programs—at least the interesting ones—are created iteratively through a series of interactive write-test cycles that take into account feedback at each step along the way. Your abstractions may be brief, but if you're always having to change them to different (perhaps equally brief) abstractions, you likely won't save much effort. Instead of considering the length of the final program, maybe we should consider the length of the process required to get there.

In every language, programs end up looking different from how they start. Most programs start with just a simple sketch that is filled out and detailed as the author learns more about the problem. Before we come back to brevity and duality, this chapter walks us through the development of a simple program that will motivate the discussion: our forth environment.

Hm, where were we? Ah yes, we had rambled on a lot about abstract registers, abstract machines, and threaded code, as well as defining a word lookup utility called forth-lookup, an inner interpreter for our cons threaded code, and a system for collecting lists representing primitives in our forth system. But what will forth on lisp be? Well, what is the most natural form for any abstraction that mixes behaviour and state? Closures, of course. Our old friends, let and lambda. Hacking up this idea might give the following macro:

(defmacro new-forth ()
  `(let ,forth-registers
     (forth-install-prims)
     (lambda (v)
       (let ((word (forth-lookup v dict))) 
         (if word
           (forth-handle-found)
           (forth-handle-not-found))))))

Our list of forth abstract registers, forth-registers, gets spliced directly into the expansion, initially binding all of the abstract registers to nil. Notice that we have left a lot of holes in the functionality of this macro. We have discovered that we are going to have to define a macro forth-install-prims which installs our primitive forms, as well as the macros forth-handle-found and forth-handle-not-found. But the most important thing we learn from this sketch is that, yes, this closure design looks like it could work. The idea, which came to us by just following the default lisp design, entails forth being a closure that is invoked once for every word we want to give it. Our sketch outlines an implementation for the following use case. Here we imagine creating a new forth environment:

(defvar my-forth (new-forth))

Here is some forth code for squaring the number 3 and printing the result:

3 dup * print

We could execute it on our forth environment like so:

(progn
  (funcall my-forth 3)
  (funcall my-forth 'dup)
  (funcall my-forth '*)
  (funcall my-forth 'print))

GO-FORTH

(defmacro! go-forth (o!forth &rest words)
  `(dolist (w ',words)
     (funcall ,g!forth w)))

Although this is a clumsy interface to use, we are programming lisp so we know we can always create a macro to hide these details, and that is exactly what is done with the go-forth macro. Notice that go-forth uses the automatic once-only functionality of defmacro! because the first argument provided to go-forth is inside a loop defined with dolist and will probably not be evaluated exactly once as might be intended by users of the macro. With go-forth, feeding forth code into our forth environment becomes much cleaner:

(go-forth my-forth
  3 dup * print)

At this point it might occur to us that we will eventually want to execute some forth bootstrapping code when creating new forth environments. So we need to be able to invoke the closure while creating it. This might require changing the program's let over lambda design or possibly creating some sort of wrapper function around our new-forth macro that uses the new-forth macro, loads in the standard library, and returns the resulting forth.

FORTH-STDLIB

(defvar forth-stdlib nil)

(defmacro forth-stdlib-add (&rest all)
  `(setf forth-stdlib
         (nconc forth-stdlib
                ',all)))

Because forth code is just a list of symbols and other atoms, our standard library that provides all of the bootstrapping we need (except for a few more primitives) can be stored in a list. The variable forth-stdlib keeps this list of forth code to be executed when new forths are created and the forth-stdlib-add macro expands into lisp code that will concatenate new forth code onto the forth-stdlib list.

What is the easiest way to adapt new-forth to support loading this standard library? Do you remember the alet macro we wrote in section 6.3, Alet and Finite State Machines? The purpose of this macro was to create a duality of syntax with COMMON LISP's let while binding the anaphoric variable this around the provided code. This refers to the result that will be returned from alet—the forth closure.

So changing our sketch is even easier than expected. All we have to do is change the first let symbol in our sketch to an alet and then add some code to load the standard environment into this, the forth closure10 Done after the primitives have been installed so that the standard library can make use of them. . We didn't have to re-arrange anything because alet's syntax was deliberately aligned with let's. Here is what this next iteration looks like:

(defmacro new-forth ()
  `(alet ,forth-registers
     (forth-install-prims)
     (dolist (v forth-stdlib)
       (funcall this v))
     (lambda (v)
       (let ((word (forth-lookup v dict))) 
         (if word
           (forth-handle-found)
           (forth-handle-not-found))))))

Remember that alet introduces a layer of indirection using a closure and therefore makes our forth environment slightly less efficient. However, just as we don't know if this efficiency burden will be too much, we also don't know we won't end up needing this indirection. To eliminate the indirection, use the alet% macro defined just prior to alet.

Perhaps now, or perhaps later on when we're trying to build and debug our forth environment, it might occur to us that it would also be useful to be able to access the forth abstract registers from outside the forth environment. Unfortunately, these variables are closed over with a let over lambda. We will have to change our program again to make them accessible. There are, of course, many ways to do this. We could embed and return multiple closures in our forth environment, some of which could save and access the abstract registers, or we could re-consider our let over lambda strategy entirely. But before we do that, are there any dualities to help us? Remember plambda from section 6.7, Pandoric Macros? Its purpose was to create a duality of syntax with lambda, but one that creates closures that are actually open to the outside world. Changing our sketch to support this is a simple matter of prefixing a p character onto the lambda we return as our closure and adding the list of variables we want to export. Our list is conveniently available to us in forth-registers11 Although splicing these variables in with unquote will only work when writing macros, read macros let us do similar things when writing functions. . Our sketch becomes:

(defmacro new-forth ()
  `(alet ,forth-registers
     (forth-install-prims)
     (dolist (v forth-stdlib)
       (funcall this v))
     (plambda (v) ,forth-registers
        (let ((word (forth-lookup v dict))) 
          (if word
            (forth-handle-found)
            (forth-handle-not-found))))))

With the forth closure opened up, we have available the following use case. This pushes five items onto a forth stack:

* (go-forth my-forth
    1 2.0 "three" 'four '(f i v e))

NIL

And we can pandorically open my-forth to examine its parameter stack:

* (with-pandoric (pstack) my-forth
    pstack)

((F I V E) FOUR "three" 2.0 1)

NEW-FORTH

(defmacro new-forth ()
  `(alet ,forth-registers
     (setq dtable (make-hash-table))
     (forth-install-prims)
     (dolist (v forth-stdlib)
       (funcall this v))
     (plambda (v) ,forth-registers
       (let ((word (forth-lookup v dict))) 
         (if word
           (forth-handle-found)
           (forth-handle-not-found))))))

This was the process performed to arrive at our final version of the macro new-forth. The final definition is identical to the last sketch except that it also sets the dtable abstract register to point to a hash-table (explained soon).

Programming, at least the interesting kind, is not about writing programs, but instead, changing them. In terms of productivity, brevity only takes us so far. We could rename lambda to, say, fn, but this brevity feature doesn't save much except for a few key-strokes here and there12 Many of us think lambda is just fine, thank you. . What does save us effort, however, is having lots of abstractions similar to lambda that we can use to change what code means without modifying the code itself much. Duality of syntax saves us effort.

Just like putting earmuffs on your special variable names can bite you by forcing you to add or remove asterisks if you change your mind about whether a variable should be special or lexical13 Or possibly leaving incorrect documentation in code. , needlessly separating syntax and avoiding dualities can cause much pointless effort during programming. Another example: sharp quoting your lambda forms is a bad idea because it means you have just that much more to modify when you decide a function really needs to be an alambda or when you decide to use the lambda form in the function position of a list. Generalised variables also provide a very important duality: when writing macros, the same form can be spliced into expansions for both accessing and modifying the variable. COMMON LISP's dual meaning for the empty list and the false boolean value is yet another example—there is no real reason these two should be the same except for duality of syntax. Duality is also why this book has promoted closures instead of other CLOS features14 Here the word "other" refers to the fact that even programming with closures is, in COMMON LISP, a feature of CLOS. CLOS is so fundamental that you can't escape it (nor would you want to). such as defclass and defmethod. There is usually less friction when modifying programs that use closures than when modifying programs that use classes and objects because we have so many good dualities of syntax for closures and because programming macros that build closures is more uniform15 That said, generic functions are very important because they introduce dualities with multi-methods. . With these and other examples in mind, we can finally give a clear definition of what is meant by duality of syntax:

Let L be a programming language, F a feature in that programming language, and A and B arbitrary programs in L. F provides a duality of syntax feature if the modifications required to change A into B become fewer than in a version of L without F.

Which leads to the theory of duality:

The effort required to construct a program is inversely proportional to the amount of dual syntax available in the programming language used.

While the concept of duality of syntax and the impact of its benefits are both quite clear, how to actually design good dualities is much less so. What are the most useful dualities in a certain language? How can we tell which of two different languages will provide better dualities of syntax for some given problem?

Because with lisp we control the programming language completely, we can design our language with as much or little dual syntax as we please. Following this train of thought is, in my opinion, the most fruitful area of programming language research today. Using lisp macros, just how similar can we make all of our disparate programs to one another so that changing them into new programs becomes that much easier16 Sometimes changing programs into new programs is called development, especially when the resulting program is larger than the original. ?

In both the definitions of brevity and duality, whether the feature F is effective or not depends on the programs being written or changed. Sometimes features that provide brevities or dualities can, in certain situations, actually increase the effort required. The best approach may be to provide as many useful brevity and duality features as possible while removing the ones that end up being more trouble than they are worth.

Going Forth

In this section, we really get forth going by filling in the holes left in the new-forth macro from the previous section. After having verified that the forth threading mechanism works, we bootstrap a forth programming environment and, along the way, explain what forth immediacy is and how it relates to lisp macros.

FORTH-INSTALL-PRIMS

;; Prim-form: (name immediate . forms)
(defmacro forth-install-prims ()
  `(progn
     ,@(mapcar
         #`(let ((thread (lambda ()
                           ,@(cddr a1))))
             (setf dict
                   (make-forth-word
                      :name ',(car a1)
                      :prev dict
                      :immediate ,(cadr a1)
                      :thread thread))
             (setf (gethash thread dtable)
                   ',(cddr a1)))
         forth-prim-forms)))

In the definition of new-forth, we left a hole in the macro that is to be filled by forth-install-prims. We would like to use a named abstraction without throwing out our lexical environment, so it must be a macro. The point of this macro is to compile and install the primitives into the forth dictionary when a new forth instance is created. Forth-install-prims expands into a progn form with each sub-form being an instruction to append a primitive word onto the dict linked list, wrap the provided code in a lambda, and set the word's name and immediate slots. In addition, the function created for each word by lambda, called thread, is added to our dtable hash-table (explained soon). Because all of these functions will be created in the scope of our original new-forth macro, they have full access to the forth environment specified by our abstract registers. Notice that the thread binding does not capture thread from any user provided code so we don't need to name it with a gensym.

We have said that forth provides a meta programming system not completely dissimilar from lisp's and that this system is based around a concept called immediacy. In traditional forths, there is a variable called state which is either zero or non-zero. If it is zero, the forth is considered to be in a regular interpreting (executing) state. If we give a word to forth in this state, that word will be looked up and executed. If, however, the base variable is non-zero, the forth is said to be in a compilation state. If we present a word to forth in this state, the address of the presented word is appended to the current thread being compiled—generally the most recently created word in the dictionary. There is one exception, however, and this is the important point about immediacy. If we are in compilation state and we are given an immediate word, that word will be executed instead of compiled. So, like lisp, forth allows us to execute arbitrary forth code at compile time.

FORTH-PRIMS-COMPILATION-CONTROL

(def-forth-prim [ t ; <- t means immediate
  (setf compiling nil))

(def-forth-prim ] nil ; <- not immediate
  (setf compiling t))

Because we are building our forth as an abstract machine on lisp, we don't have to suffer arbitrary mappings of fixnum values to true and false. In lisp, we have a dynamic type system that lets us enjoy arbitrary mappings of all values to true and false. In place of the forth variable state, our forth system uses the compiling abstract register to store our compilation state as a lisp generalised boolean. The traditional forth words used to control the compilation state are [ and ], the open and close square brackets. [ takes us out of compilation mode and thus necessarily must be an immediate word. ] puts us back into compilation mode and so is only executed when we are interpret mode and doesn't have to be immediate. This choice of symbols may seem strange now but will become clearer in high level forth code. These square brackets allow us to designate a block of code to be executed in the middle of compiling a forth thread. In a certain sense, these brackets are like lisp's backquote and unquote operators. Here is how these words are usually used in forth code:

... compiled words ...
[ interpret these words ]
... more compiled words ...

Like most of forth, these words are transparently specified which permits us use them in unusual ways. For example, these words aren't balanced in the same sense that lisp parenthesis are. If we choose, we can use them in the opposite direction:

... interpret these words ...
] compile these words [
... more interpreted words ...

We even have the appearance of nesting, but this is not really nesting because we only have a single boolean state: Compiling or not-compiling.

... compiled words ...
[ interpret these words
  ] compile these words [
  interpret these words
]
... more compiled words ...

FORTH-COMPILE-IN

(defmacro forth-compile-in (v)
  `(setf (forth-word-thread dict)
         (nconc (forth-word-thread dict)
                (list ,v))))

Our forth uses the forth-compile-in macro as an abbreviation macro. This macro compiles a forth word into our current thread, the thread of the last word created. Because our threads are represented by cons cells, we can use the lisp function nconc to simply append a pointer to the destination word's thread onto our current thread.

FORTH-HANDLE-FOUND

(defmacro forth-handle-found ()
  `(if (and compiling
            (not (forth-word-immediate word)))
     (forth-compile-in (forth-word-thread word))
     (progn
       (setf pc (list (forth-word-thread word)))
       (forth-inner-interpreter))))

Another hole we left in the new-forth macro was what forth should do if it was able to lookup a provided word in the dictionary. This hole is filled by the macro forth-handle-found. This macro implements forth immediacy as described above. If we're compiling and the word looked up is not immediate, we compile it into our current thread. Otherwise we set our program counter pc to point to the thread of the looked up word and run the inner interpreter to execute the word. Recall that this macro will be expanded into a lexical environment in which word is bound to the looked-up forth word.

FORTH-HANDLE-NOT-FOUND

(defmacro forth-handle-not-found ()
  `(cond
     ((and (consp v) (eq (car v) 'quote))
        (if compiling
          (forth-compile-in (cadr v))
          (push (cadr v) pstack)))
     ((and (consp v) (eq (car v) 'postpone))
        (let ((word (forth-lookup (cadr v) dict)))
          (if (not word)
            (error "Postpone failed: ~a" (cadr v)))
          (forth-compile-in (forth-word-thread word))))
     ((symbolp v)
        (error "Word ~a not found" v))
     (t
        (if compiling
          (forth-compile-in v)
          (push v pstack)))))

Our final hole in new-forth was what forth should do if it isn't able to find a word in its dictionary. Forth-handle-not-found fills this hole and implements some special cases. Recall that forth-handle-not-found will be expanded into a lexical environment that contains a binding v that refers to the value passed to forth. We also know that if this code is invoked, v will not refer to any word in the dictionary. If v is a symbol, forth-handle-not-found will throw an error. If the value is not a symbol, the behaviour is to push v onto the parameter stack or, if we are compiling, to compile it into the current thread. Two special cases are checked for, however. If v is a list with the first element quote, we push the quoted value onto the parameter stack. This is so that we can push symbols onto the parameter stack without them being interpreted as words. The second special case is if v is a list with the first element postpone. Postpone is an ANSI Forth word that unites and clarifies a couple traditional forth words. Postpone is used to always compile a word, even if that word is immediate. So, if we are in compilation mode a postponed immediate word will be compiled into our current thread even though it is immediate. Here is an example of postponing the [ word:

... compiling ...
(postpone [)
... still compiling ...

With all the holes filled in our new-forth macro we are now in a position to create new forth instances with the new-forth macro. Earlier we created a special variable called my-forth with defvar. Even if we hadn't, we could implicitly declaim it special along with assigning it a value with a top-level setq17 Some implementations prevent you from doing this. The solution is to upgrade to one that does (such as CMUCL) or to use defparameter instead of setq in such cases. :

* (setq my-forth (new-forth))

#<Interpreted Function>

We can now use forth with the go-forth macro:

* (go-forth my-forth
    2 3 * print)

6
NIL

But so far we have only defined the words dup, *, and print. To do anything useful we will need more primitives. Like lisp, production quality forth implementations have a large number of words defined for programmer convenience. Over decades of use, many common programming patterns have been identified, abstracted into words, and then introduced into common forth vernacular. Like lisp, having the ability to extend the language defined as part of the language has resulted in much valuable experimentation. Because it is this philosophy and process we are investigating, we will not define many forth words that experienced forth programmers rely on. Instead, we aim for the minimal set of primitives required to explain forth's meta-programming system so we can compare it to lisp macros.

FORTH-PRIMS-DEFINING-WORDS

(def-forth-prim create nil
  (setf dict (make-forth-word :prev dict)))

(def-forth-prim name nil
  (setf (forth-word-name dict) (pop pstack)))

(def-forth-prim immediate nil
  (setf (forth-word-immediate dict) t))

Three more primitives are defined, none of which are immediate or naked: create, name, and immediate. The create word appends a nameless word to the dictionary. Name pops a value from the parameter stack and sets the name of the last word in the dictionary to this value. Immediate simply sets the last word defined to be an immediate word. By default, words are not immediate.

Recall that we can execute code with our go-forth macro on our my-forth forth environment. Here we square the number 3 and print the result:

* (go-forth my-forth
    3 dup * print)

9

Do we have enough of forth to start bootstrapping forth with forth words itself? Although we don't really have defining words yet, thanks to the transparent specification of threaded code, we can begin to write forth words using forth. For example, here we use create to append a new empty word to the dictionary:

* (go-forth my-forth
    create)

NIL

We now use ] to begin compiling, add the words dup and * to the thread, then use [ to take us out of compilation mode:

* (go-forth my-forth
    ] dup * [)

NIL

Now we have a new word in our dictionary—one with a complete forth thread that, when executed by our inner interpreter, will square the number on the top of the stack. But this word isn't very useful unless we have a way of accessing it. We can give it a name using the word name. The name we assign will be the key we use to access our new thread:

* (go-forth my-forth
    'square name)

NIL

Notice how the first value we passed to forth is quoted. Recall that we decided this behaviour should result in forth pushing the symbol square onto the parameter stack. This symbol is then consumed by the word name. Now that our word is named, we can evaluate it like any other word using the symbol square:

* (go-forth my-forth
    3 square print)

9
NIL

So the general technique for creating new words is the following pattern:

create
] ... compiled words ... [
'whatever name

FORTH-START-DEFINING

(forth-stdlib-add
  create
    ] create ] [
  '{ name)

But we can use a bit of forth meta-programming to improve this interface. A definition of a new forth word, {, is added to the standard library. Its thread consists of two pointers, the first pointing to the word create and the second pointing to the word ]. So when the thread for this word is executed, it will append a new word to the dictionary and flip us into compilation mode. Forth generally uses the : word for this purpose, but this conflicts with lisp's use of : so we have elected to use { to start word definitions.

FORTH-STOP-DEFINING

(forth-stdlib-add
  { (postpone [) [
  '} name immediate)

Similarly, we add a complementary word } to the standard library (replacing the traditional forth ;). There is actually no reason to define this word—the only thing it does is take us out of compilation state. We already have the word [ to do that for us. Despite this, defining { is useful because it gives us normal balanced brackets18 As opposed to the backwards balanced brackets forth's square brackets give for word defining. by creating a pair of words { and } that make defining new words intuitive.

We can now create a forth to take advantage of these new standard library features (throwing out our previous definition of the word square):

* (setq my-forth (new-forth))

#<Interpreted Function>

Here is what square looks like with the new word defining words { and }:

* (go-forth my-forth
    { dup * } 'square name)

NIL
* (go-forth my-forth
    5 square print)

25

And new threads can refer to our custom created words just as easily as primitives. Here is how we can define the word quartic as a thread with two pointers to our square word:

* (go-forth my-forth
    { square square } 'quartic name)

NIL

(Expt 1/2 4) is indeed 1/16:

* (go-forth my-forth
    1/2 quartic print)

1/16
NIL

Because non-symbols are directly compiled into the forth thread and our inner interpreter treats non-functions as data items to push onto the stack when encountered, we can include numbers into a word definition:

* (go-forth my-forth
    { 3 } 'three name
    three three * print)

9
NIL

Recall that we look up all elements passed to forth to see if they have been previously named in the dictionary using the eql function. The consequence of this is that we can use any lisp object to name a word. Here, we use a number19 In many forths, common numbers like 0, 1, -1, etc are defined as words because a compiled reference to a word typically takes less memory than a compiled literal. :

* (go-forth my-forth
    { 4.0 } '4 name
    4 4 * print)

16.0
NIL

MEMORY-PRIMS

(def-forth-prim @ nil
  (push (car (pop pstack))
        pstack))

(def-forth-prim ! nil
  (let ((location (pop pstack)))
    (setf (car location) (pop pstack))))

Forth is an excellent language for learning how to use pointer scoping. Forth defines two simple operators that are used to read and write values from memory: @ (fetch) and ! (store). Because our forth words are stored in cons cells instead of memory words, dereferencing a pointer with fetch is implemented by taking the car of a pointer. Setting it with store is implemented by setting its car using setf. Fetch will pop a value from the parameter stack, assume it is a cons cell, fetch its car, then push that on the stack. Store will pop a value from the parameter stack, assume it is a cons cell, pop another value from the stack, and store it into the first value's car. For example, here is how we can create and print a cyclic list:

* (let ((*print-circle* t))
    (go-forth my-forth
      '(nil) dup dup ! print))

#1=(#1#)
NIL

So now we're programming in forth using threaded code. Or are we? Did we ever leave lisp? The distinction between the two languages is so blurry that it can barely be discerned. The remainder of this chapter attempts to make this distinction even blurrier while explaining forth meta-programming further.

Going Forther

FORTH-UNARY-WORD-DEFINER

(defmacro forth-unary-word-definer (&rest words)
  `(progn
     ,@(mapcar
         #`(def-forth-prim ,a1 nil
             (push (,a1 (pop pstack))
                   pstack))
         words)))

COMMON LISP has a lot of functions that we would like to be able to include in our forth threads. Forth-unary-word-definer expands into as many def-forth-prim forms as elements passed to its macro body. The elements are assumed to be symbols representing either functions or macros, but they could also be lambda forms. The only restriction with primitives named by lambda forms is that to invoke such primitives you will need to pass the same (eq) lambda form to the forth environment. Here is the expansion when passed one symbol, not:

* (macroexpand
    '(forth-unary-word-definer
       not))

(PROGN
  (DEF-FORTH-PRIM NOT NIL
    (PUSH (NOT (POP PSTACK))
          PSTACK)))
T

We can use any COMMON LISP function that accepts one argument and forth-unary-word-definer will define a forth primitive that applies this function to the top element of the forth parameter stack.

FORTH-BINARY-WORD-DEFINER

(defmacro! forth-binary-word-definer (&rest words)
  `(progn
     ,@(mapcar
         #`(def-forth-prim ,a1 nil
             (let ((,g!top (pop pstack)))
               (push (,a1 (pop pstack)
                          ,g!top)
                     pstack)))
         words)))

An extension on this idea is forth-binary-word-definer which does the same thing except for operators that accept two values. The forth convention of treating the second-to-top element as the first argument to binary functions like - and / is enabled by creating a temporary let binding to hold the top element of the parameter stack. Here is an expansion for the word -:

* (macroexpand
    '(forth-binary-word-definer
       -))

(LET ()
  (PROGN
    (DEF-FORTH-PRIM - NIL
      (LET ((#:TOP1767 (POP PSTACK)))
        (PUSH (- (POP PSTACK) #:TOP1767)
              PSTACK)))))
T

FORTH-AND-LISP-WORDS

(forth-unary-word-definer
  not car cdr cadr caddr cadddr
  oddp evenp)
(forth-binary-word-definer
  eq equal + - / = < > <= >=
  max min and or)

Exercise: When we use forth-binary-word-definer, how is it possible that we are able to treat macros like and and or as if they were first-class values?

Difficult exercise: Why was using a gensym (g!top) necessary for avoiding unwanted variable capture in forth-binary-word-definer? Hint: We've already discussed it in this section.

So these macros let us add various lisp functions to our forth primitive environment so we can use them from within forth. Here is an example use of a unary primitive, cadr:

* (go-forth my-forth
    '(a (b) c) cadr print)

(B)
NIL

And a binary one, <:

* (go-forth my-forth
    2 3 < print)

T
NIL

So far our forth threads have been directed acyclic graphs, that is they consist of cons cell structure that doesn't point to itself anywhere (isn't self-referential) and which eventually terminates at our primitives, the leaves of the tree. For example, we can use pandoric macros to get at the thread that we created in the previous section when we defined the quartic word:

* (with-pandoric (dict) my-forth
    (forth-word-thread
      (forth-lookup 'quartic dict)))

((#<Interpreted Function>   ;; square->|->dup
  #<Interpreted Function>)  ;;         |->*
 (#<Interpreted Function>   ;; square->|->dup
  #<Interpreted Function>)) ;;         |->*

The above comments only show it from the angle we printed the form in lisp. What we can't see from the code or the comments is that this thread structure is actually shared. To see that, use eq:

* (eq (car *) (cadr *))

T

Or look at it with *print-circle* set:

* (let ((*print-circle* t))
    (print **)
    t)

(#1=(#<Interpreted Function>  ;; square->|->dup
     #<Interpreted Function>) ;;         |->*
 #1#)                         ;; ————|
T

Threaded code can allow forth amazing memory and size advantages. Entire forth systems are compiled code that is threaded together like this—from the network drivers to the highest level user programs. What's more, notice that we can cleanly extract the thread from quartic without taking a lot of extraneous other threads. For instance, we have many more primitives in our language, such as + and cadddr but they don't appear at all in the thread above. It's almost like we have a mark-sweep garbage collection algorithm that extracts only the threads that will be needed to execute a given forth word. In lisp this process is called tree shaking and is generally not very effective. In forth, however, it is extremely effective.

Unfortunately, the quartic thread we pandorically extracted from my-forth is not really that useful to us. It still permanently resides in the my-forth closure. That is, the lambda expressions representing the dup and * primitives have had their references to the forth abstract registers captured by an expansion of our macro, new-forth. Can we pull this code back up to the lisp macro surface so as to embed it in new programs? We will return to this soon but first we go a bit further into forth meta-programming.

At some level in any language—usually a level hidden from the programmer—it is necessary for code to be able to refer to itself. The most convincing example of this necessity is the observation that code needs to be able to refer to itself somehow in order to implement looping, recursion, and conditional expressions like if statements. The difference between Flub languages and non-Flub languages is that Flub prevents you from directly customising how and where self-references are inserted. But, as we are doing now, lisp's non-Blub status means we can make it a non-Flub.

Our forth system in its current state (without it being able to insert self-references) is almost a pure Flub. In a similar sense as to how pure functional languages deliberately define a language to be devoid of side-effects and non-static mappings, pure Flub languages are defined to be devoid of self-referential code constructs like loops and recursion. A consequence of this is that interpreting pure Flub threads will always terminate. Our forth environment is not completely pure because we can—and will—violate this, but is pure in the sense that if only used as so-far described will result in pure Flub threads. Pure Flub is not very useful so let's ruin the Flub purity of our forth environment. Instead of going in the Flub direction—towards Flub languages like COMMON LISP where code threading is opaque and inaccessible—let's go in the direction of forth and make this attribute of code macro customisable.

BRANCH-IF

(def-forth-naked-prim branch-if nil
  (setf pc (if (pop pstack)
             (cadr pc)
             (cddr pc))))

The branch-if primitive is the first naked primitive presented so far. Recall that naked primitives are primitives that don't update the program counter abstract register (pc) automatically. Instead, they must update it themselves. Branch-if will pop a value off the parameter stack. If the value is non-null, pc is set to the contents of the next cell in the thread being interpreted. If the value is nil, pc is resumed as usual except that it skips over the next cell in the thread being interpreted.

For example, the following creates a forth environment so we can take advantage of our new branch-if primitive, and defines two words: double and if-then-double.

* (go-forth (setq my-forth (new-forth))
    { 2 * } 'double name
    { branch-if double "Not doubling" print }
        'if-then-double name)

NIL

Double simply multiplies the top element of the parameter stack by two, doubling it. If-then-double requires two items on the parameter stack. The top element is consumed and the second from top element being doubled only if the top element was non-null. Notice that because the next value in the thread after branch-if is a pointer to another thread (double), control of execution is transfered to this other thread without pushing a resume location onto the return stack. In lisp this is called a tail-call. So if we pass nil20 We need to quote nil because we don't want it to be looked up in the forth dictionary. to if-then-double then the branch is taken, no doubling happens, and the string is printed:

* (go-forth my-forth
    4 'nil if-then-double print)

"Not doubling"
4
NIL

But if the value is non-null, the branch is not taken, the doubling does happen, and the string is not printed:

* (go-forth my-forth
    4 't if-then-double print)

8
NIL

EXIT

(forth-stdlib-add
  { r> drop } 'exit name)

There is an easier way to exit from a word though, and it is implemented with a new word called exit. An interesting property of forth is that the word being called can decide whether or not it was a tail-call. Exit is a normal forth word and so is invoked as usual: forth pushes the current thread location onto the return stack and then sets the program counter to point to the start of the exit word. When exit is invoked, because it has direct access to the return stack with the primitives r> and >r, we can cause the calling word to never get back control of execution by simply removing the resume location from the return stack and dropping it out of existence. Here is an example use of exit:

* (go-forth my-forth
    { "hello" print
      exit
      ;; Never gets here
      "world" print } 'exit-test name

    exit-test)

"hello"
NIL

COMPILER-PRIMS

(def-forth-naked-prim compile nil
  (setf (forth-word-thread dict)
        (nconc (forth-word-thread dict)
               (list (cadr pc))))
  (setf pc (cddr pc)))

(def-forth-prim here nil
  (push (last (forth-word-thread dict))
        pstack))

So branch-if implements a jump, or a goto instruction, possibly jumping to the value stored in the subsequent cell of the thread we are currently executing. Taking values from the thread you are currently executing is a common pattern in forth and requires naked primitives. Another primitive, compile, also uses this pattern. Compile is a naked primitive that will take the value of the next cell in the thread currently executing and then compile this value into the thread of the last word added to the dictionary—usually the word we are currently compiling. Here is a simple primitive that pushes the last cons cell of the thread being compiled onto the parameter stack. Our here is slightly different from the regular forth word here. In forth, here normally pushes the next location that will be compiled to instead of the last location compiled. This is because, in traditional forth, the memory location to be compiled next is known at this time—it will be the next adjacent memory cell. With cons threaded code we can't know this because we have not yet consed that memory.

With compile and here available we can now begin to write forth macros. Remember that when a forth word is immediate, at compile-time it will be executed instead of being compiled into the thread of the last word defined. In similar ways to how we can write macros to adapt and extend lisp, we can use immediate words to adapt and extend forth. In lisp, the basic data structures used for meta-programming are lists. In forth, the basic data structures are stacks.

You might have noticed that our forth environment doesn't even provide an if statement. We have a conditional branching primitive called branch-if, but so far this has only been useful for making tail-calls to other forth words. Recall that forth words are represented by threads and we can put a value for any thread into the cell jumped to by the branch-if. What if we put a value that leads to a part of the thread currently being compiled? We would have, in a sense, a tail-call to another part of our current forth word. Well, if statement is exactly that—a conditional tail call to the end of the if statement that is taken only when the condition is null.

IF

(forth-stdlib-add
  { compile not
    compile branch-if
    compile nop
    here } 'if name immediate)

Because we are programming completely in forth now, there is no need to add new primitives. To add an if statement to forth, we merely append some forth code to our standard library with the forth-stdlib-add macro. Notice that if is defined as an immediate word, meaning that it should only be used when compiling. But since it is immediate, it will be executed, not compiled. When immediate words are encountered, nothing is automatically compiled into the target thread. So if itself compiles in three words to the target thread: not, branch-if, and nop. It then executes the word here which leaves the address of the last compiled word (the nop) on the stack. It leaves it on the stack? That is a strange thing for a word to do at compile time. What stack does it leave it on? Technically, the stack in use at compile time is called the control stack. In most forths the control stack is one and the same as the parameter stack. The distinction is necessary because of the variety of ways that forth can be implemented. Sometimes, especially in cross-compilation environments, the control stack is completely separate from what will eventually be the parameter stack. But here—as with most interactive forth environments—we use the parameter stack for the control stack.

So, if pushes a value corresponding to a location where a nop was compiled. What use is this? The nop itself is not very important, but instead what precedes it. In the cell immediately before the nop there is a branch-if instruction compiled in. Whatever we change the value of the nop to be will be the location our inner interpreter branches to if the if condition turns out to be null.

THEN

(forth-stdlib-add
  { compile nop
    here swap ! } 'then name immediate)

But why did we put a nop there instead of the memory address? It's because we don't yet know the memory address. We need to wait for the programmer to execute another immediate word—then—which will consume the value left on the control stack by if. Then will compile in a nop itself and write the location of this nop over the nop compiled in by if. So all the words between the if and the then will be skipped over if the condition is null.

ABS

(forth-stdlib-add
  { 0 swap - } 'negate name
  { dup 0 < if negate then } 'abs name)

Abs is a word that makes use of if and then to calculate the absolute value of the top item on a stack. It simply checks if the value is below 0 and, if so, it calls another word, negate, to convert the negative value into its absolute value.

The most important reason why the control stack is used in this compilation process is because by using a stack it is possible to have control structures like if statements nest. That is, we can include if statements inside other if statements as long as we make sure to balance all if words with corresponding thens.

ELSE

(forth-stdlib-add
  { compile 't
    compile branch-if
    compile nop
    here swap
    compile nop
    here swap ! } 'else name immediate)

Because forth is a non-Flub language, how such threads are created and threaded together with control structures like if statements is transparently specified and open for us to adapt and extend. Most languages have an else clause associated with if statements; maybe we should add one too. Another immediate word else is added to the standard library. Else compiles into an unconditional branch to the terminating then so that if we took the true (secondary or consequent) branch, we will skip the false (tertiary or alternant) branch. Else then makes use of the value left on the stack by if to replace this nop with the location of the start of the else clause. It then leaves the location of its own nop on the stack for then to make use of. Because the behaviour we want then to perform is the same whether the location on the control stack was left by if or by else, then still works even if there is no else clause.

MOD2

(forth-stdlib-add
  { evenp if 0 else 1 then } 'mod2 name)

The word mod2 uses if, else, and then to reduce an integer to its natural residue under a modulus of 2. It pushes a 0 if the top of stack is even or a 1 if it is odd.

BEGIN-AGAIN

(forth-stdlib-add
  { compile nop
    here } 'begin name immediate
  { compile 't
    compile branch-if
    compile nop
    here ! } 'again name immediate)

Because our conditionals perform tail-calls to other parts of the thread being compiled, there is no reason why we can't use the exact same techniques to create iteration constructs like loops. The most basic forth loop is defined by the begin and again immediate words. These two words provide a simple infinite loop and are implemented very similarly to if and then, except that the address saved on the control stack between seeing these two words corresponds to an address that should be compiled into a branch statement—not a location to compile an address. Here is a simple loop that counts down to 1 from a number provided on the stack and then exits from the word:

* (go-forth my-forth
    { begin
        dup 1 < if drop exit then
        dup print
        1 -
      again } 'countdown name

    5 countdown)

5
4
3
2
1
NIL

Notice in the above example that an if and then construct is nested inside the begin-again loop. Thanks to forth's control stack it is perfectly acceptable to nest any control structures that respect the stack. To respect the stack, a control structure should avoid popping values it didn't push and should avoid leaving any extra values when finished. But just like we often choose to violate referential transparency when constructing lisp macros, in forth we often choose to not respect the stack when compiling. The following example is identical to the previous, except we don't use the word exit to escape from the loop. Instead, we flip into compile mode with the [ and ] words and swap the pointers placed there by if and begin so that we can use their matching then and again words out of order:

* (go-forth my-forth
    { begin
      dup 1 >= if
                 dup print
                 1 -
                 [ swap ] again
               then
      drop
    } 'countdown-for-teh-hax0rz name

    5 countdown-for-teh-hax0rz)

5
4
3
2
1
NIL

In the above, the code compiled by again, which is the code to bring us back to begin, is only executed inside the if statement. Very few other languages allow you access to the compiler in this way—only non-Flub languages to be precise. Because of this freedom, forth programmers are sometimes even more accustomed to macro combinations than are lisp programmers. Although the lisp code in this book uses macro combination techniques regularly, these techniques, and the leverage they can enable, aren't exploited to nearly their full possible extent by most existing lisp code. However, as this book has tried to illustrate, lisp is exceptionally well-suited to macro combinations. Such combination techniques are where I think some of the biggest wins in programmer productivity will be found in the next decade or so of language research.

Going Lisp

So far this chapter has defined a minimalist forth environment and demonstrated some of the most important forth meta-programming concepts from a lispy perspective. Hopefully it has shown just how little effort it can take to design and implement a forth when you have the right tool (COMMON LISP). We can write forth programs to write forth programs—but we already knew that. That's what forth is all about. Further, because of lisp's macro system we can write lisp programs to write forth programs. But can we write forth programs to write lisp programs?

PRINT-FORTH-THREAD

(defun get-forth-thread (forth word)
  (with-pandoric (dict) forth
    (forth-word-thread
      (forth-lookup word dict))))

(defun print-forth-thread (forth word)
  (let ((*print-circle* t))
    (print (get-forth-thread forth word))
    t))

Recall that our forth threads are cons cells linked together and that the leaves of these trees are either functions (representing primitives) or atoms (representing data to push onto the parameter stack). Because we decided to make the forth abstract registers accessible through pandoric macros, writing utilities to acquire and print forth threads is easy. Get-forth-thread pandorically opens the forth closure forth passed to it then retrieves and returns the thread of the word given in word. Print-forth-thread prints this resulting thread with *print-circle* bound to t in case it contains cycles.

To demonstrate, suppose we define two forth words square and square3:

* (go-forth my-forth
    { dup * } 'square name
    { 3 square print } 'square3 name)

NIL

In the compiled forth thread, all the symbols and other word information has been removed. All we have is a piece of list structure extracted from the dictionary of the forth my-forth:

* (print-forth-thread my-forth 'square3)

(3
 (#<Interpreted Function>
  #<Interpreted Function>)
 #<Interpreted Function>)
T

The above code has no cycles and is thus a pure-Flub program. As said earlier, almost all interesting programs contain cycles. To create conditionals and loops we can use the naked forth primitive branch-if which allows us to change the pc abstract register to point somewhere indicated by the value in the subsequent cell in the forth thread being executed. We also were able to implement tail calls by directly accessing the return stack with >r and r>. Unlike most other languages, we can directly customise which calls are tail calls—even from inside the word being called.

But it seems we're missing one construct of central importance to lisp: recursion. Can a word call itself? We saw how we can use branch-if to jump back to the start of a word—tail recursion. What we would really like to do, however, is have a word call itself through the usual thread mechanism. To do this, it must have the start of its thread location stored as a cell in our thread so the current location is stored on the return stack and then it must set pc to the start of the word. So far none of our words has been able to make use of full recursion, however, because we don't name the word until we are done compiling it—it is inaccessible to us when we search the dictionary trying to compile a recursive call. Luckily, there is a simple trick we can use to get around this. We can simply flip out of compile mode and name the word we are compiling before we compile a recursive call. Here is an example definition of a fully recursive factorial calculator:

* (go-forth (setq my-forth (new-forth))
    { [ 'fact name ]
      dup 1 -
      dup 1 > if fact then
      * })

NIL

Sure enough, (fact 5) is 120:

* (go-forth my-forth
    5 fact print)

120
NIL

Exercise: Some forths use a word recurse which simply looks up the thread of the word currently being compiled and inserts it into the thread being compiled. This is called anonymous recursion. Write an immediate word that does this as an alternative to the above trick for implementing named recursion.

Fact's thread is more complicated than square3 above. It contains self-referential code:

* (print-forth-thread my-forth 'fact)

#1=(#2=#<Interpreted Function>
    1 #<Interpreted Function> #2# 1
    #<Interpreted Function>
    #<Interpreted Function>
    #<Interpreted Function>
    #4=(#<Interpreted Function>
        #<Interpreted Function>)
    #1# . #4#)
T

In the above, #2# refers to the dup primitive and was compiled twice. #1# refers to the fact thread itself, implementing the recursion.

These structures look a lot like the lisp list structure that we use to write lisp programs, don't they? Because we understand the abstract machine that will execute these threads, we can, with a few restrictions explained shortly, compile these forth threads back into lisp list structure that can be inserted into an expression by a macro and compiled with our lisp compiler. This process is known as flubifying the code because we convert the compiled program from a uniform, programmable data structure (a thread) into an opaque, inaccessible block of code (a compiled COMMON LISP function).

There are, of course, major differences between forth threads and lisp list structure we can evaluate or insert into macros. First, the forth primitives are simple pointers to functions (displayed here as #<Interpreted Function>), but we need the lisp list structure that these functions were created with. Now is finally the time to explain the dtable abstract register we created. Dtable is a hash-table that gives a mapping from these functions to the list structure that created them, populated when the forth is created.

A large difference between forth threads and our lisp programs is that forth threads assume they can make use of a return stack—a concept that doesn't really exist in Flubs like COMMON LISP. We want to remove the need for our inner-interpreter code and, instead, let the lisp compiler handle this with regular lisp control structures like function calls and tagbody/go forms.

The remaining code in this chapter is presented differently than most of the rest of the code in this book in that its implementation is not described in detail, but rather from a high-level perspective. This is because the mechanics of the implementation are complicated and messy and, honestly, not that interesting. Suffice it to say that I suspect most lisp programmers would implement it similarly.

FLUBIFY-AUX

(defmacro flubify-aux ()
  `(alambda (c)
     (if c
       (cond
         ((gethash (car c) prim-ht)
           (assemble-flub
             `(funcall
                ,(gethash (car c) prim-ht))
             (self (cdr c))))
         ((gethash (car c) thread-ht)
           (assemble-flub
             `(funcall #',(car (gethash (car c)
                                  thread-ht)))
             (self (cdr c))))
         ((eq (car c) branch-if)
           (assemble-flub
             `(if (pop pstack)
                (go ,(gethash (cadr c) go-ht)))
             (self (cddr c))))
         ((consp (car c))
           (flubify forth (car c) prim-ht
                    thread-ht branch-if)
           (self c))
         (t
           (assemble-flub
             `(push ',(car

Cracking Windows Open: Porting RADV to WIN32

RADV is the open source Mesa Vulkan driver for AMD GPUs. Over the years, it has become a cornerstone of the Linux graphics stack. It has now effectively become the de facto Vulkan driver for AMD hardware on Linux. AMD even discontinued their PAL-based (Platform Abstraction Library) alternative in its favor, consolidating their open-source efforts around Mesa.

On Windows however, AMD users are still served only by the proprietary driver. But don't panic, we have a solution: porting RADV to Windows, bringing the same open source Vulkan implementation that has proven itself on Linux to them. It's such an improbable proposition that we might actually get there faster than light (spoiler alert: that's going to be a little slower).

An open source Vulkan driver on Windows opens up a number of possibilities: a shared codebase across platforms, easier debugging and experimentation, faster turnaround on fixes, and a path for the community (e.g., game developers) to report problems or contribute improvements that benefit everyone regardless of operating system.

From first triangle...

This effort builds directly on the groundwork laid by Faith Ekstrand, who first explored the feasibility of running RADV on Windows and presented her findings at XDC 2024. We strongly suggest you go watch the talk, but here is a quick summary.

A central theme of her talk is that since Windows 10, the WDDM2 interface provides a much better foundation for a third-party driver than what came before. It defines a clear model for how a user-mode driver (UMD) interacts with the operating system and the kernel-mode driver (KMD).

However, there is of course a significant catch: many D3DKMT calls can carry private driver data - opaque, vendor-specific blobs whose contents are entirely up to the driver. This means the UMD and KMD remain tightly coupled and that this interface is totally undocumented.

To work around this, Faith created the wddm2-pdd-re tool to log the WDDM2 calls and the private data content for some D3D12 applications. Using this approach, she was able to reverse engineer enough of the private interface — querying adapter info, allocating buffers, creating queues, and submitting commands — to get RADV to submit work to the proprietary kernel driver. And ultimately she was even able to get a rotating 3D model displayed on her screen in all its glory.

...to first game

Where Faith's work established that this could be done, this project picked up the thread. The goal was to improve upon it to make it more flexible (support for different hardware by reducing hard-coded values), more portable (breaking free of WSL with native Windows support), and far more stable (no longer crashing after two minutes of deqp-vk testing).

We improved command stream handling and synchronization and added support for, among others, sparse bindings, tessellation and task shaders, and dynamic querying of GPU properties. It is still not conformant (even though the success rate has dramatically increased), but the real highlight is being able to run our first game with RADV: Counter-Strike 2. Anyone can try it by switching the renderer with the -vulkan argument.

...through darkness

Like any port and/or reverse-engineering project, it brought some challenges:

• New generation is so oversensitive: We were experimenting on Gen 11 hardware (RX 7900 XT) while Faith was working with a 10th Gen GPU (RX 7800 XT). We weren't able to replicate Faith's results for quite a while, due to architecture changes between these two generations. The main problem was that it would hang as soon as we tried to do anything more complicated than clearing a surface to a hard-coded color. Unfortunately, with no tools to debug such hangs on Windows, this forced us to improve the tooling to reduce discrepancies with the proprietary driver as much as possible. We upgraded the reverse-engineering utility into a full WDDM2 logging layer, able to analyze any application run with the official Vulkan driver, and added support to dump the command streams, registers, and shader code.
• Compilers are not playing nice: Mesa is developed primarily against GCC and Clang, and its code base relies on a number of assumptions that don't stand when compiling with MSVC. For instance, MSVC doesn't agree with the other compilers on how to handle enums: it can interpret a value as signed and limit them to 32-bit values, which can result in quite surprising behaviors.
• I've heard you liked opaque blobs: So let's put opaque data inside your opaque API call - D3DKMTEscape. WDDM2 has the possibility of vendor-specific hooks through that method, where the entire content is completely up to the vendor. There is no defined structure, and the only limitation is that it shouldn't replicate something for an existing "standardized" call. Fortunately for us, it looks like we can safely ignore those for now, as they seem to focus on advanced features such as multi-GPU rendering.

...and beyond

The biggest open question for making this production-ready is the interface to the proprietary kernel driver. Developing our own KMD is not really an option, so we need to communicate with AMD's. Right now, doing so relies on reverse-engineered knowledge of private data structures — which is inherently fragile. To make matters worse, the UMD and KMD ship together as a matched pair with no backward-compatibility guarantee, so these structures can change between driver versions with no notice.

For RADV to become genuinely stable and maintainable on Windows, we need one of:

• A stable, documented interface to the proprietary kernel-mode driver, or
• A shim library that mediates communication with the KMD through the private data channel, giving us a stable surface to build against even as the underlying blobs evolve.

There is also significant work remaining on presentation. While Jesse Natalie did some work on WSI for Windows (in the context of the Dozen driver), RADV currently only supports the slower CPU path for presentation. More work is needed to enable the use of DXGI swapchains to improve performance, but that requires being able to import an image from D3D12 (more opaque metadata, hurrah!). Even more work would be required to achieve zero-copy swaps, the pinnacle of graphics pipelines. It could potentially lead to a 3x performance gain for applications that are not GPU-bound. Accomplishing that last step would also require direct involvement from AMD and probably Microsoft (as there are some limitations related to image sharing). There are precedents to that kind of quality-of-life improvements for open-source projects, so there is hope!

We hope this work demonstrates the value of an open Vulkan driver on Windows and helps motivate everybody involved to work toward that stable foundation. For anyone interested, the work is currently sitting in this branch.

Thanks also to Valve for sponsoring this phase of our work to bring open drivers to Windows.

User Interfaces of the Demo Scene

Exploring the tools of the trade

Ahh, the demo scene - a digital art subculture, a motley gang of creative nerds, and a favourite pastime of mine. So much amazing art, music and code has been produced by sceners, and anyone who comes into contact with the scene might start to wonder exactly how - apart from hours of dedicated grind, of course.

The scene has a long and storied tradition of building its own tools. Sometimes from scratch, sometimes by stealing ideas or even code from existing offerings. In a combination of teenage inexperience, old habits, a penchant for experimenting and a desire to make things look cool, this has resulted in some rather peculiar user interfaces.

A few of them are presented below. Most of them are for the Amiga, but several other platforms are represented as well.

---

Elite Sinus Producer

Not wanting to be accused of clickbaiting, let's start off with one of the main attractions: Elite Sinus Producer (sinus here is to be understood as sine), made by Ipec Elite for the Amiga. Demos are usually described as "real time", which is true in the sense that they're (almost) never just animation players, and that demo effects are produced, frame by frame, from code. However, to achieve seemingly impossible technical feats, extensive "cheating" is employed. The most common cheat is probably the so called precalc, meaning that instead of doing complex maths on a 7 MHz (or even slower) CPU, lookup tables are utilized. A plethora of tools for creating such lookup tables exist. This is one of them.

When first starting Elite Sinus Producer, the user is met with this menu. Upon pressing an F-key to make a selection, the corresponding menu option is highlighted and a sample of a cuckoo clock is played. Loudly.

---

Here I've selected the "Flower" option. Pretty, no? This can then be saved to disk as (presumably) assembly source code. Any sprite would look nice when moving along this path!

---

There's also a handy help screen, which is so outlandish I had to grab a short film clip of it: Part of the background consists of moving blue raster bars, and the other part flashes between red and cyan. This obviously helps immensely when reading the text, displayed in a font designed with nothing but legibility in mind.

---

Text Based Interfaces

Plenty of scene related tools are either command line utilities or predominantly text based. First out, we have the assemblers. There was a wide variety of assemblers for the Amiga, but the scene always favoured Seka, Asm-One and other derivatives of the same concept. There were so many different versions and hacks (Trash'm-one, for example) that the sprawling family tree rivals that of Unix systems.

Here's Seka 2.0, and as we can see, it's based on a commercial assembler. This type of assembler always asks the user for the size of the working memory to be allocated. They then enter a command line mode, which can be used to examine RAM memory and CPU registers, and load source files into the editor proper.

---

Here's AsmOne in one of its many incarnations. It's quite similar to Seka (in fact, it's "Seka-Updated"), but has more built-in commands and presumably other improvements as well (I've never been much of a coder). Interface-wise, it opens its own screen, as opposed to running in a window on the default Workbench screen.

---

What if you found a piece of cool music or graphics in, say, a game? What if you wanted to steal some samples, or a sprite, or perhaps just save an entire tune for easy listening? Then you'd need a ripper - a tool for hunting through your computer's memory, looking for remnants of such data after quitting the game. There were tons of various such rippers for the Amiga. Here's Multi-Ripper, which has a fairly representative set of features.

---

Here's another type of ripper, specifically written to look for Seka assembly sources in memory after the computer had crashed. The Amiga, like most other home computers, had no memory protection, and demo coding is a notoriously crash-prone activity. Saving often was common practice, but even seasoned coders sometimes messed up and forgot. With a bit of luck, the code could be extracted from memory after a warm reboot.

---

Here's another type of sine precalculator, called The Sinus Creator. I have no idea if the numbers I've entered in the screenshot make sense, but the two-window text interface is interesting. Of course, the result can be saved as a Seka source file.

---

Music Trackers

Demo music has historically been made in trackers. Far from traditional notation, a tracker lets the musician enter tones along with various modifiers and effects in something that's more reminiscent of a programming editor rather than regular composing. They also let the user manage instruments, whether sampled or synthesized.

Sample-based trackers on the Amiga have an even more sprawling family tree than that of Amiga assemblers, but they all originate from Karsten Obarski's commercial Ultimate Soundtracker from 1987. Being commercial and thus costing money, it was soon picked apart by sceners, which resulted in NoiseTracker, which was then revamped into ProTracker, which in turn exists in so many various versions and re-hashed hacks that it's nigh impossible to keep track (heh) of. The sprawl is even sprawlier than that of Amiga assemblers!

SoundMonitor 1.0 for the Commodore 64 (by Chris Huelsbeck) isn't strictly speaking a scene release, and wasn't called "tracker". However, the interface (with one "track" per available sound channel) is what inspired the previously mentioned Ultimate Soundtracker, and is thus included here for posterity and correctness.

---

NoiseTracker by the Swedish duo Mahoney and Kaktus wasn't the first, but it was immensely popular and came to define the tracker experience for years to come. Its legacy lives on just not in Protracker on the Amiga, but on several other platforms as well.

---

Here's Protracker's file picker. Just like in NoiseTracker above, it's accessed by clicking the "Disk Op." button in the rather dense interface. It's hard to describe what a strange experience this UI delivers, because it's almost - but not quite - intuitive to someone used to more mainstream Amiga programs. It's easy to misclick, misunderstand or just plain miss things. To illustrate its idiosyncratic design, take note of the vertical EXIT button, which quits back to the main menu. It's conveniently placed between the up and down arrows used for scrolling the purple file and directory listing.

---

Here's Digicomposer 1.0 for the Atari ST/e, which in turn builds on Noisetracker for the Atari. The interface is clearly more than just inspired by its Amiga counterpart. It's but one of a whole menagerie of trackers for the Atari ST, some "digi" (sample-based) like this one, some for YM-chip based music.

---

Fasttracker II for MS-DOS is iconic in its own right. With support for Gravis UltraSound and other advanced PC sound hardware, it has features for 16-bit samples, bizarre amounts of sound channels, and even lets the user play a simple version of Snake if they so desire.

---

Abyss' Highest Experience is a chiptune tracker for the Amiga, intended to sound like the Commodore 64's SID chip. The user interface is an interesting crossover between Soundtracker and a more modern, Workbench 2.0-like look.

---

I don't know much about Megatizer for the Atari ST, but it sure does look cool!

---

JamCrackerPro for the Amiga eschewed the traditional tracker UI and opted for a system-friendly, multi-window interface.

---

Disk Copiers

In order to distribute demos (and pirated software), disk copying was a common activity on the scene. Commodore provided a disk copier in AmigaOS, but it wasn't always up to the task of copying demos and games that bypassed the file system by writing directly to the tracks of a floppy. Thus, special software was needed!

Like SoundMonitor, X-Copy isn't strictly a scene release. Although originally written by sceners, it was released commercially and then heavily pirated on the scene. The prominently featured grids contain one square for each track on the disk, displaying the status for copying that particular part of the floppy. When a copy was finished (which could take some time), the program helpfully played a little "boing" sound.

---

X-Copy was released in many versions during the Amiga heydays. Here's X-Copy 3.0, apparently in an illicit variety distributed by the cracking group Paradox.

---

Personally, I preferred D-Copy, mostly because I thought the user interface looked cool (and I still do). It is, as far as I know, a bona fide non-commercial scene product.

---

Other Tools

A short article like this can only ever scratch the surface of the numerous demo scene tools ever created. Here are but a few more various tools and interfaces, selected for being interesting and/or representative of their kind.

This is Titanics Cruncher for the Amiga. A cruncher uses asymmetrical compression of executable files. This saves space on disk, with the tradeoff being longer loading times, due to the decrunching (decompression) of the executable upon running it. Several other crunchers exist, and on a variety of platforms.

---

Before the Internet, there was the Bulletin Board System, or BBS. Sceners were usually interested in Elite BBS:es, meaning ones that offered pirated software for download. There were few, if any, Elite BBS:es without cool ANSI (or PETSCII, on the C64) graphics, for example in animated menu screens. Thus, many various ANSI editors popped up for different platforms. This is Digital Intelligence's Ansi-Editor v2.4 for the Amiga, and it has a very peculiar user interface. The toolbar at the bottom displays the currently active colours, but you can't select them from there. Click all you want, to no effect - you have to use the pull-down menu to actually pick one.

---

Before Twitter, Facebook, IRC and even widespread access to modems and BBS:es, scroll texts were the communication medium of choice for the scene. Cool scrollers required cool fonts, and it could be helpful with a dedicated font (or charset) editor. Here's Charedit by Escape, for MS-DOS.

---

Home computers often came with custom or otherwise esoteric hardware. The Atari Falcon, for example, sported a Motorola 56001 digital signal processor. DSPdit by tSCc (short for The Sirius Cybernetics Corporation) is an editor and assembler made specifically for DSP56k programming. It uses the standard GEM toolkit, which gives it a very clean and professional look.

---

The first computer virus for the Amiga was a bootblock virus, infecting the boot sector of floppy disks. As such, it could potentially ruin games and other floppies with custom bootblocks. It was constructed by the Swiss Cracking Association, SCA. The virus spread like wildfire and perhaps SCA was plagued by a guilty conscience, because later they also produced the very first virus killer for the Amiga - designed to counter their own virus. The mega-mighty interface is easy enough to grasp!

---

As mentioned above, plenty of Amiga demos were so called trackmos, meaning they didn't use the file system, but rather stored data straight on the tracks of a floppy disk. This soon resulted in various tooling becoming available for creating such trackmos. Mostly it was just code shared between sceners, but there were also complete software suites, like TrackmoDOS by Poison of NOVA. Among other things, it came with this orthodox file manager for writing and deleting files to a trackmo floppy. Being a scener tool, the GUI of course sports some rather spiffy colour gradients.

---

This is RAW, which isn't really a tool, but rather a disk magazine, or diskmag. A diskmag is just that - a periodical published on one or more floppy disks. Most of the articles were usually scene related, although some mags branched out and featured everything from short stories and poems to essays about history and politics. RAW was one of the most popular Amiga diskmags during the early 1990s, and as we can see, the UI was shiny and textured long before Frutiger Aero was a thing. It even came with a built-in palette editor (pictured), allowing the user to customize the text colours.

---

This is FuckPaint, a pixel painter for the Atari Falcon. It is included here solely on the merit of its name, and its equally classy "Analizer" tool.

---

Although not at all a scene production, Deluxe Paint must be mentioned here. To the best of my knowledge, the scene never produced a pixel painter for the Amiga (except later ports of Grafx2), presumably because nobody saw the need for one. I don't think a single Amiga user existed that didn't have a copy of this software, either bought separately, bundled with the machine or just pirated. It was also completely dominant on the PC. There were other Amiga pixel painters, such as Brilliance and Personal Paint, but their combined market share was a mere fraction of this giant, which also dominated graphics creation for games well into the the mid-1990s.

---

That's enough demo scene interfaces for one helping. For those still wanting more, I recommend this gallery of utildisk menus.

Take care and happy hacking!

If you managed to put Tailscale on a jailbroken Kindle before it updated too far ahead, you got something pretty great, even if it wasn't the full Tailscale experience. But good things come to those who wait (or dig around on GitHub).

Open-source developers have improved the Tailscale experience on one of the weakest computers you own. If your Kindle is jailbroken, an updated version of Mitanshu Sukhwani's Tailscale implementation offers a few new things:

• Tailscale SSH enabled by default, so you don't have to enable USBnetworking SSH and its very obvious default user/password
• A proxy mode that lets apps like KOReader to reach other nodes on your tailnet, like a Calibre/OPDS or Wallabag server
• A full TUN mode that, on some Kindles, can make Tailscale networking work at the device level

Let’s dig into each one and how to set them up. As before, this is community code working on a very unofficial device state; bring your patience along.

Tailscale on a Kindle, now with proxies

The last time we wrote about Tailscale on a Kindle, the client was basic, but it worked. The Kindle showed up on your tailnet, complete with a green dot in the web admin console. You could reach the Kindle by its Tailscale IP address. You could even SSH into the Kindle over Tailscale, which was handy for further tinkering.

But “reachable via Tailscale” is not the same as “routing all incoming and outgoing traffic across your tailnet,” it turns out. Tailscale on a jailbroken Kindle is typically forced to run in userspace mode, which means it cannot use the device's own network routing layer, known as TUN mode. You could start Tailscale, and then start an app like KOReader, but when you tried to connect to another Tailscale device, like your Calibre server at 100.x.y.z, it would go like this:

• KOReader (or any app) asks the Kindle’s root OS how to reach 100.x.y.z
• The Kindle, lacking Tailscale routing, cannot reach that Tailscale IP address
• KOReader drops the connection

An update to the Kindle KUAL app by greywolf1499 provides different modes that work around this. Now, when you try to reach another Tailscale IP address on your Kindle, it can go like this:

• You start Tailscale in proxy mode
• You set up KOReader or another app's proxy settings to connect to 127.0.0.1:1055
• KOReader tells the proxy it wants to reach 100.x.y.z
• Tailscale’s daemon tailscaled, listening on port 1055, routes the connection through Tailscale
• E-books, articles, and other data flows between your Tailscale-running Kindle and other Tailscale devices

This Tailscale proxy offers two modes, SOCKS5 and HTTP CONNECT, for apps that may prefer one or the other. This opens up a good bit more utility for your more-connected Kindle.

What you can do with a proxied Kindle

A few wild ideas, depending on how dug in you want to get:

• Calibre or Wallabag servers, as mentioned
• Audiobookshelf connection through KOReader
• Use Readest to track reading progress across devices
• Linking KOReader’s RSS reader to a a self-hosted feed server
• Accessing minimalist dashboards and web pages in the (pretty bad) Kindle browser
• Using a Bluetooth keyboard and the kterm app to SSH into tailnet devices

Is that last one all that practical? Not really. But is there a pleasant warmth, knowing that you've added the least likely thin client to your what-if kit? For some types, types I know quite well: yes.

The Tailscale plugin for KOReader (with Kobo and PocketBook support)

If you don’t really need any Tailscale powers outside the highly capable KOReader app, check out this Tailscale KOReader plugin. It doesn’t make your Kindle accessible over your tailnet, like the KUAL-based app. But it does automatically create the proxy interfaces that are needed for reaching your content servers from your KOReader-running Kindle—or your Kobo device, or your PocketBook.

I haven't been able to really try this extension out myself; my 11th-generation standard Kindle doesn't play well with it at the moment. It's been "Tested on Kindle PW5/PW6, Kobo, and PocketBook"—it's nice to see Tailscale come to some other KOReader-friendly devices, too.

Installation is not too hard, at least if you made it this far into jailbreaking already. You copy the plugin into KOReader’s plugins directory, trigger an “Install/Update Tailscale” from KOReader’s menu, copy a Tailscale key into a directory, then toggle Tailscale on in the KOReader menu. From there, you configure KOReader with one of its proxy addresses (127.0.0.1:1055 for SOCKS5, :1056 for HTTP CONNECT), then give other plugins the Tailscale IP addresses you need to reach.

Victoria Riley Barnett’s repository notes that the plugin works great with a SyncThing plugin for KOReader. KOReader is like its own separate OS for jailbroken Kindles at this point,

---

So now you’ve got a lot more options and weird projects available to you, through this already quite-strange little slab. If you’ve worked up a weirdly useful Tailscale setup on your Kindle, Kobo, or other e-paper device, we’d love to hear about it. Let us know on Reddit, Discord, Bluesky, X, Mastodon, or LinkedIn.

Every few months, and especially this week, the AI space gets criticized for circular financing and customer relationships. OpenAI raises money from Microsoft, spending it on Microsoft servers; Nvidia backstops CoreWeave debt, and CoreWeave buys Nvidia GPUs. “The bubble is about to burst!” analysts scream.

Analysts point to dot-com deals with circular investments in 1999, arguing that this is all bound to happen again[1][2]. They are wrong.

Circular deals are more interesting than “good” or “bad.” They show the AI industry isn't just maturing, but modifying the idea of “AI” to something that is less a technology product, and more a commodity. Imagine that—intelligence available like electricity, and the underlying financial system structured accordingly.

The evolution of circular deals points to the commodification of AI, where compute capacity is moving from a business model where you buy a product (e.g., the hardware, or space in a data center) to something so fungible that you buy it the way you buy electricity, copper, natural gas, or other commodities.

We'll explore why circular deals are particularly important in commodity industries and what this implies for understanding the trajectory of AI. First we'll cover how major investments in traditional commodities markets work to ground our analogy more clearly. Next, we'll cover the investments that mimic this process in AI, showing how such investments can be healthy. Finally, we'll explore a few examples where circular deals do not abide by this analogy and how these deals are introducing risks that could one day turn into awful surprises for the companies themselves, their investors, or the entire sector.

Buying and Selling Commodities: A Circularity Primer

Complex commodity infrastructure like mines, refineries, and ports comes with such large development costs that a bank lending a development company money can potentially risk its own solvency in doing so. Circular deals with multiple customers or beneficiaries, and potentially even governments, are often the only solution that gets shovels in dirt or ships in the water.

Let's look at a simplified example of such a deal. Imagine you want to develop your region's economy with several wells and a pipeline, and you can't afford it. You partner with a commodity trading firm who agrees to make your company attractive to banks or bond buyers by guaranteeing they will buy all your oil at a certain price. This means you are guaranteed revenue for the foreseeable future, and the banks know they can trust you'll repay their loans. The trading firm might even take an equity stake in your company for good measure—potentially to encourage better governance or oversight.

With such a relationship between you and the oil trader, you've got oil, a large bank loan, and a guaranteed customer… you've got yourself a circular commodities deal!

This is not a contrived example; it's a common strategy developed and evolved since the 1960s. Japanese commodities traders and development banks financed infrastructure to enable commodity development, committing to future purchases and equity deals[3]. Jamaica did so in the 1980s[4].

More recently, the US government began facilitating circular deal making to stimulate the critical minerals sector in its bid for supply chain resilience. Last year, MP Materials, a relative newcomer to magnet manufacturing and critical minerals, announced a 10-year relationship with the Department of War, where the latter committed to buying all neodymium-praseodymium (i.e., magnets) from the company for at least $110/kg[5]. Since then, such off-take agreements have been announced between MP Materials and General Motors[6], and are in fact quite common in the electric vehicle space[7]. These direct relationships also incentivize equity ownership in the commodities producers and even underlying mines because it's very clear that these producers and mines will have revenue in the coming years.

Fungible commodities with large global markets are particularly well suited to such deals because the counterparty guaranteeing to be a customer (i.e., the oil trading firm in our example above) knows there is a large market they can tap into. They likely have a history of successfully making such sales, otherwise they wouldn't have billions of dollars and a pristine reputation they can leverage.

AI is Fungible and Expensive to Develop

The frontier generative AI industry—be it model development or inference—is very much dependent on Nvidia. GPUs are effectively a fungible commodity thanks to Nvidia's development of the underlying infrastructure and standardization across all firms in the space. Three forces are enabling AI chips and associated data centers to act like a fungible commodity:

1. The majority of compute today comes from Nvidia GPUs, so two data centers using similar chips are almost the same for the purposes of hyperscalers, model builders, and those needing inference.
2. Model development and inference at scale requires massive data centers sometimes costing well over $10 billion. These data centers can't be developed unless it's via a creative financing deal or via a company with access to immense wealth (i.e., the hyperscalers).
3. Aside from GPUs, memory and other hardware, electricity is the other bottleneck. You can't build an 11-figure data center without access to large amounts of reliable electricity.

As a result, GPUs, electricity, and data centers are all effectively fungible, with large order backlogs and pent-up demand. If you build a data center and can't leverage it for your own business, there's a good chance you can sell it to someone who can—they might even pay a premium for availability in the short term.

SpaceX and Meta, despite trying to build top-tier foundation models, are profitably leasing their own data centers to others—these data centers are working today and ready for frontier lab workloads. Well-capitalized labs are willing to pay a huge premium for the privilege. SpaceX leases its Colossus 1 and 2 data centers for over $2 billion per month[8], at a significant markup over other smaller clouds and data centers[9].

Let's now return to the aspiring startup or neocloud. You are starting up and, like our oil example earlier, have proven yourself on a small scale but now need your own data center or access to thousands of GPUs. What can you do?

Enter Nvidia and the circular deal—much like the oil deal.

Nvidia provides the capital and product access to your startup, prioritizing your access to its GPUs so you can get the hardware you need. Nvidia has a $1 trillion order backlog[10] and knows it can resell your hardware or make better use of it if you fail, so it goes a step further: it becomes the guaranteed buyer of your compute if you can't take advantage of it… much like SpaceX and Meta above. This is not theoretical—SemiAnalysis provides estimates for Nvidia compute off-take agreements and pricing[11]. As with our oil example, Nvidia might want some equity as well—and you will want them to have it, should you need to call in any favors later.

This is, of course, one type of circular deal, and a relatively simple one at that. It's been used in CoreWeave's $6.3 billion deal with Nvidia[12], alongside smaller data center operators like Firmus ($505 million[13]).

This helps explain why we see so many interconnecting and circular relationships—OpenAI cancels its deal with Oracle, so Meta swoops in[14]; SpaceX leases servers to Google, Google invests in Anthropic; and so on.

Wall Street gets circular deals, and humanity gets artificial general intelligence… purportedly.

When Circular Deals Go Ugly

Nvidia argues it is supporting a global AI ecosystem. Unfortunately, this doesn't preclude it from overextending itself. A $6.3 billion deal with CoreWeave is one thing, but committing up to $750 billion[2] is another.

The success of this approach for startups and neoclouds also assumes that AI and data centers continue to be fungible and “resellable”. Should standards or chipsets change, or should technologies make it easier to run local models or models on small clusters, then the business models might fail and the off-take agreements Nvidia has might not help the ecosystem much.

… and When They Go Bad

FT diagram of TeraWulf's $3.2bn bonds: Anthropic pays Fluidstack for compute, Fluidstack's lease payments flow to a Morgan Stanley lockbox servicing TeraWulf's bondholders, and Google backstops the leases on tenant default

Figure 1: Google's backstopping of TeraWulf bonds, which are enabling the building of Fluidstack data centers leased by Anthropic. [original]

Figure 1 shows the FT's[15] breakdown of a recent TeraWulf bond deal. In this case, TeraWulf can obtain financing given Google's backstop of any lease failures, should Fluidstack not be able to pay TeraWulf, or Anthropic unable to pay Fluidstack[16]. Google is skipping the entire reselling process and simply committing to cover any delinquencies. This gives TeraWulf the benefits of Google acting as a guarantor of the loan without the loan appearing on Google's balance sheet.

Similarly, Meta's $27.3 billion Hyperion data center bond sale is an off-balance sheet one[17], as is its more recent $12.3 billion deal marketed by BlackRock[18].

Many finance professionals argue that these bonds are ultimately guaranteed by the impressive and continually growing revenues and profits from the hyperscalers, so there is nothing to worry about. The reason these bonds find so many customers, despite the circularity label, is that the final guarantors (i.e., the hyperscalers) generate billions of dollars of profit every year and can easily cover these costs, should it come down to that.

… but this is today, and it's with the current bond deals. Will tomorrow's bond deals be supported by the hyperscalers in the same way? And what happens if hyperscaler revenue trends change? What if the banks begin expanding the backstop agreements from hyperscalers to “generally pretty decent” companies? What if the backstop fails to be enforced? Will we see hyperscaler-backstop-backed-bonds grouped together, collateralized, and resold the way Mortgage Backed Securities were in 2007?[19]

Implications and Conclusion

Circular deals are not bad. In fact, they are critical in the development of the AI ecosystem much like such deals are used in critical minerals, oil, electric vehicles, and other capital intensive industries with incredibly high startup costs.

The circularity helps illustrate the commodification of AI today, and how the technology might one day be more like electricity or an internet connection, rather than a product one buys or subscribes to.

When circular deals are supported by overextended lenders, or when overextended lenders try to move such deals off their balance sheets, they become ugly and bad—in other words, incredibly risky. This industry is likely to grow much more in the coming years, so it is important to watch for the lowering of standards or aggregation of risk. As SemiAnalysis writes[11]:

AI Debt Financing will become a multi-trillion-dollar credit market, with over $7T of debt outstanding by 2029 driven both by AI IT Capex and AI Datacenter Capex needs [...] This will make it the second largest asset backed debt market after the US mortgage-backed financing market at just over $13T.

In this evolution, there might be a few bad deals along the way. It's important to keep your eyes open.

References and Footnotes

1. Bloomberg; AI Circular Deals: How Microsoft, OpenAI and Nvidia Keep Paying Each Other
2. Bloomberg; Nvidia’s $750 Billion in Deals Reignite Circular AI Fears
3. Soyeun Kim; From Goa to Angola: Exploring Ideas and Mechanisms in Japan’s Kaihatsu-Yunyū
4. A good overview of commodity trading and development, including Jamaica's aluminum industry and circular deals thereof, is available in The World for Sale—https://www.amazon.com/World-Sale-Javier-Blas/dp/0190078952
5. CSIS; Rare Earth Export Restrictions One Year Later
6. MP Materials; General Motors and MP Materials Enter Long-Term Supply Agreement to Scale Rare Earth Magnet Sourcing and Production in the U.S.
7. McCarthy Tétrault; Car and Battery Makers are Getting Closer to the Mining Business
8. Wikipedia; Colossus (data center)
9. SemiAnalysis; Meta Compute: Everyone Wants To Be A Neocloud
10. CNBC; Nvidia CEO Jensen Huang: At least $1T backlog through 2027
11. SemiAnalysis; Nvidia GPU Debt Backstop Unleashes the AI Project Trinity: Capital, Offtake and Datacenters
12. Reuters; CoreWeave, Nvidia sign $6.3 billion order for cloud computing capacity
13. Bloomberg; Nvidia-Backed Data Center Builder Firmus Raises $505 Million
14. Tom's Hardware; OpenAI’s massive Stargate data center canceled as firm can’t reach terms with Oracle, operator struggles with reliability issues — Meta said to be interested in snatching excess capacity
15. Financial Times; Morgan Stanley cashes in on AI boom with debt deals
16. It is incredibly interesting that Anthropic's deals with IREN, TeraWulf, and other companies enabled Situational Awareness to make billions of dollars via equity investing, and then they went on to fund Fluidstack's $830 million Series A, which is now being used to support Anthropic's compute requirements.
17. Financial Times; A closer look at the record-smashing ‘Hyperion’ corporate bond sale
18. Bloomberg; BlackRock Dodges AI Bond Flop as $12.3 Billion Debt Deal Rallies
19. Hey there, fellow reader, hyperscaler-backstop-backed-bonds grouped, collateralized, and resold could be extremely profitable if you believe AI is the future but aren't sure which companies will succeed in paying off their debts!