This mini-book provides a brief overview of many concurrency topics in Go. Each topic comes with interactive examples — feel free to experiment with them by changing the code and clicking Run. There's also a PDF version with static examples.

This is a quick refresher on Go concurrency, not a beginner's guide. If you want to learn concurrency from the ground up with practical exercises, check out my other book — Gist of Go: Concurrency.

The book is AI-free.

# Goroutines

The foundation of concurrency in Go is goroutines – functions started with the go keyword:

func main() {
    var wg sync.WaitGroup
    wg.Add(2)
    go func() {
        defer wg.Done()
        fmt.Println("worker 1")
    }()
    go func() {
        defer wg.Done()
        fmt.Println("worker 2")
    }()
    wg.Wait()
}

worker 2
worker 1

The Go runtime juggles these goroutines and distributes them among operating system threads running on CPU cores. Compared to OS threads, goroutines are lightweight, so you can create hundreds or thousands of them.

Goroutines are completely independent. The main function is also a goroutine, but it starts implicitly when the program starts. When main ends, other goroutines also shut down.

We use a wait group (sync.WaitGroup) to wait for goroutines to finish in the example above. A wait group has a counter inside. Calling Add(n) increments it by n, while Done() decrements it by one. Wait() blocks the calling goroutine (in this case, main) until the counter reaches zero. This way, main waits for both workers to finish before it exits.

WaitGroup.Go automatically increments the wait group counter, runs a function in a goroutine, and decrements the counter when it's done:

func main() {
    var wg sync.WaitGroup
    wg.Go(func() {
        fmt.Println("worker 1")
    })
    wg.Go(func() {
        fmt.Println("worker 2")
    })
    wg.Wait()
}

worker 2
worker 1

# Channels

Goroutines can pass values to each other through channels. A channel is like a window where one goroutine can throw something and another can catch it:

func main() {
    messages := make(chan string)

    go func() { messages <- "ping" }()

    msg := <-messages
    fmt.Println(msg)
}

ping

Sending a value through a channel is a synchronous operation. When the sending goroutine writes a value to the channel (ch <- val), it blocks and waits for someone to receive that value (<-ch). Only then does it continue.

Output channel

Returning an output channel from a function and filling it within an internal goroutine is a common pattern in Go. This allows the caller to receive values through the channel while the owning function retains control of it:

func generate(start, stop int) chan int {
    out := make(chan int)
    go func() {
        for i := start; i < stop; i++ {
            out <- i
        }
    }()
    return out
}

Closing a channel

To signal readers that all data has been sent, the writer goroutine closes the channel with close():

func generate(start, stop int) chan int {
    out := make(chan int)
    go func() {
        defer close(out)
        for i := start; i < stop; i++ {
            out <- i
        }
    }()
    return out
}

The reader checks the channel's status with a second value ("comma OK") when reading:

func main() {
    in := generate(5, 10)
    for {
        num, ok := <-in
        if !ok {
            break
        }
        fmt.Print(num, " ")
    }
}

5 6 7 8 9

While the channel is open, the reader receives the next value and a true status. If the channel is closed, the reader gets a zero value and a false status.

A channel can only be closed once. Closing it again or writing to a closed channel causes a panic.

The only reason to close a channel is to signal to its readers that all data has been sent. If this isn't important to the readers, then you don't need to close it. When a channel is no longer used, Go's garbage collector will free its resources, whether it's closed or not.

Channel iteration

range automatically reads the next value from the channel and checks if it's closed. If the channel is closed, it exits the loop:

func main() {
    nums := generate(5, 10)
    for n := range nums {
        fmt.Print(n, " ")
    }
}

5 6 7 8 9

Range over a channel returns a single value, not a pair, unlike range over a slice.

Directional channels

You can protect yourself from accidental write/close errors by setting the channel direction. Channels can be:

• chan (bidirectional): for reading and writing (default);
• chan<- (send-only): for writing only;
• <-chan (receive-only): for reading only.

You can't read from a send-only channel or write to a receive-only channel (nor can you close it).

Channels are usually initialized for both reading and writing, and specified as directional in function parameters. Go automatically converts a regular channel to a directional one:

stream := make(chan int)

go func(in chan<- int) {
    in <- 42
}(stream)

func(out <-chan int) {
    fmt.Println(<-out)
}(stream)

42

Buffered channels

Buffered channels work like a FIFO queue with a fixed-size buffer for storing values.

As long as the buffer has free space, writing to the channel doesn't block the goroutine. Similarly, as long as the buffer contains values, reading from the channel doesn't block the goroutine:

stream := make(chan int, 3)
stream <- 11
stream <- 12
stream <- 13

fmt.Println(<-stream)
fmt.Println(<-stream)

11
13

By default, if you don't specify a buffer size, a channel is unbuffered (buffer size equals zero).

Buffered channels work with the built-in len() and cap() functions:

stream := make(chan int, 3)
stream <- 11
fmt.Println(cap(stream), len(stream))

3 1

Reading from a closed buffered channel returns values from the buffer and a true status. Once all values are taken, it returns a zero value and a false status, like a regular channel:

stream := make(chan int, 1)
stream <- 11
close(stream)

val, ok := <-stream
fmt.Println(val, ok)
// 11 true

val, ok = <-stream
fmt.Println(val, ok)
// 0 false

11 true
0 false

nil channel

Like any type in Go, channels have a zero value, which is nil.

Writing to or reading from a nil channel blocks the goroutine indefinitely:

var stream chan int

go func() {
    // blocks forever
    stream <- 1
}()

// blocks forever
<-stream

Closing a nil channel causes a panic:

var stream chan int
close(stream)
// panic: close of nil channel

# Select

The select statement is somewhat like switch, but specifically designed for channels. Here's what it does:

• Checks which cases are not blocked.
• If multiple cases are ready, randomly selects one to execute.
• If all cases are blocked and there is a default case, executes it.
• If all cases are blocked and there is no default case, waits until one is ready.

Select is used to manage data flow in pipelines:

// merge sends values from in1 and in2 to the output channel.
func merge(in1, in2 <-chan int) <-chan int {
    out := make(chan int)
    go func() {
        defer close(out)
        for in1 != nil || in2 != nil {
            select {
            case val1, ok := <-in1:
                if ok { out <- val1 } else { in1 = nil }
            case val2, ok := <-in2:
                if ok { out <- val2 } else { in2 = nil }
            }
        }
    }()
    return out
}

// Suppose we send 10..12 to in1, 20..22 to in2,
// and call merge(in1, in2)

10 11 20 12 21 22

To cancel goroutines:

// process modifies values from in and send them to out
// until in is exhausted or cancel is closed.
func process(cancel chan struct{}, in <-chan int) <-chan int {
    out := make(chan int)
    go func() {
        for val := range in {
            select {
            case out <- val*10:
            case <-cancel:
                fmt.Println("canceled")
                return
            }
        }
    }()
    return out
}

// Suppose we send values 11 and 12 to in
// and then call close(cancel)

110
120
canceled

For non-blocking operations:

// multiplier returns a function that multiplies
// the input by 10 and sends it to the channel
// or returns an error if the channel is busy.
func multiplier(ch chan<- int) func(n int) error {
    return func(n int) error {
        select {
        case ch <- n*10:
            return nil
        default:
            return errors.New("busy")
        }
    }
}

func main() {
    nums := make(chan int, 1)
    multiply := multiplier(nums)

    err := multiply(11)
    fmt.Println(<-nums, err)
    // 110 <nil>

    err = multiply(12)
    fmt.Println(<-nums, err)
    // 120 <nil>

    err = multiply(13)
    err = multiply(14)
    fmt.Println(err)
    // busy
}

110 <nil>
120 <nil>
busy

And for much more.

# Pipelines

A pipeline is a sequence of operations where each step takes input data, processes it in a specific way, and outputs it. The input and output of each operation is a channel.

A typical pipeline looks like this:

• Reader: Reads input data from a file, database, or network.
• N processors: Transform, filter, aggregate, or enrich data using external sources.
• Writer: Writes the processed data to a file, database, or network.

func read[T any]() <-chan T {
    out := make(chan T)
    go func() {
        defer close(out)
        for {
            // read data from somewere
            data := // ...
            out <- data
        }
    }()
    return out
}

func process[T any](in <-chan T) <-chan T {
    out := make(chan T)
    go func() {
        defer close(out)
        for inData := range in {
            // process the data
            outData = // ...
            out <- outData
        }
    }()
    return out
}

func write[T any](in <-chan T) <-chan struct{} {
    done := make(chan struct{})
    go func() {
        defer close(done)
        for data := range in {
            // write the data
        }
    }()
    return done
}

Output channel

A goroutine can signal other goroutines that it has finished its work using an output channel:

func generate(start, stop int) <-chan int {
    out := make(chan int)
    go func() {
        defer close(out)
        for i := start; i < stop; i++ {
            out <- i
        }
    }()
    return out
}

func main() {
    nums := generate(5, 10)
    for n := range nums {
        fmt.Print(n, " ")
    }
}

5 6 7 8 9

Done channel

If a goroutine doesn't need to return results, it can signal completion using a done channel:

func work() <-chan struct{} {
    done := make(chan struct{})
    go func() {
        defer close(done)
        fmt.Println("work done")
    }()
    return done
}

func main() {
    done := work()
    <-done
}

work done

Cancel channel

To terminate a goroutine early, a calling goroutine can use a cancel channel:

func generate(cancel chan struct{}, n int) <-chan int {
    out := make(chan int)
    go func() {
        defer close(out)
        for i := 1; i <= n; i++ {
            select {
            case out <- i:
            case <-cancel:
                return
            }
        }
    }()
    return out
}

func main() {
    cancel := make(chan struct{})
    defer close(cancel)

    nums := generate(cancel, 10)
    fmt.Println(<-nums)
    fmt.Println(<-nums)
    fmt.Println(<-nums)
}

1
2
3

Error handling

There are three approaches to error handling in concurrent pipelines.

➊ Return on the first error:

// calculate produces answers for the given numbers.
func process(in <-chan int) (<-chan int, <-chan error) {
	out := make(chan Answer)
	errc := make(chan error, 1)
	go func() {
		defer close(out)
		for n := range in {
			ans, err := fetchAnswer(n)
			if err != nil {
				errc <- err  // return with error
				return
			}
			out <- ans
		}
		errc <- nil          // return with nil
	}()
	return out, errc
}

➋ Use a result type:

// Result contains an answer or an error.
type Result struct {
	answer int
	err    error
}

// calculate produces answers for the given numbers.
func calculate(in <-chan int) <-chan Result {
	out := make(chan Result)
	go func() {
		defer close(out)
		for n := range in {
			ans, err := fetchAnswer(n)
			out <- Result{ans, err}  // return answer + error
		}
	}()
	return out
}

➌ Collect errors separately:

// calculate produces answers for the given numbers.
func calculate(in <-chan int, errc chan<- error) <-chan int {
	out := make(chan Answer)
	go func() {
		defer close(out)
		for n := range in {
			ans, err := fetchAnswer(n)
			if err == nil {
				out <- ans   // send answer
			} else {
				errc <- err  // or error
			}
		}
	}()
	return out
}

# Time

Besides handling date and time, the time package offers tools for managing time-sensitive operations in concurrent programs.

After

time.After() returns a channel that is initially empty, but receives a value after the timeout period. It's useful for timing out operations:

// withTimeout executes a function with a given timeout.
func withTimeout(timeout time.Duration, fn func()) error {
    done := make(chan struct{})
    go func() {
        defer close(done)
        fn()
    }()

    // blocks until fn completes or the timer expires,
    // whichever happens first
    select {
    case <-done:
        return nil
    case <-time.After(timeout):
        return errors.New("timeout")
    }
}

withTimeout() waits for fn() to complete, but thanks to time.After(), it won't wait longer than the timeout duration:

func main() {
    var err error

    // completes in time
    err = withTimeout(
        50*time.Millisecond,
        func() { fmt.Println("work done") },
    )
    fmt.Println("err =", err)

    // gets canceled on timeout
    err = withTimeout(
        50*time.Millisecond,
        func() {
            time.Sleep(100 * time.Millisecond)
            fmt.Println("work done")
        },
    )
    fmt.Println("err =", err)
}

work done
err = <nil>
err = timeout

Timer

A timer (time.Timer) is a structure with a C channel to which it sends the current time when it triggers (expires). Timers are useful for planning future executions:

done := make(chan struct{})

timer := time.NewTimer(50 * time.Millisecond)
go func() {
    eventTime := <-timer.C  // blocks for 50ms
    fmt.Println("work done at", eventTime)
    close(done)
}()

<-done

work done at 2009-11-10 23:00:00.05

Stop() stops the timer and returns true if it hasn't expired yet, and false otherwise:

// timer expires after 50ms
timer := time.NewTimer(50 * time.Millisecond)
go func() {
    eventTime := <-timer.C
    fmt.Println("work done at", eventTime)
}()

// after 10ms, the timer hasn't expired yet
time.Sleep(10 * time.Millisecond)

if timer.Stop() {
    fmt.Println("execution canceled")
} else {
    fmt.Println("too late to cancel")
}

execution canceled

It's often more convenient to use the time.AfterFunc() wrapper function. It waits for duration d and then executes function f:

done := make(chan struct{})
work := func() {
    fmt.Println("work done")
    close(done)
}

// executes work after 50ms
time.AfterFunc(50*time.Millisecond, work)
<-done

work done

time.AfterFunc() returns a timer that you can cancel before execution starts:

// executes the function after 50ms
timer := time.AfterFunc(50*time.Millisecond, func() {})

// after 10ms, the timer hasn't expired yet
time.Sleep(10 * time.Millisecond)

if timer.Stop() {
    fmt.Println("execution canceled")
}

execution canceled

If a timer is used in a loop, it's better to create a single timer and reset it instead of creating a new instance on each iteration:

// consumer reads tokens from the input channel and alerts
// if a value does not appear in a channel after an hour.
func consumer(in <-chan token) {
    const timeout = time.Hour
    timer := time.NewTimer(timeout)
    for {
        timer.Reset(timeout)
        select {
        case <-in:
            // do stuff
        case <-timer.C:
            // log warning
        }
    }
}

// Suppose we send 10,000 values to the in channel
// and measure memory usage.

Memory used: 4 KB, # allocations: 6

Ticker

A ticker is like a timer, but it keeps firing until you stop it. Tickers are useful for executing periodic tasks:

// fires every 50ms
ticker := time.NewTicker(50 * time.Millisecond)
defer ticker.Stop()

go func() {
    for {
        // waits for ticker to fire on each iteration
        at := <-ticker.C
        fmt.Println("work done at", at)
    }
}()

// enough time for the ticker to fire 3 times
time.Sleep(160*time.Millisecond)
ticker.Stop()

work done at 2009-11-10 23:00:00.05
work done at 2009-11-10 23:00:00.10
work done at 2009-11-10 23:00:00.15

NewTicker(d) creates a ticker that sends the current time to the channel C at interval d. You must stop the ticker eventually with Stop() to free up resources.

If the channel reader can't keep up with the ticker, the ticker will skip ticks.

# Context

The main purpose of context is to cancel operations, either manually or by timeout/deadline.

The function accepts a context and uses its Done() channel to listen for cancellation:

// work performs a task for 50 ms unless canceled.
// Returns an error when canceled.
func work(ctx context.Context) error {
    done := make(chan struct{})

    go func() {
        time.Sleep(50 * time.Millisecond)
        fmt.Println("work done")
        close(done)
    }()

    select {
    case <-done:
        return nil
    case <-ctx.Done():
        return ctx.Err()
    }
}

Cancel manually (context.Canceled error):

func main() {
    // empty context
    ctx := context.Background()
    // manual canellation context
    ctx, cancel := context.WithCancel(ctx)
    defer cancel()

    done := make(chan struct{})
    go func() {
        // takes 50 ms unless canceled
        err := work(ctx)
        fmt.Println("err =", err)
        close(done)
    }()

    // cancels after 10 ms
    time.Sleep(10 * time.Millisecond)
    cancel()
    <-done
}

err = context canceled

Cancel by timeout (context.DeadlineExceeded error):

func main() {
    ctx := context.Background()
    // cancels after 10 ms
    ctx, cancel := context.WithTimeout(ctx, 10*time.Millisecond)
    defer cancel()

    done := make(chan struct{})
    go func() {
        // takes 50 ms unless canceled
        err := work(ctx)
        fmt.Println("err =", err)
        close(done)
    }()

    <-done
}

err = context deadline exceeded

Cancel by deadline (context.DeadlineExceeded error):

func main() {
    ctx := context.Background()
    // cancels at now + 10 ms
    deadline := time.Now().Add(10 * time.Millisecond)
    ctx, cancel := context.WithDeadline(ctx, deadline)
    defer cancel()

    done := make(chan struct{})
    go func() {
        // takes 50 ms unless canceled
        err := work(ctx)
        fmt.Println("err =", err)
        close(done)
    }()

    <-done
}

err = context deadline exceeded

Context is layered. A context object is immutable. To add new properties to a context, a new (child) context is created based on the old (parent) context. The shorter timeout between the parent and child contexts always wins. The child context can only shorten the parent's timeout, not extend it:

func main() {
    // parent context with a 100 ms timeout
    const dur100ms = 100 * time.Millisecond
    parentCtx, cancel := context.WithTimeout(context.Background(), dur100ms)
    defer cancel()

    // child context with a 10 ms timeout
    const dur10ms = 10 * time.Millisecond
    childCtx, cancel := context.WithTimeout(parentCtx, dur10ms)
    defer cancel()

    // now the work gets canceled
    err := work(childCtx)
    fmt.Println("err =", err)
}

err = context deadline exceeded

Multiple cancels are safe. You can call cancel() on the context as many times as you want. The first cancel will work, and the rest will be ignored.

You can specify a custom cancellation cause using context.WithCancelCause(), context.WithTimeoutCause() and context.WithDeadlineCause(). This cause is accessible through context.Cause():

ctx, cancel := context.WithCancelCause(context.Background())
cancel(errors.New("the night is dark"))
fmt.Println(context.Cause(ctx))

the night is dark

You can register a function to execute when the context is canceled with context.AfterFunc():

ctx, cancel := context.WithCancel(context.Background())
cleanup := func() { fmt.Println("cleanup") }
context.AfterFunc(ctx, cleanup)
cancel()
time.Sleep(10 * time.Millisecond)

cleanup

Context can pass additional information about a call using context.WithValue(), which creates a context with a value for a specific key. But it's generally better to avoid passing values in context. It's better to use explicit parameters or custom structs instead.

# Wait groups

The sync.WaitGroup type lets you wait for one or more goroutines to finish:

const n = 10
var wg sync.WaitGroup
wg.Add(n)
for range n {
    go func() {
        defer wg.Done()
        fmt.Print(".")
    }()
}
wg.Wait()

..........

A WaitGroup doesn't know anything about the goroutines it manages. It works with an internal counter. Calling wg.Add(1) increments the counter by one, while wg.Done() decrements it. wg.Wait() blocks the calling goroutine until the counter reaches zero.

The Go method combines Add, starting a goroutine, and Done:

var wg sync.WaitGroup
for range 10 {
    wg.Go(func() {
        fmt.Print(".")
    })
}
wg.Wait()

..........

All methods are safe to use from multiple goroutines.

Normally, all Add calls happen before Wait. But technically, there's nothing stopping you from doing some of the Add calls before Wait and some after (from another goroutine).

You can call Wait from multiple goroutines. They will all block until the group's counter reaches zero.

# Data races

A data race happens when multiple goroutines access shared data, and at least one of them modifies it. We need to protect the data from this kind of concurrent access.

A data race doesn't always cause a runtime panic. That's why Go provides a special tool called the race detector. You can turn it on with the race flag, which works with the test, run, build, and install commands.

var total int

// There's a data race on total.
var wg sync.WaitGroup
wg.Go(func() { total++ })
wg.Go(func() { total++ })
wg.Wait()

fmt.Println("total:", total)

total: 2

go run -race main.go

==================
WARNING: DATA RACE
...
2
Found 1 data race(s)

Channels are safe for concurrent reading and writing, and they don't cause data races.

Ways to prevent data races:

• Avoid concurrent data modification (typically by using channels).
• Synchronize access with mutexes.
• Use only atomic operations.

Race conditions

A race condition happens when an unpredictable order of operations from multiple goroutines leads to an incorrect system state:

// There's a race condition when working with balance.
withdraw := func(amount int) {
    if getBalance() < amount {
        return
    }
    time.Sleep(time.Millisecond)
    setBalance(getBalance() - amount)
}

setBalance(50)

var wg sync.WaitGroup
wg.Go(func() { withdraw(40) })
wg.Go(func() { withdraw(40) })
wg.Wait()

fmt.Println("balance:", getBalance())

balance: -30

If individual operations are concurrent-safe, Go's race detector won't find any issues. Because of this, it doesn't catch race conditions:

go run -race main.go

balance: -30

You can't fully eliminate uncertainty in a concurrent environment. Events will happen in an unpredictable order — that's just how concurrency works. However, you can prevent a race condition — often by protecting a composite operation with a mutex:

var mu sync.Mutex
withdraw := func(amount int) {
    mu.Lock()
    defer mu.Unlock()

    if getBalance() < amount {
        return
    }
    time.Sleep(time.Millisecond)
    setBalance(getBalance() - amount)
}

setBalance(50)

var wg sync.WaitGroup
wg.Go(func() { withdraw(40) })
wg.Go(func() { withdraw(40) })
wg.Wait()

fmt.Println("balance:", getBalance())

balance: 10

Compare-and-set

Sometimes you can prevent a race condition without using mutexes by applying an atomic compare-and-set operation or one of its flavors:

// CompareAndSet changes the value to new if the current value equals old.
// Returns true if the value was changed.
CompareAndSet(old, new any) bool

// CompareAndSwap changes the value to new if the current value equals old.
// Returns the old value.
CompareAndSwap(old, new any) any

// CompareAndDelete deletes the value if the current value equals old.
// Returns true if the value was deleted.
CompareAndDelete(old any) bool

// etc

The idea is always the same:

• Check if the assumed (old) state matches reality.
• If it does, change the state to new.
• If not, do nothing.

# Mutexes

The sync.Mutex type protects shared data and parts of your code from being accessed concurrently:

var total int
var mu sync.Mutex

var wg sync.WaitGroup
for range 100 {
    wg.Go(func() {
        mu.Lock()
        time.Sleep(time.Millisecond)
        total++
        mu.Unlock()
    })
}
wg.Wait()

total: 100

The mutex guarantees that only one goroutine can run the code between Lock() and Unlock() at a time.

A mutex is used in these situations:

• When multiple goroutines are modifying the same data.
• When one goroutine is modifying the data and others are reading it.

If all goroutines are only reading the data, you don't need a mutex.

TryLock

The TryLock method tries to lock the mutex, just like a regular Lock. But if it can't, it returns false right away instead of blocking the goroutine:

var total int
var mu sync.Mutex

var wg sync.WaitGroup
for range 100 {
    wg.Go(func() {
        if !mu.TryLock() {
            return
        }
        defer mu.Unlock()
        time.Sleep(time.Millisecond)
        total++
    })
}
wg.Wait()

total: 1

RWMutex

The sync.RWMutex type distinguishes between readers and writers. It provides two sets of methods:

• Lock / Unlock lock and unlock the mutex for both reading and writing.
• RLock / RUnlock lock and unlock the mutex for reading only.

var total int
var mu sync.RWMutex
var wg sync.WaitGroup

// 10 writers.
for range 10 {
    wg.Go(func() {
        mu.Lock()
        defer mu.Unlock()
        time.Sleep(time.Millisecond)
        total++
    })
}

// 10 readers.
for range 10 {
    wg.Go(func() {
        // Try switching from RLock/RUnlock to Lock/Unlock
        //and see how it affects the elapsed time.
        mu.RLock()
        defer mu.RUnlock()
        time.Sleep(time.Millisecond)
        _ = total
    })
}

wg.Wait()

elapsed: 10ms

Here's how it works:

• If a goroutine locks the mutex with Lock(), other goroutines will be blocked if they try to use Lock() or RLock().
• If a goroutine locks the mutex with RLock(), other goroutines can also lock it with RLock() without being blocked.
• If at least one goroutine has locked the mutex with RLock(), other goroutines will be blocked if they try to use Lock().

This creates a "single writer, multiple readers" setup.

Locker

Both sync.Mutex and sync.RWMutex implement the same sync.Locker interface:

type Locker interface {
    Lock()
    Unlock()
}

By using Locker instead of a specific mutex type, you can build components that don't depend on a specific lock implementation. This lets the client decide which lock to use.

Channel as mutex

You can use a channel instead of a mutex to protect shared data:

var total int
lock := make(chan struct{}, 1)

var wg sync.WaitGroup
wg.Go(func() {
    lock <- struct{}{}
    defer func() { <-lock }()
    total++
})
wg.Go(func() {
    lock <- struct{}{}
    defer func() { <-lock }()
    total++
})
wg.Wait()

total: 2

# Semaphores

A semaphore is like a container with N available slots and two operations: acquire to take a slot and release to free a slot. Here are the semaphore rules:

• Calling acquire takes a free slot.
• If there are no free slots, acquire blocks the goroutine that called it.
• Calling release frees up a previously taken slot.
• If there are any goroutines blocked on acquire when release is called, one of them will immediately take the freed slot and unblock.

You can implement a simple semaphore with a buffered channel, where N is the channel's size. To acquire the semaphore, send a value into the channel. To release it, take a value from the channel:

// Try changing nConc and see how the elapsed time changes.
const nConc = 4
const nCalls = 100
sema := make(chan struct{}, nConc)

var wg sync.WaitGroup
for range nCalls {
    sema <- struct{}{} // acquire
    wg.Go(func() {
        defer func() { <-sema }() // release
        time.Sleep(time.Millisecond) // do some work
    })
}
wg.Wait()

elapsed: 25ms

For more complex situations, use the golang.org/x/sync/semaphore package.

Rendezvous

A rendezvous lets two goroutines wait for each other:

• There are two goroutines — G1 and G2 — and each one can signal that it's ready.
• If G1 signals but G2 hasn't yet, G1 blocks and waits.
• If G2 signals but G1 hasn't yet, G2 blocks and waits.
• When both have signaled, they both unblock and continue running.

You can implement a simple rendezvous with a wait group:

var rend sync.WaitGroup
rend.Add(2)

var wg sync.WaitGroup
wg.Go(func() {
    fmt.Println("before rendezvous")
    rend.Done()
    rend.Wait()
    fmt.Println("after rendezvous")
})
wg.Go(func() {
    fmt.Println("before rendezvous")
    rend.Done()
    rend.Wait()
    fmt.Println("after rendezvous")
})
wg.Wait()

before rendezvous
before rendezvous
after rendezvous
after rendezvous

Barrier

A barrier is a general case of a rendezvous. It lets N goroutines wait for each other:

• The barrier has a counter (starting at 0) and a threshold N.
• Each goroutine that reaches the barrier increases the counter by 1.
• The barrier blocks any goroutine that reaches it.
• Once the counter reaches N, the barrier unblocks all waiting goroutines.

You can implement a simple barrier with a wait group:

const n = 4
var bar sync.WaitGroup
bar.Add(n)

var wg sync.WaitGroup
for range n {
    wg.Go(func() {
        fmt.Println("before the barrier")
        bar.Done()
        bar.Wait()
        fmt.Println("after the barrier")
    })
}
wg.Wait()

before the barrier
before the barrier
before the barrier
before the barrier
after the barrier
after the barrier
after the barrier
after the barrier

# Signaling

The sync.Cond (conditional variable) type lets one goroutine signal to another that it's ready, and lets the other goroutine wait for that signal.

A Cond includes a mutex and has two methods — Wait and Signal.

• Wait unlocks the mutex and suspends the goroutine until it receives a signal.
• Signal wakes the goroutine that is waiting on Wait.
• When Wait wakes up, it locks the mutex again.

cond := sync.NewCond(&sync.Mutex{})
done := false

var wg sync.WaitGroup
wg.Go(func() {
    cond.L.Lock()
    fmt.Println("G1 is ready to signal")
    done = true
    cond.Signal()
    cond.L.Unlock()
})
wg.Go(func() {
    cond.L.Lock()
    for !done {
        cond.Wait()
    }
    fmt.Println("G2 received the signal")
    cond.L.Unlock()
})
wg.Wait()

G1 is ready to signal
G2 received the signal

If there are multiple waiting goroutines when Signal is called, only one of them will be resumed. If there are no waiting goroutines, Signal does nothing.

You can also use the Broadcast method. While Signal wakes up only one goroutine waiting on Cond.Wait, the Broadcast method wakes up all such goroutines.

You can signal with a channel:

signal := make(chan struct{}, 1)

go func() {
    // do something
    signal <- struct{}{}
}()

go func() {
    <-signal
    // do something
}()

And broadcast too:

broadcast := make(chan struct{})

go func() {
    // do something
    close(broadcast)
}()

go func() {
    <-broadcast
    // do something
}()

go func() {
    <-broadcast
    // do something
}()

Broadcasting with a condition variable is limited: it only sends a signal, not the actual data, and it only works once. With channels, you can build a publish/subscribe system that doesn't have these limitations:

type Publisher struct {
    sbox []chan int // subscription channels
    mu   sync.Mutex // protects the state
}

func (p *Publisher) Subscribe() <-chan int {
    p.mu.Lock()
    defer p.mu.Unlock()
    sub := make(chan int, 1)
    p.sbox = append(p.sbox, sub)
    return sub
}

func (p *Publisher) Broadcast(v int) {
    p.mu.Lock()
    defer p.mu.Unlock()
    for _, sub := range p.sbox {
        select {
        case sub <- v:
        default:
        }
    }
}

# Run once

The sync.Once type makes sure that the given function runs only once. If multiple goroutines call Once.Do at the same time, only one will run the function, while the others will wait until it returns:

total := 0
initState := func() {
    total += 1
}

var once sync.Once

var wg sync.WaitGroup
wg.Go(func() {
    once.Do(initState)
    // do something
})
wg.Go(func() {
    once.Do(initState)
    // do something
})
wg.Wait()

total: 1

Once is perfect for one-time initialization or cleanup in a concurrent environment.

Besides the Once type, the sync package also includes three convenience once-functions:

// Calls f only once.
func (o *Once) Do(f func())

// Returns a function that calls f only once.
func OnceFunc(f func()) func()

// Returns a function that calls f only once
// and returns the value from that first call.
func OnceValue[T any](f func() T) func() T

// Returns a function that calls f only once
// and returns the pair of values from that first call.
func OnceValues[T1, T2 any](f func() (T1, T2)) func() (T1, T2)

# Object pool

The sync.Pool type helps reuse memory instead of allocating it every time, which reduces the load on the garbage collector:

pool := sync.Pool{
    New: func() any {
        buf := make([]byte, 1024)
        return &buf
    },
}

// Only allocates 4*1024 B, despite 4000 loop iterations.
var wg sync.WaitGroup
for range 4 {
    wg.Go(func() {
        for range 1000 {
            buf := pool.Get().(*[]byte)
            sink = buf
            pool.Put(buf)
        }
    })
}
wg.Wait()

Memory allocated: 4 KB

Get takes an item from the pool. If there are no available items, it creates a new one using New (which we have to define ourselves, since the pool doesn't know anything about the items it creates). Put returns an item back to the pool.

Things to keep in mind:

• New should return a pointer, not a value, to reduce memory copying and avoid extra allocations.
• The pool has no size limit. If you start 1000 more goroutines that all call Get at the same time, 1000 more buffers will be allocated.
• After an item is returned to the pool with Put, you shouldn't use it anymore (since another goroutine might already have taken and started using it).

# Atomics

An operation without synchronization can only be truly atomic if it translates to a single processor instruction. Such operations don't need locks and won't cause issues when called concurrently (even the write operations).

There are only a few atomics, and they're all found in the sync/atomic package:

Int32     Bool
Int64     Value
Uint32    Pointer
Uint64

Each atomic type provides the following methods:

• Load reads the value of a variable.
• Store sets a new value.
• Swap sets a new value (like Store) and returns the old one.
• CompareAndSwap sets a new value only if the current value is still what you expect it to be.

var n atomic.Int32
n.Store(10)
swapped := n.CompareAndSwap(10, 42)
fmt.Println("CompareAndSwap 10 -> 42:", swapped)
fmt.Println("n =", n.Load())

CompareAndSwap 10 -> 42: true
n = 42

Numeric types also provide an Add method that increments the value by the specified amount.

All methods are either translated into a single CPU instruction or are otherwise guaranteed to be atomic, so they are safe to use from multiple goroutines.

The composition of atomics is always non-atomic:

var delta atomic.Int32
var counter atomic.Int32

func increment() {
    // Not atomic; causes a race condition.
    delta.Add(1)
    sleep(10)
    counter.Add(delta.Load())
}

// After 100 concurrent increments,
// the final value is NOT guaranteed.

counter = 9386

A bulletproof way to make a composite operation atomic and prevent race conditions is to use a mutex:

var delta int32
var counter int32
var mu sync.Mutex

func increment() {
    // Atomic; doesn't cause a race condition.
    mu.Lock()
    delta += 1
    sleep(10)
    counter += delta
    mu.Unlock()
}

// After 100 concurrent increments, the final value is guaranteed:
// counter = 1+2+...+100 = 5050

counter = 5050

Sometimes you can use an atomic type instead of a mutex to exit early:

type Gate struct {
    closed atomic.Bool
}

func (g *Gate) Close() {
    if !g.closed.CompareAndSwap(false, true) {
        return // ignore repeated calls
    }
    // The gate is closed.
    // We can free resources now.
}

# Testing

If your concurrent program uses channels or custom types with synchronization methods like Wait, you can use those in your tests. This way, your tests won't be much more complicated than if the code were synchronous:

// Calc calculates something asynchronously.
func Calc() <-chan int {
    out := make(chan int, 1)
    go func() {
        out <- 42
    }()
    return out
}

func Test(t *testing.T) {
    // Wait for the Calc goroutine to finish.
    got := <-Calc()
    if got != 42 {
        t.Errorf("got: %v; want: 42", got)
    }
}

PASS

If there aren't any suitable synchronization "handles" in the code you're testing, you can use the synctest package. It exports two functions:

func Test(t *testing.T, f func(*testing.T))
func Wait()

synctest.Test runs an isolated bubble. The bubble uses a fake clock, and you can manually control goroutine synchronization with synctest.Wait.

synctest.Wait blocks until all goroutines in the bubble — except the one that called Wait — have either finished or are durably blocked. This lets you wait for a specific goroutine to finish or get blocked, so you can check the program's state:

// NewProc starts the calculation.
func NewProc() *Proc {
    p := &Proc{done: make(chan struct{})}
    go func() {
        p.res = 42
        <-p.done // (X)
        p.res = 0
    }()
    return p
}

func Test(t *testing.T) {
    synctest.Test(t, func(t *testing.T) {
        p := NewProc()
        defer p.Stop()

        // Wait for the goroutine to block at point X.
        synctest.Wait()
        if got := p.Res(); got != 42 {
            t.Fatalf("got %v, want 42", got)
        }
    })
}

PASS

The fake clock in synctest.Test move forward only if: ➊ all goroutines in the bubble are durably blocked; ➋ there's a future moment when at least one goroutine will unblock; and ➌ synctest.Wait isn't running. Thanks to this, time-dependent tests run instantly:

// Calc processes a value from the input channel.
// Times out if no input is received after 3 seconds.
func Calc(in chan int) (int, error) {
    select {
    case v := <-in:
        return v * 2, nil
    case <-time.After(3 * time.Second):
        return 0, ErrTimeout
    }
}

func Test(t *testing.T) {
    synctest.Test(t, func(t *testing.T) {
        ch := make(chan int)
        got, err := Calc(ch) // runs instantly

        if err != ErrTimeout {
            t.Errorf("got: %v; want: %v", err, ErrTimeout)
        }
        if got != 0 {
            t.Errorf("got: %v; want: 0", got)
        }
    })
}

PASS

The following operations durably block a goroutine:

• A blocking send or receive on a channel created within the bubble.
• A blocking select statement where every case is a channel created within the bubble.
• Calling Cond.Wait.
• Calling WaitGroup.Wait if all WaitGroup.Add calls were made inside the bubble.
• Calling time.Sleep.

Blocking on mutexes, I/O, or system calls is not considered durable, and the synctest bubble can't handle them.

# Scheduling

At the hardware level, CPU cores are responsible for running parallel tasks.

At the operating system level, a thread is the basic unit of execution. There are usually many more threads than CPU cores, so the operating system's scheduler decides which threads to run and which ones to pause.

At the Go runtime level, a goroutine is the basic unit of execution. The runtime scheduler runs a fixed number of OS threads, often one per CPU core. There can be many more goroutines than threads, so the scheduler decides which goroutines to run on the available threads and which ones to pause. The scheduler keeps switching between goroutines to make sure each one gets a turn to run on a thread, instead of waiting in line forever.

  CPU                  OS                   Go runtime
┌──────────┐  run on ┌──────────┐  run on ┌────────────┐
│ Cores    │ <────── │ Threads  │ <────── │ Goroutines │
└──────────┘         └──────────┘         └────────────┘

This is how Go handles concurrency.

Goroutine scheduler

The goroutine scheduler's job is to run M goroutines on N operating system threads, where M can be much larger than N. Here's a very simplified version of it's algorithm:

• If there's a free thread, assign it a goroutine from the queue.
• If a running goroutine gets blocked (for example, while reading from a channel), put it back in the queue and assign a different goroutine to the thread.
• If a running goroutine gets stuck in a syscall, start a new thread to run other goroutines until the blocked goroutine finishes the syscall.
• Check the running goroutines every 10 ms. Preempt long-running goroutines and return them to the queue to prevent starvation.

┌─────┐┌─────┐┌─────┐┌─────┐
│ G17 ││ G18 ││ G19 ││ G20 │                        queue
└─────┘└─────┘└─────┘└─────┘

┌─────┐      ┌─────┐      ┌─────┐      ┌─────┐
│ G15 │      │ G16 │      │ G13 │      │ G14 │      running
└─────┘      └─────┘      └─────┘      └─────┘
  │            │            │            │
┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐
│ Thread E │ │ Thread F │ │ Thread C │ │ Thread D │
└──────────┘ └──────────┘ └──────────┘ └──────────┘

┌─────┐      ┌─────┐
│ G11 │      │ G12 │                                syscalls
└─────┘      └─────┘
  │            │
┌──────────┐ ┌──────────┐
│ Thread A │ │ Thread B │
└──────────┘ └──────────┘

The number of threads running Go code is controlled by the GOMAXPROCS environment variable or the runtime.GOMAXPROCS function.

A goroutine is a structure that starts out using about 2 KB of memory, mostly for its stack. The stack can grow if needed. Since goroutines are so lightweight, you can run tens of thousands or even hundreds of thousands of them on a small machine.

# Diagnostics

To troubleshoot concurrent programs in production, we use metrics, profiling, and tracing.

Metrics show how the Go runtime is performing, like how much heap memory it uses or how long garbage collection pauses take. Each metric has a unique name and a value, which can be a number or a histogram.

You can use the runtime/metrics package to get a complete list of metrics or check the values of specific ones:

samples := []metrics.Sample{
    {Name: "/sched/gomaxprocs:threads"},
    {Name: "/sched/goroutines:goroutines"},
}
metrics.Read(samples)

for _, s := range samples {
    fmt.Printf("%s: %v\n", s.Name, s.Value.Uint64())
}

/sched/gomaxprocs:threads: 8
/sched/goroutines:goroutines: 1

In practice, people rarely do this manually. Instead, all metrics are automatically exported using Prometheus or OpenTelemetry libraries.

Profiling helps you understand exactly what the program is doing, what resources it uses, and where in the code this happens. Go uses a sampling profiler that's suitable for production.

The most commonly used profiles are CPU, which shows how much processor time each function uses, and heap, which shows how much heap memory each function uses. Goroutine, block, and mutex profiles help identify problems related to concurrency.

The easiest way to add a profiler to your app is by using the net/http/pprof package. To collect a profile with the given name, call the /debug/pprof/{name} endpoint. To view the collected profile, use the go tool pprof utility:

go tool pprof -proto \
  "http://localhost:6060/debug/pprof/profile?seconds=N" > cpu.pprof
go tool pprof -http=localhost:8080 cpu.pprof

You can also profile manually:

// CPU profile.
file, _ := os.Create("cpu.prof")
defer file.Close()
pprof.StartCPUProfile(file)
defer pprof.StopCPUProfile()
// ...

// Any other profile.
file, _ := os.Create(name + ".prof")
defer file.Close()
pprof.Lookup(name).WriteTo(file, 0)

Tracing records certain types of events while the program is running, mainly those related to concurrency and memory. When the profiling server from the net/http/pprof package is running, call the /debug/pprof/trace endpoint to collect a trace. To view the results, use the go tool trace utility.

You can also collect a trace manually:

file, _ := os.Create("trace.out")
defer file.Close()
trace.Start(file)
defer trace.Stop()
// ...

You can set up automatic tracing with a sliding window that's limited by size or duration. This is called "flight recording". It lets you always keep a recent trace available in case something goes wrong:

cfg := trace.FlightRecorderConfig{
    MinAge:   5 * time.Second,
    MaxBytes: 3 << 20, // 3MB
}
rec := trace.NewFlightRecorder(cfg)
rec.Start()
defer rec.Stop()

# Final thoughts

We've covered a number of Go tools for writing concurrent programs:

• Goroutines for running concurrent tasks.
• Channels and select as flexible communication tools.
• Timers and tickers for working with time.
• Context for canceling operations.
• Wait groups for synchronizing goroutines.
• Mutexes to prevent race conditions.
• Condition variables for signaling events.
• Once for safe one-time initialization.
• Pools to reduce garbage collector load.
• Atomic operations.

If you like the book, please recommend it to your friends or colleagues. If you're interested, check out my other books and projects.

I'm glad you finished the book. Thank you, and I'll see you next time!

Added to last week’s revelations, new filings reveal OpenAI's lawless determination to do virtually anything to win the AI arms race, including using books from a “sketchy Russian website” and releasing products “that will make people unemployed”

(New York, September 21, 2026) Explosive new documents released last Thursday in Authors Guild v. OpenAI—the major book authors’ ongoing lawsuit alleging OpenAI and Microsoft illegally trained their products on copyrighted books—show that executives and employees at both companies intentionally broke the law and their disdain for human authorship. The brief meticulously documents AI developers’ knowledge of the damage their actions could do to authors, publishers, and American culture itself, warning that “OpenAI’s GPT models pose an existential threat to those who write and publish books.”

Plaintiffs in the lawsuit include The Authors Guild, writers David Baldacci, Taylor Branch, Michael Connelly, Sylvia Day, Jonathan Franzen, Christopher Golden, Andrew Sean Greer, John Grisham, David Henry Hwang, Matthew Klam, George R.R. Martin, Jodi Picoult, Stacy Schiff, James Shapiro, and others. Their filings (here and here) mark a turning point in the case, putting the defendant companies’ guilty knowledge and reckless actions at the center of the narrative.

Authors Guild CEO Mary Rasenberger observed, “These filings reveal shocking disdain for writers and their work through repeated, intentional decisions to steal books rather than pay for them with full knowledge that their products will destroy the careers of authors. OpenAI pursued its mass piracy scheme even in the face of clear evidence that doing so would degrade American culture by substituting human works with AI slop and would put thousands of writers out of work. We are grateful to our entire legal team for uncovering and telling this critical story.”

Alongside equally scorching but separate revelations in a brief filed by news media organizations, the authors’ brief reveals:

• OpenAI and Microsoft Knew Their Systems Would Replace Human Writers – OpenAI Policy Director Jack Clark stated in May 2020: “Our work on AI and Creativity is going to increasingly lead to us creating systems that substitute for the labor of [] people . . . The better we do on GPT-X, the more worried genre fiction authors will become about us substituting for them on Amazon.” He then doubled down: *“*Our work in this area will make people unemployed. . . There will be a point where a bunch of artists express worry about what we’re doing here and we’ll likely ignore their concerns and release anyway . . .”
• “Autocompleting” Other People’s Work – OpenAI hired Tarun Gogineni in 2022 to lead its efforts to improve the writing quality of its models. Gogineni stated his “research mission” was to have GPT models write the “last two books of [Martin’s] A Song of Ice and Fire” series by “finishing it one day.” Gogineni even mused that he would “rest easy knowing that even if GRRM [George R. R. Martin] dies early, GPT-5 will autocomplete his series.”
• “Acceptable Economic Disruption.” – “Although [Gogineni] was aware of authors’ complaints that the ‘datasets are stolen’ and that authors were losing work to AI-generated competition, he did not find those complaints ‘all that sympathetic’ and instead viewed them as ‘acceptable economic disruption.’ He wrote that the world would soon experience ‘the death of the reader’ as ‘machines creat[ed] slop for more machines.’”
• Microsoft Knew OpenAI Used Pirated Books from the Start – “Microsoft knew about OpenAI’s use of LibGen as early as April 2019 when Sam Altman and Dario Amodei presented an early version of GPT-3 to Bill Gates and disclosed the use of LibGen to Gates and Microsoft’s Chief Technology Officer Kevin Scott, among others.”
• OpenAI Feared “Optics,” Not the Law – “Dario Amodei, OpenAI’s then-Research Director, responded that ‘as a training set [LibGen is] a bit sketchier.’ [OpenAI researcher Sam] McCandlish explained: ‘I was just worried about optics – i.e. ‘openai uses copyrighted data from sketchy russian website’ showing up on [Hacker News] would be unfortunate.”
• They Tried to Hide the Evidence – In an effort to cover its tracks the company called “Project Clear,” OpenAI deleted its LibGen files in the summer of 2022. “On June 15, 2022, in an OpenAI Slack channel, [OpenAI Corporate Designee] Paino asked: ‘general q: how concerned are we about mentions of libgen? (they’re all over google docs/slack/github . . .). That evening, [OpenAI VP of Research Bob] McGrew wrote to Paino and Nicholas Ryder: “Given how much OpenAI is in the news, now is the right time to excise Libgen from our systems and storage. What would be involved in that?’”

All quotes taken from Class Plaintiffs’ Memorandum of Law in Support of Motion for Partial Summary Judgment (Docket 1982) and Class Plaintiffs’ Corrected Rule 56.1 Statement of Undisputed Material Facts (Docket 1987).

This briefing was filed in the class action case brought on behalf of the Authors Guild and book copyright owners, Alter v. OpenAI and Microsoft, with lead counsel Justin A. Nelson from Susman Godfrey. The case is part of multidistrict litigation against those defendants pending in Manhattan. We expect more briefing over the next couple months and a hearing in early 2027.

Ember-1: half the tokens, same answers

Ember-1 is a new specialized model from Fireworks Research that delivers Kimi K3’s quality with 40% fewer tokens. Built on Kimi K3, it learned to cut unnecessary reasoning while keeping the thinking that matters. We tested it on external benchmarks, in live customer A/B tests, and on our own coding and agent workloads, and quality held up in every setting. Available today, Ember-1 kicks off an ongoing series of specialized models by Fireworks, shaped by what developers want next. Ember is just the start of what you could build with the Fireworks Training platform.

How Fireworks Research built Ember-1

We heard from users that they needed K3’s coding capabilities at a lower cost, because its long reasoning traces made automated coding expensive at scale. Turning down K3's reasoning effort didn't solve this. Lower effort settings gave up too much quality. To keep the quality and cut the tokens, the model had to learn to reason more efficiently, and that meant training it.

Getting there took serious research. Our team ran more than 50 training experiments and over 200 evaluations, and developed new training algorithms along the way to shorten reasoning without losing accuracy. We did it all on Fireworks Serverless Training. Because we didn’t have to provision or manage GPUs, we could launch experiments as soon as we had an idea, pay only for what we ran, and move from research to launch in a fraction of the usual time and cost.

We trained across a broad set of tasks so the token savings would carry over to many workloads. We used our own data, and no customer data to train this model. We then evaluated Ember-1 on the Specialized Intelligence Index, public benchmarks, and live production traffic to confirm it used fewer tokens with no drop in quality. Ember-1 is Fireworks’ own model and the first in a series of models from Fireworks Research.

The problem: thinking models think too much

Reasoning models like Kimi K3 spend the majority of their generated tokens, sometimes more than 90%, on internal reasoning rather than the answer itself. This thinking structure is expensive on a single request, but it gets much worse in multi-turn agentic workloads. Every turn replays all prior reasoning back to the model, so context grows roughly quadratically with the number of turns. Long reasoning traces from early turns get re-read (and re-billed) on every subsequent call.

Is all that reasoning actually necessary? Our experiments said no. The reasoning Kimi K3 emits is far longer than the task requires, and the excess can be removed without touching the answer. This was how we created Ember-1, an economical version of Kimi K3 built from specialized intelligence.

From an observation to a premium model

Not all of K3's reasoning is wasted. Some of it is self-reflection: revisiting an assumption, responding to feedback, or tracing an outcome back to an earlier decision can help the model recover from mistakes. The opportunity is to preserve this ability while reducing unnecessary reasoning and escaping unproductive loops. We believe that learning from tasks and environment feedback can teach the model to reason more efficiently while maintaining its capabilities.

For agentic tasks, this learning extends across the interaction. The model explores possible actions, incorporates new observations, and refines its reasoning as it progresses. Feedback connects decisions to their consequences, encouraging useful reflection throughout the task.

We carried these insights into a training collection spanning mathematics, coding, instruction following, conversation, search, tool use, and software engineering, covering both standalone problems and extended interactions to enforce adaptation to observations and outcomes. Task feedback guides on-policy planning and learning, with an emphasis on preserving capability across this range of settings.

Results on public benchmarks and live A/B tests support this direction: across seven benchmarks and two customers’ production traffic, Kimi K3’s reasoning could be shortened by 35–50% without sacrificing accuracy. The internalized behavior also shows restrained token use on unsuccessful attempts, reducing prolonged, unproductive reasoning.

The Specialized Intelligence Index: Ember-1 sets a Pareto frontier for Bedside Bench

Earlier this week, we introduced the Specialized Intelligence Index (SII) to benchmark open, closed, and specialized models against real-world tasks created by industry experts.

We evaluated Ember-1 on Doximity’s Bedside Bench, a physician-validated benchmark spanning 500 clinical cases across 10 specialized categories.

The result? Ember-1 set a new Pareto frontier for Bedside Bench across both open and closed models including GPT-5.6 Sol, GPT-6 Astra, and Claude Opus 5 on cost/task.

Figure 1: Pareto Frontier from SII on Bedside Bench

Figure 2: Score vs. Duration Chart on Bedside Bench SII

Evaluating Pareto across more industry benchmarks

We also evaluated Ember-1 on the quality-vs-cost frontier across some other industry benchmarks. We computed per-benchmark cost using the public Kimi K3 API pricing (uncached input $3/M tokens, cached input $0.30/M, output $15/M) and plotted it against pass rate for three arms: K3 at reasoning effort low, K3 at reasoning effort high, K3 at reasoning effort max (default), and Ember-1. Across every benchmark with more than 50 test samples, Ember-1 sits on or near the Pareto frontier, matching K3-max quality at a fraction of the cost, and strictly dominating K3-low. We also analyzed GPT-6 Astra, Claude Opus-5 and GLM 5.3, and found that Ember-1 was a leader on the Pareto frontier.

Figure 3: Average of 5 Industry Benchmarks on Open and Closed Model Cost/Task

We took a double-click on the results directly comparing Ember-1 to the original K3, and found the following results:

Industry BenchmarksNK3 LowK3 HighK3 maxEmber-1Ember-1 vs. K3 Max Terminal Bench 2.1

89

76.4%

77.6%

80.9%

82.0%

-51.9% / -23.1 USD

SWE-bench Verified

500

80.4%

86.0%

93.2%

92.2%

-15.5% / -68.1 USD

SWE-Interact

75

6.7%

13.3%

21.3%

20.0%

-32.5% / -60.8 USD

DeepSWE 1.1

113

55.8%

62.8%

66.4%

75.2%

-23.7% / -126.9 USD

τ-2 Bench Airline

50

64%

64%

64%

66%

-5.9% / -0.3 USD

The most cost optimized way to run K3 is no longer to make it think less, but to run Ember-1, the model that learned to think efficiently.

Customer validation: Live A/B tests

Benchmarks only tell you so much. Like what we found in the Specialized Intelligence Index results, we wanted to test the model on more real workloads, and to test the model using production traffic. The real test is often whether the model holds up on production traffic, in products users depend on.

We ran live A/B tests with two customers on their production coding workloads. In both cases, Ember-1 delivered impressive token savings, approximately 35% fewer tokens per task at comparable quality. Most of the downstream product metrics held or improved, including task completion, success scores, and failure rates all moving in the right direction at substantially lower token cost. Following the A/B tests, one customer is now running Ember-1 in live production, with plans to scale it up to replace the base model entirely.

ScoreStepsOutput TokensReasoning Token reductionTotal token reductionKimi K3

0.751

23.8

49.3K

-

-

Ember-1

0.753

21.4

29.9K

71.3%

39%

Internal validation: Our own developers didn't notice

A large part of Fireworks’ internal coding/cowork traffic is powered by our own inference service. Before any customer saw the model, we put Ember-1 to work internally and let our own developers use it for everyday coding work including things like vibe testing at scale on real tasks.

The outcome we're proudest of: no news. No news is good news. Developers carried on their coding workloads without noticing the switch, while consuming substantially fewer tokens. For a model whose entire value proposition is "same answers, fewer tokens," an invisible rollout on internal traffic is the strongest possible signal.

What's next

Ember-1 is rolling out as a serving option alongside the base Kimi K3 model as a Research Preview release on Serverless. To support the rapidly growing open-source ecosystem, we're introducing research releases to give developers two-week serverless access to new research models, making them permanent based on community demand. For agentic coding and other workloads where reasoning tokens account for most of the cost, it delivers the same quality at roughly half the token cost.

Fireworks Research will continue to push the frontier of model efficiency by bringing specialized intelligence to more Ember models to enable you to deploy the most economical models, and reduce your token spend. Token efficiency is becoming a theme of Fireworks.

Looking to take Ember-1 one step further, and optimize it for your use case? We are also launching training support for Ember-1, enabling enterprises to build customized, token-efficient models tailored to their needs with their own data. The future of open models is specialized models trained on your specific workload.

Trying Ember-1 out on your workloads? We'd love to hear about your experience, so tag us on X (@FireworksAI_HQ) and let us know what you're building!

Typed decisions with a probability for every option, in a single forward pass: matching Jev's accuracy and speed with an LLM.

TL;DR: In this post, we show how an off-the-shelf LLM can make typed decisions in a single forward pass. This approach makes it possible to turn an LLM into a Jev-like decision model.

We evaluate the approach using GLM-5.3-Flash running on Privatemode. Using a benchmark constructed from public data sets, we show that this setup delivers results that are on par with TypeSafe's Jev in terms of decision accuracy/correctness and speed.

As a bonus, the setup with GLM-5.3-Flash enables typed decisions on images, which is not possible with Jev.

Why typed decisions

Much of what software asks an LLM is a decision. "Which team should handle this ticket?", or "Does this contract clause belong in the liability section?".

In such cases, software typically requires that the LLM's response follows a certain format like JSON and that it comes from a pre-defined set like "yes" and "no".

Given the right instructions, LLMs can typically already fulfill this reliably. However, in the basic approach, speed and costs become an issue: For each decision, the LLM needs to write a whole JSON object, and a reasoning model may think for hundreds of tokens before that. Further, you also don't learn the confidence of the model (unless you explicitly ask it). All these aspects can matter a lot in practice and have so far prevented people from employing LLMs for decision making in high-volume/high-throughput scenarios.

Specialized decision models (or "System One" models) like Jev and Laya are designed to address this. You pass in a piece of state and a set of named options, and you get back the chosen option together with a confidence value (i.e., probability) for each one.

Turning an LLM into a decision model

Initially, we asked ourselves if an LLM could be turned into a decision model with Jev-like properties. The short answer is: "yes". In the following, we show how it works.

To understand our approach, it's important to understand how LLMs work:

An LLM never writes text directly. Given a prompt, an LLM outputs a probability distribution over its entire vocabulary of tokens. In text generation, in the simplest case, the token with the highest probability is selected as the next token. The selected token then is appended to the prompt and the whole process repeats. As described above, this is costly and slow if you just want to set a few fields in a JSON object.

Our core insight is that it's unnecessary to have the LLM predict the whole JSON object, as we already know its shape. We're only interested in the LLM's typed judgement for a given input.

We realized that it's possible to craft prompts so that we get the typed judgement in a single run of the LLM — with no fine-tuning, on the model exactly as it ships. This is the difference to Jev and Laya, which are models trained for the purpose. The basic steps are as follows:

1. Number the options. The state, the question, and the output options go into the prompt as JSON, with an index on every option. The instruction asks the model to answer with choice_index: followed by an index.
2. Prefill the answer. The prompt ends with choice_index:. Consequently, the first token the model produces will be an index into the pre-defined options.
3. Evaluate the output. Rather than reading the token the model emits, we read the probabilities it assigned to all option indexes at that single position. Normalized over the options, these give a probability for each answer, and we simply pick the most probable one.

The prompt numbers the options and it ends with the assistant’s answer already begun as choice_index:, so the next token is the index. A mask on the vocabulary allows only the option indexes, the API returns their log probabilities, and normalizing them over the options gives a probability for each answer. The logit values in the middle panel are illustrative.

We implemented the above steps for GLM-5.3-Flash running on vLLM (in Privatemode). We use the /chat/completions endpoint with continue_final_message and add_generation_prompt: false, because these let the model continue the prefilled assistant turn from step 2 instead of starting a new one. They also let us pass images next to the text, which is what makes typed decisions on images possible.

For vLLM and GLM-5.3-Flash, we found the following details to matter:

1. vLLM's allowed_token_ids can be used to limit the LLM's output vocabulary only to allowed options. It drops every other token to -inf. We set it, but it is a guardrail rather than a requirement.
2. top_logprobs is not enough for step 3. It reports the distribution before the restriction is applied, so formatting tokens such as a leading space take up the top slots, and some options drop off the list and appear to have a probability of zero. vLLM's logprob_token_ids solves this: it returns the log probability of exactly the token ids you ask for.
3. The token ids of the indexes depend on the model's tokenizer. Digits aren't always single tokens. GLM-5.3-Flash, for example, has a single token for 12. Rather than shipping a model-specific tokenizer, the library gets the token ids from the server, which keeps it simple to use with any model: sending a prompt to /completions with echo returns its exact tokenization by the model that is actually serving.

You can find our implementation in the below repository.

edgelesssys/privatemode-decisions — The Python library: token oracle, prompt, masking and renormalization, against any vLLM-backed endpoint.

Try it

The playground below runs GLM-5.3-Flash queried with the above setup on Privatemode, directly from your browser. Pick one of the examples, among them a scanned invoice and a question that depends on your local time, or write your own questions and add images. Each answer comes back as a distribution over its options, typically within a few hundred milliseconds.

The distribution is often very useful, e. g., to decide whether to include a human-in-the-loop. The model solves most classic trick questions, but not all of them.

Benchmark results

We evaluated our approach using a custom benchmark, which is available in the below repository.

edgelesssys/privatemode-decisions-benchmark — The benchmark: methodology, frozen dataset specs, harness and aggregation. Every number in this post can be recomputed from it.

We compared three systems on 29 public, labeled datasets: GLM-5.3-Flash hosted on Privatemode and queried with the technique above, TypeSafe's Jev, and Convai's Laya.

The datasets have between 2 and 151 options and cover intent routing, sentiment, topic classification, moderation, entailment, question answering, legal text, and scanned documents. Both English and German text is included in the corpus. All three systems receive the same state, the same option names in the same order, and the same instruction.

We ran each dataset twice. Even at temperature 0, our GLM-5.3-Flash and Jev changed up to 3.5% of their answers between identical runs: temperature 0 removes the randomness from sampling, but batching and floating-point arithmetic still keep a forward pass on a busy server from being bit-reproducible. We therefore treat smaller differences as noise.

We ran Jev and Laya with their default settings and did not tune our prompt on these datasets.

Accuracy

Compared across the 28 text datasets, GLM-5.3-Flash and Jev are on par. Each is more accurate on 10 datasets; on the remaining 8, the two are within one percentage point of each other. The median gap is 0.7 percentage points in Jev's favor, which is not statistically significant (p = 0.64). Laya, a model with 421 million parameters that we ran locally, scores lower than both on most datasets. Its median gap is 13 to 15 percentage points, which is statistically significant (p < 0.001).

The number of options has a larger effect on accuracy than the choice between Jev and GLM-5.3-Flash.

The typical gap is 0.7 percentage points, slightly in Jev’s favor. Two equally accurate systems would show a gap at least this large in about 6 out of 10 comparisons, so it is well within chance.

Across datasets, the number of options changes along with everything else about the task. Three datasets, however, label the same questions twice, once coarsely and once finely, so the task stays the same and only the number of options changes. On TREC, going from 6 to 42 options, Jev drops from 92.1% to 85.6%, GLM-5.3-Flash from 91.2% to 79.6%, and Laya from 88.4% to 51.2%. On MASSIVE, going from 18 scenarios to 59 intents raises the scores of Jev and GLM-5.3-Flash in both languages, while Laya's score drops. The number of options alone doesn't determine how hard a task is.

The same questions labeled twice, once coarsely and once finely, so the only variable that changes is the number of options. TREC splits 6 question types into 42; MASSIVE splits 18 scenarios into 59 intents, in English and in German. Numbers on the right are accuracy at the finer granularity.

Latency and cost

We measured latency in separate runs with one request at a time, because timings taken under load measure the queue rather than the model.

As Privatemode is hosted in the EU and Jev is hosted in the US, we ran four of the datasets from Germany and from the US at the same time. From Germany, Privatemode answered in 180 ms and Jev in 264 ms. From the US, the order reverses: 164 ms for Jev against 299 ms for Privatemode.

On cost, Jev is cheaper. One million decisions cost about EUR 62 with GLM-5.3-Flash and about EUR 16 with Jev, at each service's list prices.

Time per decision is the model call as a user sees it, network included, measured one request at a time from Germany and from the US at the same time, on four datasets. The dot is the median; the band runs from fast requests (10th percentile) to slow ones (95th percentile). Cost is what one million decisions cost at each service’s list prices, the median over the 28 datasets both systems answer, so the scanned documents Jev cannot read are not in it.

Most of the difference comes from the price per input token, and some from how each system packages a question. Jev adds roughly 270 tokens of fixed overhead and about 10 tokens per option. The GLM-5.3-Flash prompt adds about 55 tokens of fixed overhead and about 20 per option. Below about 21 options, it sends fewer tokens than Jev; above that, it sends more.

One point per dataset: how many input tokens GLM-5.3-Flash on Privatemode sends per question, minus how many Jev sends for the same question. The question text is identical for both, so it cancels; what is left is the packaging. Jev adds a large fixed block and little per option, GLM-5.3-Flash a small fixed block and more per option. The dashed line is the trend across all datasets: GLM-5.3-Flash starts about 219 tokens below Jev and adds about 11 more per option. The outlier at 13 options is scotus, whose long court opinions the two tokenizers split differently.

Laya runs locally, so there is no comparable latency or price per decision for it.

Multimodal decisions

The state doesn't have to be text. GLM-5.3-Flash is a vision-capable model, so a question can come with images, such as a scanned invoice, a photo of a damaged parcel, or a screenshot. The image goes into the same prompt, and the answer is still a single token with a probability for every option. The playground's Scanned document example shows this; you can also paste or drop in your own image.

According to its documentation, Jev works on text, and Laya is a text encoder, so neither takes images as input. On RVL-CDIP, a set of 1,600 scanned business documents in 16 classes, GLM-5.3-Flash reaches an accuracy of 70.2% and is the only one of the three that can answer. A document costs more than a sentence: the image adds about 1,350 input tokens, so a million document decisions cost about EUR 270.

Capabilities

With many options, each system hits a limit. Laya's option names share a budget of 192 tokens, which is enough for the 77 intents of banking77 but not for the 151 intents of CLINC150. Privatemode's deployment of GLM-5.3-Flash reports at most 128 entries in logprob_token_ids, while the mask in allowed_token_ids takes every option. So the library sends a question with 151 options twice, identically, and reads the probabilities of 128 options from the first response and of the other 23 from the second. Both requests run the same forward pass, so the merged result is the distribution a single request would return, up to the run-to-run noise above. On CLINC150, GLM-5.3-Flash reaches 87.5% this way, against 78.4% for Jev. The second request and the long option list cost time: a decision takes 719 ms, against 249 ms for Jev. Past that, the ceiling is the 191 option indexes that GLM-5.3-Flash spells as a single token.

Some tasks only one or two of the systems can handle at all.

Numbers are accuracy on the named dataset.

Further findings

Some of the remaining errors are in the labels. When most systems agree on an answer and the dataset's label disagrees, the label is often one of two defensible answers. banking77 has many such pairs, for example get_physical_card and order_physical_card, or declined_transfer and failed_transfer. About 17% of banking77's examples fall into this category, so the highest score any system could reach there is about 85% rather than 100%.

Reasoning helps, at a price. As a control, we let the same GLM-5.3-Flash reason before it answers, on all 29 datasets. It is more accurate in every band of option counts, from 89.9% against 85.5% with two options to 82.0% against 79.2% between 21 and 80. It also writes hundreds of tokens per decision instead of one and costs about EUR 350 per million decisions, against EUR 62. At the other end, plain embedding similarity, which picks the option closest to the text with no decision model at all, reaches between 45.9% and 72.8% depending on the band.

Renaming the options affects the systems differently. We re-ran every dataset with each option replaced by a synonym and nothing else changed. On boolq, where true and false became correct and wrong, GLM-5.3-Flash lost 20 points, while the other two lost less than three. Because this renaming also changes the meaning of the question, it doesn't isolate memorization. The results for every dataset are in the benchmark repository.

Build it yourself

Our library is written in Python and works with any vLLM-backed endpoint; it relies on vLLM's extensions to the OpenAI API, such as allowed_token_ids and logprob_token_ids. Its README explains how to set it up with Privatemode, and an AGENTS.md file tells coding agents what an implementation has to get right. The benchmark repository contains the methodology, the dataset specifications, the test harness, the aggregation, and every raw run as a download, so every number in this post can be reproduced without running anything again. If you know a setting that serves any of the three systems better, find a mistake, or want to add a dataset or another system, we welcome pull requests.

With Privatemode, these decisions are protected by confidential computing: your data stays encrypted in memory even during processing, and the client verifies the deployment's attestation report before sending anything. You can read more on our security page.

Build it on Privatemode

Create an account, then give this post to Claude Code, Codex, or the coding tool of your choice. Together with the GitHub repository, it has everything needed to build typed decisions into your own code, protected by confidential computing.

Local AWS cloud emulator for integration tests. Run your app with normal AWS clients, stay fully local, and use fakecloud SDKs when your tests need deeper visibility.

105 services. 7,508 operations. 248,557/248,557 Smithy variants pass — true 100% conformance.

Get Started · Why fakecloud

Why fakecloud?

fakecloud gives you a local AWS environment that behaves like infrastructure, not a mock. Your app uses the regular AWS SDK, CLI, and IaC tools. Unlike today's LocalStack Community setup, you do not need an account, auth token, or paid plan just to keep core development flows local.

The SDKs make that workflow nicer, not narrower. Start fakecloud, run the code you actually ship, inspect emails/messages/invocations after the fact, and force async AWS-style behavior to happen on demand.

Built for realistic local testing

Real APIs for your app, purpose-built tooling for your tests.

Test what your app really does

Use normal AWS clients against localhost, then verify the effects with fakecloud instead of stitching together polling, fixtures, or private test hooks.

First-party test clients

TypeScript, Python, Go, PHP, Java, and Rust SDKs wrap the /_fakecloud/* endpoints for resets, assertions, and manual control of async processors after your app has already used the normal AWS APIs.

105 AWS services

S3, SQS, SNS, EventBridge, EventBridge Pipes, EventBridge Scheduler, Lambda, EC2, DynamoDB, IAM, STS, Organizations, SSM, Secrets Manager, CloudWatch Logs, CloudWatch (Metrics & Alarms), KMS, CloudFormation, Cloud Control API, SES, Cognito User Pools, Cognito Identity, Kinesis, Firehose, RDS, RDS Data API, Aurora DSQL, Resource Groups, Resource Groups Tagging API, ElastiCache, MemoryDB, EKS, Cloud Map, Step Functions, API Gateway v1 (REST), API Gateway v2 (HTTP), Bedrock, Bedrock Agent, Bedrock Agent Runtime, Bedrock Runtime, ECR, ECS, Elastic Load Balancing v2, CloudFront, Route 53, WAF v2, Application Auto Scaling, Athena, ACM, and Glue.

Conformance tested

True 100% conformance across all 3,932 implemented API operations: 248,557/248,557 Smithy-model-generated test variants pass on every commit, backed by end-to-end tests against the official AWS SDKs.

Cross-service flows are wired up

30+ service-to-service integrations: S3 notifications, SNS fanout, EventBridge targets, DynamoDB Streams, CloudWatch Logs subscriptions, Cognito triggers, Step Functions task integrations, API Gateway → Lambda, and more — all exercise real service interactions.

No auth, no lock-in, no paid tier

Run a single binary or Docker image, use any dummy credentials, and keep the whole workflow local without signing in to someone else's platform.

SDKs for test assertions

Keep using the AWS SDK in your app. Use fakecloud SDKs when your tests need visibility and control on top of the emulator itself.

npm install fakecloud

Inspect SES emails, SNS messages, SQS queues, Lambda invocations, Cognito confirmation codes, and more.

View on npm

pip install fakecloud

Async and sync clients for pytest or app-level integration tests, with the same introspection and simulation coverage.

View on PyPI

go get github.com/faiscadev/fakecloud/sdks/go

Context-friendly helpers for resets, event history, SES inbound simulation, and the rest of the fakecloud test surface.

View on pkg.go.dev

cargo add fakecloud-sdk

Native Rust client for tests that need resets, state inspection, and direct access to the fakecloud introspection surface.

View on crates.io

Quick Start

Start fakecloud, point your AWS client at localhost, then assert what happened.

# Install (no Cargo or Docker needed)
$ curl -fsSL https://fakecloud.dev/install.sh | bash
# or: brew install fakecloud
$ fakecloud

# Your app uses the normal AWS SDK against localhost
$ aws --endpoint-url http://localhost:4566 sqs create-queue \
    --queue-name my-queue

# Your tests can inspect state through the fakecloud SDK
$ npm install fakecloud

How it compares

If you want a fully local workflow with real AWS APIs and no account requirement, fakecloud is built for that path. The SDKs are an extra testing advantage, not a replacement for that positioning.

	fakecloud	LocalStack Community
License	AGPL-3.0	Proprietary
Auth required	No	Yes (account + token)
Commercial use	Free	Paid plans only
Docker required	No (standalone binary)	Yes
Startup time	~300ms	~3s
Idle memory	~10 MiB	~150 MiB
Install size	~19 MB binary	~1 GB Docker image
AWS services	46 at true 100% conformance	30+ (paid-only past Community)
Test assertion SDKs	TypeScript, Python, Go, PHP, Java, Rust	Python, Java
Cognito User Pools	122 operations	Paid only
SES v2	110 operations	Paid only
SES inbound email	Real receipt rule actions	Stored but never executed
RDS	163 operations, PostgreSQL/MySQL/MariaDB via Docker, PostgreSQL aws_lambda extension	Paid only
ElastiCache	75 operations, Redis and Valkey via Docker	Paid only
MemoryDB	45 operations, full control plane	Paid only
EKS	Cluster control plane (batch 1)	Paid only
API Gateway v2	103 operations, HTTP APIs + developer portals + JWT/Lambda authorizers	Paid only
Bedrock	214 operations across 4 APIs (Bedrock + Bedrock Runtime + Bedrock Agent + Bedrock Agent Runtime)	Not available

Supported services

Every implemented API is covered by conformance and behavioral testing, with the service mix tuned for local development and integration tests.

• S3 — REST protocol · XML
• SQS — Query protocol · XML
• SNS — Query protocol · XML
• EventBridge — JSON protocol
• EventBridge Scheduler — JSON protocol
• SSM — JSON protocol
• DynamoDB — JSON protocol
• IAM — Query protocol
• STS — Query protocol
• Lambda — REST protocol · JSON
• Secrets Manager — JSON protocol
• CloudWatch Logs — JSON protocol
• KMS — JSON protocol
• CloudFormation — Query protocol
• SES — REST + Query protocol
• Cognito User Pools — JSON protocol
• Kinesis — JSON protocol
• RDS — Query protocol
• ElastiCache — Query protocol
• MemoryDB — JSON 1.1 protocol
• EKS — REST-JSON protocol
• Step Functions — JSON protocol
• API Gateway v1 (REST) — REST protocol · JSON
• API Gateway v2 — REST protocol · JSON
• Bedrock — REST protocol · JSON
• Bedrock Runtime — REST protocol · JSON
• Bedrock Agent — REST protocol · JSON
• Bedrock Agent Runtime — REST protocol · JSON
• ECR — REST protocol · JSON
• ECS — JSON protocol
• Elastic Load Balancing v2 — Query protocol
• CloudFront — REST protocol · XML
• Route 53 — REST protocol · XML
• ACM — JSON protocol
• ACM PCA — JSON protocol
• Application Auto Scaling — JSON protocol
• WAF v2 — JSON protocol
• Athena — JSON protocol
• Glue — JSON protocol
• Cognito Identity — JSON protocol
• CloudWatch (Metrics & Alarms) — Query protocol
• Firehose — JSON protocol
• EC2 — Query protocol · XML
• Organizations — JSON protocol
• Cloud Control API — JSON protocol
• EventBridge Pipes — REST protocol · JSON
• RDS Data API — REST protocol · JSON
• Aurora DSQL — REST protocol · JSON
• Resource Groups — REST protocol · JSON
• Resource Groups Tagging API — JSON protocol

How we fully migrated github.com away from CSS-in-JS.

The Primer Design System powers many of the experiences you see on GitHub today. From buttons to banners to breadcrumbs, these foundational components are required to be accessible, flexible, and performant across a wide variety of scenarios.

Back in 2023, the number of components on certain pages began to explode. This led to several performance-related challenges with our existing CSS-in-JS solution:

• Initial page loads took longer due to styles being initialized on the client
• Server-side rendering performance declined as style collection shifted from the client
• Updates to styles grew out of control as component count grew on a page

It became clear that the Primer team needed to address the issue at the source. We needed to find an alternative that would completely avoid the client and server costs that we were seeing with our current solution. Most importantly, any alternative we pick would need to work in a way that would avoid any breakage to GitHub during the migration.

Introducing CSS (Modules)

The Primer team found a solution that met all of our criteria: CSS Modules. This format would allow us to do one of our favorite things: write and use native CSS features, while still allowing some amount of the colocation and encapsulation that we had come to expect from CSS-in-JS.

With CSS Modules, styles would be authored in a CSS file alongside the JavaScript source for the component. It would also allow us to treat all class names as local by default, preventing some of the collisions and challenges that can come from global selectors. This format also removes the need for any client or server runtime behavior. Instead, styles would roll up into CSS stylesheets that were sent as part of the HTML for a page.

However, this solution was radically different from the CSS-in-JS solution we had at the time. This change would require an update to every Primer component and every component at GitHub authored using this technique. Thankfully, design systems are a perfect vehicle to deliver this kind of change at scale.

A gradual march towards CSS Modules

The situation for moving towards CSS Modules was clear. The Primer team would need to deliver updates to each of its components, moving them from CSS-in-JS to CSS Modules. At the same time, updates we made to these components could not break any usage in GitHub. Finally, the underlying technique we used for CSS-in-JS also had to continue working for any components in GitHub that were currently using it.

With all these constraints in place, we decided on an incremental migration strategy that would allow us to safely ship component updates without breaking the world. For each component, our plan was to:

• Add a new file that translated existing styles to CSS Modules
• Add the component to a feature flag that would toggle between the new and old styles
• Use existing visual regression tests to verify snapshots were identical between our CSS-in-JS solution and CSS Modules
• Gradually roll out the feature flag to our team, then to GitHub staff, and finally to all GitHub users to catch any issues along the way

This process created a strong feedback loop where issues were flagged early in the process as Primer continuously delivered these changes to GitHub. The use of feature flags allowed us to do this migration safely while giving us clear signals on the performance benefits of CSS Modules.

By December 2024, all components in Primer were migrated over to CSS Modules using this process. We saw performance wins across the board, in particular:

• 55% less time to server-side render a page
• 25% less time for components on a page to initialize

With clear performance wins from doing this work in Primer, we began to wonder if we could see similar performance wins by doing these conversions in other parts of GitHub. Similarly, how long until we could ultimately drop support for CSS-in-JS across the company?

Moving away from CSS-in-JS at GitHub

One of the trickiest parts about removing our CSS-in-JS solution from Primer was due to the usage of the sx prop. This prop was the way to style and customize components from Primer. Teams could provide an inline object to customize everything about the component. It represented the best and worst parts of CSS-in-JS:

• Excellent TypeScript support with integration with our Design Tokens
• Co-located with the component so that everything was in one place
• High runtime cost due to the dynamic nature of inline objects used for sx
• Difficulties scaling as the number of components using sx on a page grew

As a result, the first part of our journey to move away from CSS-in-JS was to reduce sx usage across GitHub. This would allow us to immediately improve performance similar to the wins we saw when migrating Primer components. It also set us up perfectly for removing CSS-in-JS entirely from the product.

The Duality of Primer

It’s important to note that, while the design system itself was officially off styled-components, a large part of the GitHub codebase itself wasn’t. With sx props having been the de facto styling standard at GitHub for years, we were looking at thousands of sx props that needed migration before we could even think of getting GitHub onto the sleek new @primer/react version which didn’t rely on styled-components.

So… how did we do this immense amount of work while increasing confidence and reducing risk? The answer: not all at once.

The original CSS migration was a little bit more nuanced than we led on: in addition to migrating the components to CSS modules, feature flagging to test in production and slowly rolling them out, we also created “wrapper” components in a transitive library we called @primer/styled-react. The whole purpose of this package was to allow sx usage into the newly migrated components. This way, the instances of the GitHub UI codebase that were using this prop could continue to consume them by importing the same component through @primer/styled-react, while we realized the performance gains of importing straight from @primer/react for the cases that didn’t.

Styled Box Zero

The next phase of the migration process was as follows:

• On a package-by-package basis:

  • Translate all sxusage into equivalent CSS modules files. This included cross referencing (See migrating to CSS variables)
  • Replace @primer/styled-react imports with @primer/react imports
  • Test in pre-production
  • Deploy

Curiously enough, while we were getting ready to undertake this massive effort, styled-components maintenance mode was announced, offering further confirmation that we were taking steps in the right direction.

The work kicked off April 2025 with a peak of ~7,760 sxprops to be migrated; we wouldn’t see it realized until May 2026. Initially, one of our great in-house developers, Ian Sanders, created a VS Code plugin that would assist with per-prop migration. A similar codemod was developed internally and utilized to migrate entire files in the GitHub codebase. The work, while requiring a bit of manual oversight and careful validation, was mostly automated. A rotation of 8 engineers migrated 6,419 props over the course of 6 months, observing Server-Side Rendering time performance gains ranging from 1% up to 22% in some pages.

In a different side of GitHub, Copilot’s capabilities were increasing exponentially. AI was getting smarter, more capable; we released Copilot coding agent and Copilot code review while this work was still underway.

By the time we picked this work back up, now April of 2026, the panorama was different; we were able to get down from 895 to 0 sxprops in the span of three weeks with a team of two engineers, relentless determination, and a whole lot of Copilot coding agents.

Battle of the themes

It was a big day: we had finally completed the sx migrations that were standing between us and full styled-components removal, years in the making… we can finally clean up these dependencies and move on to different, more exciting work, right? Wrong!

GitHub supports seven different themes, all of which offer a high contrast mode variation. All of it enabled through, you guessed it, styled-components. Before we can even think of removing these dependencies, we need to decouple our theming.

Now, this isn’t as huge a deal as it sounds. Our theming variables have always been defined in CSS through our @primer/css package, and we already planned forward for non-styled Theming when we migrated @primer/react in late 2025. It’s the JavaScript usage and utilities that are enabled by styled-components that we needed to remove. Once again, we got to work.

You know the drill by now: perform the migrations, roll it out slowly, feature flag everything. Two months and a few hiccups along the way later, we were all-systems go for dependency removal; we even feature flagged that. Better safe than sorry.

All’s well that ends well

GitHub has been running on 100% CSS modules as of June 2026. The safeguards we put in place enabled us to roll out significant architectural changes safely, stress test in production, catch errors, pivot and repair efficiently, ultimately allowing us to succeed in our goals and realize great performance gains along the way.

What looked at first like a CSS migration turned out to be a gradual re-platforming of how GitHub styles, themes, and ships UI at scale. By the end, we had not only removed sx, styled-components, and styled-system from dotcom, but completed it without breaking GitHub along the way. Enhancing the performance, user experience and delight of our products continues to be top of mind for all of us here at GitHub.

A Mastodon post by computer scientist David Chisnall has a very Unsung opener:

I have used vim since around 2000. I have written five books, a PhD thesis, a few dozen papers and over 150 articles with it. At this point, my higher brain functions are not engaged at all when I use a bunch of common vim commands, they just happen. Documents I wrote with anything else have random :w in the middle.

Chisnall goes on to talk about one specific vim feature:

Persistent undo is one of my favourite features of vim. […]

I don’t often need the persistent undo. But on the few occasions when I have needed it, it’s been invaluable: ooops, I deleted something from this file, maybe last week and one reboot ago, what was it? Undo until I find it, copy it, paste it into the current version. Or, a bit more commonly: I had this working, then I tidied it up ready to commit, now it isn’t working, what did I do?

Vim has kept this working across major version upgrades over a period of about 20 years. I don’t even think about it, it’s just part of Raskin’s First Law: A program may not harm a user’s data or, through inaction, allow a user’s data to come to harm. If vim or the computer crash, or if I close a file and come back to it six months later, my undo history is still there.

NeoVim is a fork of vim (in news for other reasons):

So I tried NeoVim when it was quite new. Vim that you are familiar with, but better? Great!

The first thing I noticed in NeoVim was that undo didn’t work. I tried opening the file in vim and undo didn’t work there either.

Neovim had changed the format of the undo files. It hadn’t upgraded the old one. It hadn’t used a different name for its undo files. It had just noticed the existence of a vim undo file, deleted it (losing all of the data in it) and replaced it with one that vim couldn’t read.

I raised an issue about this and was told that the persistent undo format was unstable and users should not rely on data being preserved in a feature explicitly called persistent undo. It had changed once and would probably change again.

And that ended my experience with NeoVim. The authors showed immediately that they absolutely could not be trusted with any of my data. Breaking persistent undo is something I could forgive as a bug, but the attitude that just because something is a persistent file on your filesystem that contains data that you might want is no reason for their program not to delete it meant they had no concept of a duty of care to their users.

I liked this post (which I quoted almost in its entirety), because it covers a few important things:

• I have never heard of the persistent undo like that, and it seems kind of amazing.
• People do remember when software loses their hard work or disrespects them.
• I can see how “It had changed once and would probably change again” can be such a powerful feeling.

I also loved it for the appearance of Raskin’s First Law. Jef Raskin, of Macintosh and Canon Cat fame, put together the three laws in his 2000 book The Humane Interface, and they go as follows:

• A computer shall not harm your work or, through inaction, allow your work to come to harm.
• A computer shall not waste your time or require you to do more work than is strictly necessary.
• An interface is humane if it is responsive to human needs and considerate of human frailties.

It was a very important and formative book for me to encounter as a young designer. I have no idea how these laws haven’t made it to Unsung before today.

the normalization of inexplicable failures

In a recent episode of President Curtis, the President struggles with opening a door on two separate occasions.

These doors don't work because there are obstructions in the way: a body initially, then roughly a billion dollars worth of gold.

In both instances, in response to the frustration, the character mutters "stupid thing sucks." This is not a reasonable model of doors! Doors should not "suck" inexplicably! I found these moments outrageously hilarious¹ but maybe my stupid brain just sucks.

Jev: Making more doors that suck

The Internet has been abuzz about Jev, an AI model developed by TypeSafe AI, which returns typed values with probability estimates. The important things about Jev are, as far as I can tell:

• it is fast and cheap,
• you can build on it quickly,
• it is fast, and
• it is cheap.

I'm not particularly good at understanding what technology will get adopted.

I still don't understand² Slack.

But this product is even more confusing to me. This isn't "getting an FTP account, mounting it locally with curlftpfs, and then using SVN or CVS on the mounted filesystem" -- you still have to do the hard part. To know if Jev is working, you have to build evals and a ground-truth pipeline. If you have evals and a ground-truth pipeline, you're already most of the way to fine-tuning your own solution.

Wait.

Do you still have to do the hard part?

Maybe my problem is expecting products to work.

Nobody buying this is running evals. They're just handing opaque questions to Jev and getting opaque responses. Charitably, this allows them to check the "AI-powered" box and ship before Friday, and when this breaks downstream logic, they can always shrug and say "well, AI makes mistakes."

Error budgets? Failure modes? Test sets? All of those can be handled later. The user can discover the failure rate! You've already shipped!

False Confidence

"Oh," the strawman responding to my post responds, "you haven't considered the fact that Jev gives you confidence scores!"

What are you going to do with those?

For you to do something reasonable with confidence scores you need to have both an understanding of the calibration of those confidence scores and also a model for the costs of the uncertainty.

On the calibration side: Jev's topline ad copy is mostly about how well they score on various benchmarks, but not about how calibrated their confidence scores are. There's a cookbook about using confidence scores to go up a tree of classification but that's fundamentally not about how good the confidence scores are.

On the modeling side: nobody wants to think about this. It always ends up being "Ehh, let's just accept any response over a threshold. 0.9 sounds about right? What's for lunch?" Their documentation itself makes up a 0.5 threshold for "do nothing" and a 0.9 threshold for "do high-risk actions" [archive] with an aside info box stating "the correct threshold values depend on your domain and the performance of the model for your use case."

At best, people use confidence scores in a cargo cult manner. At worst, people use them as an excuse for why the API call failed. The model was only 73% confident! That means my error budget is 27%!

Accountability

When a button breaks on a website, I have a model about what should have happened. Somewhere a contract got broken. My DNS is broken. Somebody shipped some slop that has JavaScript syntax errors along only a certain path. A handler threw that wasn't expected to throw. I might not have access to debug just an HTTP status 500, but I expect there to be somebody whose job is to understand why the endpoint is 500ing. The ownership is well-defined albeit opaque³.

For many users, however, the actual experience is roughly just "stupid thing sucks." Software already feels capricious; more failures just change the rate of frustration. It seems like not much of a loss to remove the possibility of following a failure to a concrete cause. Sometimes things just suck.

This leads to a normalization of inexplicability.

My fear is not that more things will fail when things are accelerated by LLM-driven development. They will. They have. Such is part of the price of building things in a novel manner.

My fear is that "sometimes it just sucks" is going to be more and more the accepted endpoint of investigations. This is sad because LLM-accelerated development can indeed help us solve some of these issues. There are plenty of automated QA workflows that aren't written because of lack of engineering time. The very eval that would get you most of the way to replacing (or even justifying the use of) Jev can be a few prompts away.

The tragedy of software engineering today is that we are actively engineering systems where neither the user nor the builder seems to have any interest in checking whether or not there's a body behind the door.

We just shrug and conclude: stupid thing sucks.

---

¹This reminds me of a saying that I find similarly hilarious: "sometimes you get the elevator, sometimes you get the shaft." This is also not a reasonable model of elevators!!

²The lock-in network effect makes sense to me but I'm still bewildered as to how people standardized on a product that does not even reliably deliver messages. I have seen messages dropped on free, paid, and enterprise instances that only show up weeks later.

³Well, maybe "well-defined" is optimistic. After Bill Gates famously failed to download Movie Maker, everybody agreed that it was presumably somebody's problem, just not necessarily theirs. Ideally we can get even this level of accountability without the customer being Bill Gates.

Reverse-engineering the vintage Intel 8087's tangent algorithm: more than CORDIC

I hope you're not tired of the 8087, because I have another article about Intel's floating-point chip.1 In 1980, Intel introduced the 8087, making floating-point operations much faster in the IBM PC and other systems. In this article, I look at the algorithm behind the chip's tangent instruction. One popular approach for trigonometric functions is an algorithm called CORDIC. Another approach is a polynomial approximation. The 8087 combined the two to obtain both high accuracy and high performance. The 8087 provided an enormous speedup over the 8086 microprocessor, computing a tangent in 90 microseconds rather than 13,000 microseconds.2

By examining the circuitry and microcode of the 8087, I can explain the algorithm behind the tangent instruction, called FPTAN. To explore the 8087's circuitry, I popped the lid off a chip with a chisel and created a high-resolution image with a microscope. The microcode ROM is the large rectangular region in the center of the die, holding the 1648 micro-instructions that control the chip. The bottom half of the chip (red box) is the datapath, the circuitry that performs floating-point calculations on 80-bit values.3

A close-up of the 8087's datapath, showing functional blocks that are used by FPTAN. Click this image (or any other) for a larger version.

Zooming in on the datapath shows the relevant functional units. The exponent ROM holds fixed exponent values that the algorithms need. The constant ROM holds constants, including the constants used by the CORDIC algorithm. The shifter is a large component; it shifts a 64-bit value left or right by arbitrary amounts. The adder is the heart of the 8087's calculations; as well as providing addition and subtraction, it is used in a loop for multiplication, division, and square roots. The B register holds one input to the adder, while multiple sources can provide the other input. The sum register holds the adder's output. The eight stack registers and the temporary registers hold floating-point numbers. Finally, the shift register holds 16 status bits for the CORDIC calculations.

The CORDIC algorithm

CORDIC is a clever algorithm for quickly computing transcendental functions with simple hardware: it uses shift and add instructions along with table lookups, but doesn't need multiplication or division. This algorithm dates back to 1956, when it was developed for the B-58 Hustler, the first bomber capable of flying at Mach 2. The aircraft had an analog navigation computer, but analog components provided limited accuracy. Engineer Jack Volder was given the task of designing a digital computer to replace the analog computer.7 One key problem was that an analog computer can easily generate sines and cosines with an electromechanical device called a resolver. But trigonometric functions are difficult to produce digitally, especially with the slow transistors of that era.

A Convair B-58A Hustler, on display in San Antonio, TX (details).

Jack Volder came up with a fast way to calculate trigonometric functions with simple hardware. He called the algorithm—and the computer that implemented it—CORDIC: "COordinate Rotation DIgital Computer". CORDIC converts an angle to a vector, where the vector's coordinates provide the necessary trig functions. The trick is to break down the angle into a sequence of special angles, angles that make vector rotation easy. These special angles are precomputed and stored in a table, so the CORDIC calculation can be performed quickly, even on 1950s hardware. Each CORDIC iteration provides an additional bit of accuracy, so the algorithm converges rapidly. CORDIC became popular, including in scientific calculators, which used decimal CORDIC instead of binary.

Some trigonometry

I'll try to keep the math to a minimum, but in this section I'll give a quick explanation of how CORDIC works. The diagram below reviews how trig functions are related to the coordinates of a point. Suppose you have an angle θ; it specifies a point (X, Y) on the unit circle. The basic formulas are X=cos θ, Y=sin θ, and Y/X = tan θ. Thus, if you can determine the coordinate (X, Y), then you can determine the value of the trig functions. If the point is not on the unit circle, e.g. (X', Y'), then you can still easily determine tan θ. (Spoiler: this is what the 8087 does.) However, sin θ and cos θ become messy.4

The relationship between an angle, the X and Y coordinates, and the trig functions.

If you've done any computer graphics, you've probably seen how a rotation matrix can rotate a point by an angle. (If you're not familiar with rotation matrices, you can read about them here or just trust that it works.) Multiplying a point (X, Y) by the rotation matrix yields the new point (X', Y') as shown below.

A point can be rotated by using a rotation matrix.

Unfortunately, since the rotation matrix (1, below) requires sin and cos, it doesn't seem like it helps solve our problem. However, we can divide the matrix by cos θ; this seems even less helpful since now the matrix (2) needs tan, which is what we want to evaluate. (Moreover, the vector's length will grow.) But the key to CORDIC is to use special angles, αn = arctan(2-n). When we substitute one of these special angles into the matrix, we get matrix (3), which is easy to evaluate in hardware: multiplying by a power of 2 can be done by shifting the bits.

Simplifying the rotation matrix.

Applying matrix (3) to the point (X,Y) gives the equations (4). These are key equations for the CORDIC process. The important thing is that these equations are fast and easy to compute in machine language or hardware, as the only operations are addition, subtraction, and binary shifting.

With that background, we can see how CORDIC works. First, we break down the desired input angle into a combination of special angles that adds up to the desired angle.5 Then we apply the rotation formula above for each special angle, starting with the unit vector (1, 0). The result is a point (X, Y) at the desired angle, and then the desired tangent is simply Y/X.6 Since the table of special angles is precomputed, the arctan operations don't slow down the process. As an aside, after the first few terms, the special angles approach 2-n, so they shrink by roughly a factor of 2 at each step.

To summarize, the CORDIC algorithm consists of looping through a table of stored angles. If the stored angle is less than the desired angle, the stored angle is subtracted from the desired angle to yield a new desired angle and the equations above (shifts, add, and subtract) are applied to yield a new vector. At the end, the tangent of the original angle is given by Y/X.

The rational polynomial approximation

The accuracy of CORDIC depends on the number of terms that are used. With 16 terms, the accuracy is approximately 2-16, or 16 bits of accuracy. To get 64 bits of accuracy would require calculating 64 terms (and a table of 64 special angles). To get an answer faster, the 8087 uses 16 bits of CORDIC and uses another algorithm for the remaining angle. (The remaining angle is the gap between the sum of special CORDIC angles and the desired angle, so it is very small, around 2-16.)

For the remaining angle, the 8087 uses a Padé approximant, which is the ratio of two polynomials. There's a whole family of Padé approximations, depending on the order of the polynomials. The 8087 uses a simple formula: 3x/(3-x2).8 Although this approximation is simple, it is very accurate for small values; its error is proportional to x4. Since x<2-16, the error will be less than 2-64, meeting the 64-bit accuracy requirement for the 8087. Moreover, the 8087 doesn't need to perform the division in the rational polynomial since FPTAN returns a separate numerator and denominator. Thus, the division is "free".

The tangent function (red), rational approximation (blue), and Taylor series (green). Disclaimer: The Taylor series isn't as bad as it appears, since the relevant range is very close to 0. Graph generated with Desmos.

If you've studied calculus, you might think that a Taylor series polynomial is the way to go, but the ratio of two polynomials is better. (One reason is that tangent blows up to infinity at π/2. A polynomial won't blow up, but the ratio of polynomials can, so it fits the tangent function better.) The graph above compares the tangent function (red), the rational approximation (blue), and the third-order Taylor series (green).

Putting the pieces together: the 8087 algorithm

The 8087's tangent algorithm has three parts: determining the CORDIC decision bits (called pseudo-division), computing the rational approximation, and applying the rotation equations based on the CORDIC decision bits (called pseudo-multiplication).9

In more detail, the first step determines which special angles to add to approximate the input angle. Each special angle is compared to the remaining input angle and subtracted if it is smaller. If the angle is subtracted, a 1 is recorded; otherwise, a 0 is recorded. Since this process is similar to how long division subtracts (or doesn't subtract) successive shifted versions of the divisor, generating 1s or 0s for the quotient, the process is known as pseudo-division. Note that the rotations are not applied in this step. Instead, this step decides which rotations to apply later.

The diagram below shows this process applied to the input angle 0.95 radians.10 The process generates the sequence of bits [1,0,0,1,0,1,0,1,0,0,1,0,0,1,1,1], where the leftmost bit indicates arctan(20) and so forth. The angle is reduced by roughly a factor of 2 at each step, so the residual angle is very small.

In the first phase of the CORDIC algorithm—pseudo-division—the input angle is reduced by special angles, leaving a residual angle at the end. ("rad" is radians, not the unit of radiation.

Next, the tangent of the remaining angle is calculated with the rational approximation function, 3x/(3-x2). The result is used as the initial vector for the next step. The division is not performed here; instead, the numerator becomes Y in the initial vector and the denominator becomes X. Thus, the expensive division step is avoided, since it happens implicitly in the answer. Multiplying by 3 is easy (shift left and add), so the only expensive operation at this step is squaring the angle, which requires a full 64-bit multiplication.

The final step applies each CORDIC rotation if the corresponding decision bit from the first step is 1. Since this is somewhat analogous to binary multiplication, which adds the multiplicand at each step if the multiplier bit is 1, this step is known as pseudo-multiplication. As described earlier, each rotation is computed with shifts, addition, and subtraction, so each rotation is inexpensive. Each rotation in this step corresponds to an angle reduction in the first step. The rotations are applied in reverse order—the smallest one first—to reduce rounding error. To accomplish this, the decision bits are stored in a 16-bit shift register in the first step, and shifted out in opposite order in this step.

In the last phase of the CORDIC algorithm—pseudo-multiplication—a vector is rotated multiple times by applying shifts and adds. The final vector provides the tangent.

In the last phase of the CORDIC algorithm—pseudo-multiplication—a vector is rotated multiple times by applying shifts and adds. The final vector provides the tangent.

The diagram above shows the process applied to the input 0.95. The initial vector (green) comes from the rational approximation function; it is very close to (3, 0), but slightly rotated due to the tiny residual angle. Each rotation step generates a new vector that matches the angle from the corresponding step in the first part; I show two of the rotation matrices. Note that the vectors diverge from the circle—a consequence of dividing each matrix by cos θ—but this doesn't affect the tangent.

The FPTAN instruction is somewhat peculiar as it doesn't return the tangent directly. Instead, it returns the Partial Tangent: the two coordinate values (X and Y) that can be divided to produce the tangent. One might wonder why the chip didn't do the division automatically. The motivation was that division was slow back then, and omitting the division allowed for optimizations in some cases.

Relevant hardware details of the 8087

In this section, I'll explain some features of the 8087 that are important for the microcode implementation of FPTAN.

The 8087 supports a variety of data types, but internally, everything is stored as an 80-bit floating-point number called a "temporary real". A number has three parts: a sign bit, a 15-bit exponent, and a 64-bit significand (the fractional part), In most cases, a floating-point number is represented by sign × significand × 2exponent. The advantage of floating-point numbers is that the exponent allows them to cover a huge range, from very small to very large. The significand is a 64-bit binary number of the form 1.bbb…: a leading 1, followed by the binary point (the binary equivalent of the decimal point) and the rest of the bits.11 One important detail is that the exponent is stored with a "bias" of 16383 added to it. Thus, the stored exponent is always positive, even if the real exponent is negative. For instance, an exponent of 1 is stored as 0x4000, and an exponent of -16 is stored as 0x3fef.

The 80-bit temporary real format and the register formats. The dot indicates the binary point, analogous to the decimal point.

The 80-bit temporary real format and the register formats. The dot indicates the binary point, analogous to the decimal point.

To use the 8087, a programmer stores values in its eight internal registers, organized as a stack. Each register holds an 80-bit floating-point number. To optimize performance, each value in the register stack also has an associated "tag" value: zero, valid, special, or empty. A value of zero is tagged as zero. A "normal" floating-point value is tagged as valid. If the value is infinity, Not a Number (NaN), or a denormalized value, then it is tagged as special. Finally, an empty tag indicates that a register does not hold a value; this allows detection of stack underflow (reading an empty register) or stack overflow (storing to a non-empty register).

The 8087 also has temporary registers that it uses internally: tmpA, tmpB, and tmpC. (These registers are heavily used for the tangent calculation.) tmpA and tmpB are 80-bit registers, along with two tag bits. However, tmpC is different: it doesn't have a sign, exponent, or tag. Moreover, the significand is 68 bits; tmpC has an additional bit to prevent overflow and three additional low-order bits for rounding, known as guard, round, and sticky.

In the microcode, each 16-bit micro-instruction performs one low-level operation. Many of the operations move data from one register to another. Other operations provide conditional jumps, subroutine calls, and returns. A bit shift operation takes two micro-instructions: the first moves a value to the shifter and configures the direction and amount of shift. The second micro-instruction (which may be a distance after the first) moves the result from the shifter to a destination.

Similarly, addition takes two micro-instructions. The first micro-instruction specifies the first argument to add and selects various options such as carry and rounding. (For subtraction, an option complements the second argument.) The second argument always comes from the B register, which confusingly is unrelated to the tmpB register. The second micro-instruction moves the result from the sum register to a destination.

Implementation of the tangent algorithm in microcode

In this section, I'll discuss some highlights of the tangent microcode. The listing below shows the 8087's microcode, along with my comments. Each line indicates a 16-bit micro-instruction. The control flow is a bit complicated, so I've put a flowchart in the footnotes.12

FPTAN:
#1039 st(0) -> tmpA        store argument in tmpA
#1040 stackPtr--           check if room on stack to push result
#1041 stack overflow?
#1042 jmp #1022 if tmp empty/special/overflow/div exit if bad argument or stack overflow
#1043 jmp #1045 if tmpA:tag ZERO is argument 0?
#1044 except:precision     precision exception except for 0
#1045 expconst 0x3ff0      Test if exponent >= -16
#1046 adder: tmpA:exp + 0 cin=1 argument exponent + 1
#1047 expConst -> Breg
#1048 adder: sumreg:frac - Breg cin=1 subtract -15
#1049 jmp #1061 if adder pos CORDIC if exponent >= -16
#1050 expconst 0x3fff      non-CORDIC path:
#1051 tmpA:exp -> Breg     check original exponent
#1052 expConst -> tmpB:frac against exponent 0
#1053 adder: tmpB:frac - Breg cin=1 subtract
#1054 sumreg:frac -> expConv 3fff - exp (i.e. -exp unbiased)
#1055 sumreg:frac -> shiftcount
#1056 jmp #1082 if exponent[6:14] != 0 if exp > -64 goto rational approximation
#1057 high bit -> tmpB:frac otherwise return original argument
#1058 expConst -> tmpB:sign,exp push 1 for X (denom): result is orig angle
#1059 tmpB -> st(0)
#1060 RNI
#1061 expconst 0x0010      CORDIC path
#1062 sumreg:frac -> loopcounter loopcounter = exp + 16
#1063 tmpA:frac -> tmpC    tmpC = original angle
#1064 trig const -> Breg/rnd
#1065 adder: tmpC - Breg cin=1, roundmode subtract first CORDIC angle
#1066 adder sign -> cordic, shl save bit  in shift register
#1067 jmp #1079 if not adder pos
#1068 sumreg:frac,rnd -> tmpC
#1069 adder: tmpA:exp + 0 cin=1 increment exp if first angle used
#1070 sumreg:frac -> tmpA:exp
#1071 jmp #1079
#1072 shift tmpC L 0 bytes, 1 bits top of CORDIC scan loop
#1073 shift L -> tmpC      shift angle left
#1074 trig const -> Breg/rnd
#1075 adder: tmpC - Breg cin=1, roundmode subtract CORDIC angle
#1076 adder sign -> cordic, shl save decision bit in shift register
#1077 jmp #1079 if not adder pos
#1078 sumreg:frac,rnd -> tmpC
#1079 jmp #1072 if not const latch zero bottom of CORDIC scan loop
#1080 tmpC -> tmpA:frac    update tmpA, tmpB
#1081 expConst -> shiftcount 16 -> shiftcount (otherwise based on exponent)
#1082 tmpA:frac -> tmpB:frac Padé approximation
#1083 tmpA:frac -> Breg    tmpA = ang
#1084 call SQUARE          square the angle
#1085 shift sumreg:frac R count byte bit
#1086 shift R -> sumreg:frac shift twice to scale square
#1087 shift sumreg:frac R count byte bit
#1088 shift R -> Breg      Breg = ang^2
#1089 1.1 -> tmpB:frac     3 (with exp 1)
#1090 adder: tmpB:frac - Breg cin=0
#1091 sumreg:frac -> tmpB:frac tmpB (X) = 3-ang^2
#1092 shift tmpA:frac R 0 bytes, 1 bits
#1093 shift R -> Breg/rnd
#1094 adder: tmpA:frac + Breg cin=0, roundmode ang + ang/2
#1095 sumreg:sign,frac,rnd -> tmpC tmpC (Y) = 3×ang with appropriate exponent
#1096 expconst 0x3ff0      exponent -15
#1097 expConst -> Breg
#1098 adder: tmpA:exp - Breg cin=1
#1099 jmp #1118 if not adder pos skip if exponent too small
#1100 shift tmpC R 0 bytes, 1 bits CORDIC pseudo-multiplication path
#1101 #15 -> loopcounter
#1102 shift R -> tmpC
#1103 jmp #1113 if not cordic[0] CORDIC: if saved decision bit...
#1104 shift tmpC R loopcount send tmpC (Y) to shifter
#1105 tmpC -> Breg/rnd     send tmpC to adder
#1106 adder: tmpB:frac + Breg cin=0, roundmode tmpB (X) not shifted because tmpC scaled
#1107 sumreg:sign,frac,rnd -> tmpC tmpC = tmpC + tmpB (implicit >>n)
#1108 shift R -> sumreg:frac,rnd shifted old tmpC to sumreg
#1109 shift sumreg:frac,rnd R loopcount shift right again because tmpC scaled
#1110 shift R -> Breg/rnd  and send to B reg
#1111 adder: tmpB:frac - Breg cin=1, roundmode
#1112 sumreg:frac,rnd -> tmpB:frac tmpB = tmpB - tmpC>>n
#1113 cordic shr           shift out shift register bit
#1114 shift tmpC R 0 bytes, 1 bits start shift of tmpC (Y)
#1115 jmp #1118 if cordic==0 done if shift register empty
#1116 shift R -> tmpC      shift tmpC (Y) right
#1117 jmp #1103 if not const latch zero done if counter at zero
#1118 expconst 0x3fff      CORDIC done
#1119 tmpB:frac -> sumreg:frac Test top bit of tmpB (X)
#1120 jmp #1124 if reg bit 63 If set, use exponent 0
#1121 expconst 0x3ffe      Otherwise, use exponent -1 and
#1122 shift sumreg:frac L 0 bytes, 1 bits shift left one bit to normalize
#1123 shift L -> tmpB:frac
#1124 expConst -> tmpB:sign,exp store exponent in tmpB
#1125 tmpC -> tmpA:frac    store tmpC (Y) in tmpA
#1126 tmpC -> sumreg:frac,sign
#1127 jmp #1132 if not sumreg[64] test overflow bit
#1128 shift tmpC R 0 bytes, 1 bits if set, normalize by shifting right
#1129 shift R -> tmpA:frac
#1130 adder: tmpA:exp + 0 cin=1 and increment exponent by 1
#1131 sumreg:frac -> tmpA:exp
#1132 tmpB -> st(0)        tmpB to top of stack: X (denom)
#1133 stackPtr++
#1134 tmpA -> st(0)        tmpA to st(1): Y (numerator)
#1135 stackPtr--
#1136 RNI                  End of FPTAN

The FPTAN microcode starts at address #1039 (decimal). The microcode starts by moving the argument from the top of the stack to the tmpA register. If the top value is empty, this indicates a stack underflow. If the next element on the stack (the position that will hold the result) is not empty, this indicates a stack overflow. In either case, the microcode indicates an "invalid" exception and ends the instruction.

One peculiar feature of the 8087 is that it flags a result with a "precision" exception if the result is not exact. (This happens very frequently, even for, say, 1/10.) The only tangent that the 8087 can compute precisely is tan(0), so all other inputs result in a precision exception (#1044).

Next, the argument's exponent is tested, splitting the execution into three paths. If the exponent is -64 or less, the argument is so small that the tangent equals the argument, within the accuracy of the system.13 In this case (#1057), the original argument is returned unchanged (with the denominator 1 pushed), and the code ends. The second case is if the argument's exponent is -17 or less. In this case, the CORDIC step is skipped, and the routine jumps directly to the rational approximation (#1082).

CORDIC pseudo-division

The most interesting case is the CORDIC path (#1061), taken if the exponent is between -1 and -16. Conceptually, the code performs 16 CORDIC steps, using 16 stored angles. However, the code is optimized, skipping the large angles for smaller inputs. Specifically, the loop counter is initialized based on the exponent of the input angle (#1062). The CORDIC pseudo-division loop (#1061-#1080) tests the angles, subtracting ones that aren't too big.14

The decision results are recorded in a 16-bit shift register, located on the far right side of the die. Presumably, this is where the layout had some unused space. Since the shift register is loaded and unloaded serially, it doesn't need access to the internal fraction bus, just two lines to shift the bits in and out, so the shift register could be located in otherwise unused space.

The code is optimized to use 64-bit integer arithmetic rather than floating-point arithmetic. This makes it tricky to understand the code since values must be viewed more as fixed-point numbers with implicit exponents. These exponents are not stored anywhere, but can be determined by analyzing the algorithm. Even the angle constants in the ROM do not have explicit exponents.15 Moreover, the implicit exponents change for each step through the loop to preserve accuracy. Roughly speaking, each cycle of the loop reduces the values by a factor of 2, and the angle constants shrink accordingly. If the values had a fixed exponent, they would end up with 16 leading zeros, wasting precision. But by scaling the values each cycle (with a left shift), the numbers continue to use the full 64 bits. In other words, the hardware is performing fast integer arithmetic, but mathematically you can think of it as fixed-point with exponents that don't physically exist in the chip. See the table in the footnotes16 for details on how the implicit exponents change through the loop.

Rational polynomial approximation

After the CORDIC pseudo-division loop, the microcode calculates the rational polynomial approximation (#1082). The small-angle path rejoins the execution flow here. The microcode to calculate the polynomial approximation has a few interesting features. The most time-consuming part is the SQUARE routine, which multiplies a fixed-point number by itself. Multiplication is complicated, so I'll give the details in a later post. But in brief, the 8087 uses Booth's Algorithm to multiply by two bits at a time (radix 4), so it is twice as fast as regular binary multiplication. The loop to perform the shifts and adds is implemented in hardware, rather than microcode. That is, one microcode instruction performs 32 additions: the hardware tests the bits, does the appropriate add or subtract, updates the loop counter, and loops. For performance, the shifting is done with specialized shifters, not the 8087's general-purpose shifter.

Except for the square, the polynomials are calculated with additions rather than multiplications. The constant 3 doesn't come from the constant ROM, but from special-purpose transistors that set the top two bits of the significand; this is usually used to supply an NaN. (If the implicit exponent is 1, this corresponds to 3.) Subtracting the square yields the numerator. For the denominator, the angle is shifted right (i.e. divided by 2) and added to itself. This performs a multiplication by 3 (technically by 1.5, but increasing the implicit exponent to 1 turns this into 3). It would be expensive to divide the numerator by the denominator; instead, the numerator and denominator are used as the initial Y and X values for the following CORDIC steps.

CORDIC pseudo-multiplication

Next is the CORDIC loop where the rotations are applied to the vector (#1100). The rotations are applied in the reverse order from how they were computed in the first step: the smallest rotations are applied first to preserve accuracy. Bits are shifted out of the shift register to indicate if the rotation should be applied or not. As soon as the shift register is all zeros, the loop stops. In other words, smaller angles go through the loop just a few times, not the full 16 times.

A close-up of the microcode ROM under the microscope. A transistor is formed where a vertical polysilicon line crosses doped silicon (pink). The 8087's ROM is unusual: it uses four transistor sizes, so it stores two bits per transistor, twice the density of a regular ROM.

A close-up of the microcode ROM under the microscope. A transistor is formed where a vertical polysilicon line crosses doped silicon (pink). The 8087's ROM is unusual: it uses four transistor sizes, so it stores two bits per transistor, twice the density of a regular ROM.

The rotations are applied by shifts and adds (or subtracts), but keeping track of the implicit exponents is a bit tricky. Since the vector starts almost horizontal, the X value is approximately 3, and the Y value is around 2-16. As the vector rotates, X remains roughly 3, but Y potentially increases by a factor of 2 each time. To provide the maximum accuracy for each number, the exponent for X (tmpB) is 1, while the exponent for Y (tmpC) starts off at -14 and increases by 1 each loop as Y is shifted right. (Remember that these exponents are not stored anywhere but are implicit.) Since the two registers have different (implicit) exponents, the values must be shifted before adding. The shift amounts aren't intuitive; the table16 in the footnotes may help clarify.

After the CORDIC loop, the final part of the code (#1118) normalizes the X and Y values, creating the floating-point values that are returned from the instruction. Recall that these aren't "real" floating-point values at this point, so they may need adjustment. Specifically, if X doesn't have a leading 1, it is shifted left one position, while if Y has two leading digits, it is shifted right one position. In either case, the exponent is adjusted accordingly. The X and Y values are put on the stack (#1132) to complete the instruction.

Conclusions

The FPTAN instruction is fairly slow as 8087 instructions go, due to its complexity. The documentation says that it takes typically 450 clock cycles. (The time has a large range; depending on the value, it can take 30 to 540 clock cycles.) For the value I examined (0.95), 33% of the time is in the CORDIC pseudo-division, 15% in the rational polynomial (mostly the squaring operation), 47% in the CORDIC pseudo-multiplication, and 5% overhead.

CORDIC is a popular way to compute trig functions, but there are alternatives such as polynomials. The 8087 is unusual because it combines CORDIC and a rational polynomial. For the Pentium, Intel moved from CORDIC to polynomial approximations; the Pentium's fast multiplication circuitry made polynomials practical. Nowadays, Intel has libraries such as MKL (Math Kernel Library) and Short Vector Math Library (SVML) that provide highly optimized implementations tailored to Intel hardware. These libraries are said to use polynomial approximations, using parallel instructions (SIMD) for performance, rather than specialized hardware. With these libraries, x87 operations and 80-bit floats are mostly obsolete. Nonetheless, I hope you've enjoyed this look at an original 8087 algorithm.

I plan to continue reverse-engineering the 8087 microcode; for updates, follow me on Bluesky (@righto.com), Mastodon (@[email protected]), or RSS. I've been working on this with the members of the "Opcode Collective", especially Smartest Blob and Gloriouscow, who converted the ROM images to microcode data and extensively analyzed the contents. See the 8087 repository on GitHub for more. AI statement: Despite the presence of the em dash, no AI was used in the writing of this article (details). The Opcode Collective used an ML algorithm to classify each ROM cell to extract the microcode.

Notes and references

1. If you want more on the 8087, here's a list of my previous 8087 posts, newest to oldest:
   Microcode in Intel's 8087 floating-point chip: the scale instruction
   The adder at the heart of Intel's 8087 floating-point chip
   Microcode inside the Intel 8087 floating-point chip: register exchange
   Instruction decoding in the Intel 8087 floating-point chip
   Conditions in the Intel 8087 floating-point chip's microcode
   The stack circuitry of the Intel 8087 floating point chip, reverse-engineered
   Die analysis of the 8087 math coprocessor's fast bit shifter
   Extracting ROM constants from the 8087 math coprocessor's die
   Two bits per transistor: high-density ROM in Intel's 8087 floating point chip
   Inside the die of Intel's 8087 coprocessor chip, root of modern floating point

2. Intel's documentation provides a performance comparison of the 8087 versus floating-point operations emulated on the 8086. The speedup varies depending on the instruction, but is typically almost a factor of 100. This speedup made the 8087 chip highly desirable for numerical applications such as CAD and spreadsheets.

3. Internally, the 8087 uses 80 bits to hold floating-point values: one bit for the sign, 15 bits for the exponent, and 64 bits for the significand. However, the 8087 uses three extra low-order bits for rounding, called Guard, Round, and Sticky. These bits ensure that a value is always rounded in the right direction. Some parts of the datapath have additional bits for sign or overflow: the shifter is 68 bits wide, and the adder is 69 bits wide. For the most part, I'll ignore these extra bits and refer to the datapath as 64 bits wide.

4. To compute sin θ from the 8087's X and Y values, the typical formula is sin θ = Y/sqrt(X2+Y2); cos θ is similar.

5. The special angles (in degrees) are approximately 45°, 26.6°, 14°, 7.1°, 3.6°, 1.8°, 0.9°, … You might wonder if adding up the special angles will always converge to the desired angle θ. A proof can be found here. Note that arctan(2-n) rapidly approaches 2-n (in radians), so it's kind of like a binary fraction expression, cutting the gap approximately in half each time.

6. The CORDIC implementation in the 8087 is somewhat different from the typical use of CORDIC, which finds sin and cos. In this approach, all the special angles are used, but an angle is either added or subtracted at each step, resulting in convergence to the desired angle. Recall that since the matrix is scaled by 1/sin αi at each step, the ending vector is no longer a unit vector. As a result, cos and sin don't match the final vector (X, Y).

   The trick is that if all the special angles are used, the overall scaling factor ends up being the same, simply the product of all the cos αi factors. (It doesn't matter if the rotation is positive or negative because cos αi = cos -αi.) Thus, the overall scaling factor can be precomputed and the initial unit vector can be prescaled by this amount. The result is that the final vector (X, Y) yields (cos θ, sin θ) directly, which is what we want.

   The key point is that there are two CORDIC approaches: using positive and negative angles, or using or omitting angles. (The first case multiplies each angle by +1 or -1, while the second case multiplies each angle by 1 or 0.) Most CORDIC implementations use the first approach, while the 8087 uses the second approach.

7. Jack Volder, the creator of CORDIC, wrote an article, The Birth of CORDIC, that describes the history in detail, as well as including photos of the B-58 navigation computer. Volder's original articles from 1959 are The CORDIC Computing Technique and The CORDIC Trigonometric Computing Technique.

   

   The CORDIC-II digital computer system. Photo by General Dynamics, from "The Birth of CORDIC".

   The photo above shows the CORDIC II computer that was used on the B-58 aircraft. It was a 30-bit serial computer, running at 198.4 kHz and weighing a hefty 134 pounds. It was implemented with diode-transistor logic (DTL) gates, using 2498 transistors and 4265 diodes, and had a drum memory (visible at the top) for storage, holding the equivalent of 39 kilobytes. Interestingly, the CORDIC operations were implemented as basic instructions in the instruction set, rather than subroutines; a CORDIC instruction took 5 ms, the same speed as a multiplication or division.

8. This Padé approximation is called the [1,2] approximation since the numerator is a first-order polynomial and the denominator is a second-order polynomial. Padé approximations can be derived from the continued fraction expansion of tan. See Constructing Padé Approximates.

   By the way, if you've taken calculus, you might think that a Taylor series is a good way to approximate a function. Unfortunately, a Taylor series is bad at approximating a function over an interval, rather than a point. You're much better off using a polynomial that minimizes the maximum error over the interval, as in the Pentium.

9. The 8087's algorithm is described in Implementation of transcendental functions on a numerics processor by Rafi Nave, who led the hardware design of the 8087.

10. The valid argument range for FPTAN is documented as 0<θ<π/4, so the demonstration value that I use, 0.95, is outside the documented range. I find the range puzzling, since the first CORDIC constant is arctan(1), π/4, which will only be useful for inputs above π/4. Thus, the 8087 uses a scarce ROM entry and an extra loop cycle to handle values outside the permitted range. I have two hypotheses about this. Perhaps the 8087 was implemented for a larger argument range, but the numerical error was larger than expected, so the range was truncated. Alternatively, the FPATAN arctan instruction might need the first CORDIC constant and it was easier to use the same constant loop for FPTAN and FPATAN. (I haven't examined FPATAN yet to confirm this.) In any case, the code works with 0.95; I use it as an example because the angles are larger and more visible.

    Another peculiarity is that the documentation is inconsistent about whether 0 is in the range or not. The book The 8087 Primer by the principle architects of the 8087 and 8086 states that the argument range excludes 0, a decision that was made to simplify the microcode. However, looking at the microcode (#1044) shows that the code explicitly checks for a zero input and handles the case.

    FPTAN appears to work for values almost up to π/2, with some accuracy loss as the result approaches infinity near π/2. However, the values from 1 to π/2 (which are outside the defined range and have an exponent of 0) probably work by coincidence, not by design. The larger exponent results in an additional loop iteration; the lookup of trig constants wraps around the algorithm starts with the last constant, atan(2-15). But due to scaling, this value ends up being interpreted as approximately 2. Since this CORDIC angle is greater than π/2, it is skipped. The algorithm then proceeds with the remaining constants, executing properly.

    Footnote to the footnote: Although the mathematical meaning of "range" is the output of a function, Intel uses it to describe the input of a function, which mathematicians call the domain.

11. One difference between the 8087's temporary real format and the external formats is that in the temporary real, the leading 1 digit is explicitly stored in the number. In the external formats, the leading 1 is implicit and not stored in the number, providing one additional bit of accuracy, but making arithmetic more complicated.

12. The flowchart below shows the control flow for the FPTAN microcode. The main blocks are the CORDIC pseudo-division, the rational approximation, and the CORDIC pseudo-multiplication. (Click it for a large image.)

    Flowchart of the FPTAN microcode.

    Flowchart of the FPTAN microcode.

13. The comparisons take multiple micro-instructions, for example, selecting a constant such as 0x3ff0 (recall that exponents are biased by 16383 or 0x3fff, so 0x3ff0 corresponds to -15), moving the constant to the adder, subtracting the constant from the exponent, and branching on the result.

14. The first step of the CORDIC algorithm (#1061-#1068) is handled separately. It took me a while to figure out why the first step of the CORDIC algorithm is special. The cause is that the CORDIC angles are close to 2-n, but not exactly. Thus, the number of CORDIC cycles does not change exactly when the exponent of the input angle changes. This is a problem because the number of pseudo-division cycles depends on the exponent, but the number of pseudo-multiplication cycles depends on the largest CORDIC angle used. Since the numbers are scaled in both loops, a mismatch in the number of cycles leaves the result off by a factor of 2. The first CORDIC step tests if the first CORDIC constant is used and increments the tmpA exponent by 1 to counteract this. (Don't worry if this doesn't make sense.)

15. The constant ROM in the 8087 holds the significands, but not the exponents. When I originally extracted the constants from the constant ROM, back in 2020 (link), I couldn't find the exponents, so I came up with plausible exponents that resulted in meaningful values. The 8087 has a separate exponent ROM. Now that I've examined the microcode, I can see that the exponents in the exponent ROM are unrelated to the significands in the constant ROM. I'm glad I didn't waste much time trying to figure out the exponents back then :-)

16. The following table shows the calculations at each step of the algorithm, using 0.95 as the input. Note that the calculations are done with integer arithmetic; the exponents (gray) are not stored as part of the numbers, but are implied by the algorithm. The exponents change in each step, so as the values get smaller during the pseudo-division stage, the exponents get smaller too, ensuring that the values can use most of the available bits. The opposite happens during pseudo-multiplication: the values start off small and grow as the vector is rotated; the exponent is increased each step to counteract this. Unlike "real" floating-point numbers, the bit representations are not normalized; sometimes they have two 1 digits to the left of the binary point, and sometimes they have leading zeros. That is, the exponents are fixed for each step; they don't change based on the value.

There is an odd bug in Windows related to Video CDs.
To reproduce it:

1. Insert a Video CD (if you don't have one, you can download this file, extract it and burn it with something that supports bin+cue images, like AnyBurn).
2. Open Explorer and copy the Video CD contents to the PC (you can copy just the MPEGAV folder, which contains the big multimedia files).

The typical outcome is that at some point the copying speed drops to zero and it stays there:

When that happens, Windows Explorer is in a "broken" state. You can close the copy window and eject the VCD and it might look like everything is fine, but there are problems. For example, new files and folders are not visible without manually refreshing the view, some parts of the Windows UI become glitchy and the Windows Explorer process can't be restarted.

The only way to get the system working properly again is to restart the PC. A "soft" restart from Windows doesn't work, though; it just gets stuck at the "Restarting" screen forever. You need to do it by pressing the restart or power button. To investigate this bug more easily, I used virtual machines (VMware Workstation works best here and you can reboot the VM without affecting the host).

Here are my findings:

• The bug first appeared with the monthly updates 2023-03 (KB5023706) for Windows 11 and 2023-04 (KB5025221) for Windows 10. The initial releases of Windows 10 and Windows 11 22H2 with no updates do not have this bug.
• You can often reproduce the bug even with a virtual CD drive (by mounting the VCD image with Alcohol 120%, for example). But doing it with a physical CD is more reliable, maybe just because the copying process lasts longer and the bug has more time to appear.
• If you can't reproduce the bug the first time, delete the copied files, reboot and try again.
• Playing the VCD content with Windows Media Player does not trigger the bug.
• Extracting the content with a dedicated app like VCDGear does not trigger the bug.
• Copying files with a 3rd party app (FastCopy, TeraCopy) does not trigger the bug, unless the app uses the standard Windows method for copying (Total Commander does, by default).
• The bug cannot always be reproduced. But I've seen it on different systems, even with untouched Windows installations. I also found a few reports from other people (example 1, example 2).

I have reported the bug to Microsoft through the Feedback Hub at the end of 2025. Nothing has happened since — no feedback, no confirmation, no fix. Granted, it is a rare bug that very few people will encounter these days, to put it mildly. But a bug that affects system stability is still somewhat urgent, I think.

Why are Video CDs different and how to deal with them?

Even though VCDs appear as regular data discs in Windows Explorer, they're actually not. While searching for clues online, I found out that, for example, Linux doesn't let you copy the files the same way that Windows does. That caused some complaints from users over the years, which prompted some explanations.

VCD's look like standard iso9660 filesystems, but they are not. Attempting to use the filesystem entries for other than pointers to the start of files will result in pointing off the disk due to invalid data. Typically when you copy a VCD you want the still images, menus, CD audio, i-frame index, etc. Copying the disk with dd retains that. Pulling the files off (and ignoring errors, the way Windows does) loses that. (source)

About .DAT files.  The ~600 MB file visible on the first track of the mounted VCD is not a real file! It is a so called ISO gateway, created to allow Windows to handle such tracks (Windows does not allow raw device access to applications at all). Under Linux you cannot copy or play such files (they contain garbage). Under Windows it is possible as its iso9660 driver emulates the raw reading of tracks in this file. (source)

Also, even if you successfully copy the .dat files several times without triggering the bug, the resulting files are not identical, probably because there's no error detection and error correction like on a typical data disc. To rip a Video CD properly, it's therefore better to use dedicated software, such as VCDGear, which will extract the AV content and try to fix the mpeg stream errors. Or clone the whole VCD with imaging software to a bin+cue image.

To sum up: yes, copying VCD files in Windows Explorer was never the optimal way to get the content. But many people were doing that, because the UI says it's possible and it more or less worked fine. A Windows update in early 2023 changed something that affects copying those files. I would expect a fix that either restores the old behavior or just prevents copying VCD files and redirects users to alternative ways. Anything would be better than making the system unstable.

Here's the story of how I put a FLIP fluid simulation on a flipdot electromechanical display. FLIP stands for Fluid Implicit Particle, and yes, the primary motivation behind doing this was the wordplay.

It was built as an installation for EMF2026. This page will mostly focus on the technical details, but for the overview and demo of it working, watch the following youtube video.

Flippin' fluid simulations

For the last few years I've been building all kinds of fluid simulations, though the only things I've made videos about are the volumetric display and the fluid pendant.

While most of them are still secret (I'll publish them eventually) they all have a common deficit, that is, the LED liquid is utterly silent. I'd been wondering whether I could simulate some swishing noises when it occurred to me that an electromechanical display, such as flipdots, would make the noise for us. Realising that it would also make a great pun sealed the deal.

But flipdot displays are crazy expensive. From what I can gather, there's only one manufacturer still in existence, and they have exclusive deals with a small number of artistic studios. Breakfast Studio is probably the most well known, and they aren't interested in talking to anyone with a budget of less than $50,000.

By pure coincidence, at one point I did a bit of contract work for a company that happened to own a big flipdot display, and I tried to convince them to let me play with it. I genuinely offered to forfeit my salary in exchange for putting a fluid sim on their display, and to my astonishment they turned me down. I guess that sealed it: I would have to do it the hard way.

Path to some panels

Asking around, especially at hackercamps, led to a bunch of interesting conversations. A lot of people are interested in flipdots. There was even talk of forming a collective and ordering newly made flipdot panels from China. Surely if anyone can produce these displays for cheap, it's someone in China. But the manufacturing process is remarkably complicated. Even the disks themselves are a sandwich of maybe six different materials – here I've dismantled one of the disks I eventually used:

It's not as simple as just a coil moves a magnetic part. The dots are non-volatile, so they hold their position when the power is removed. There are two permanent magnets (one inside the dot, above, and one in the base) and two cores which can be polarised to set the state of the dot.

Some people I spoke to had some of the new flipdot displays from AlfaZeta, but most people had old ones from eBay almost exclusively manufactured by Hanover, who still exist but no longer make these types of displays.

Aside: a number of my projects have been copied by people in China. If, after publishing this, some company in China starts churning out cheap flipdot displays I think we'll call that a win.

Eventually I got in contact with Sam, aka Look Mum No Computer. His museum of obsolete technology has put him in the enviable position of occasionally receiving cool donations of weird old tech, and one of these donations was an enormous pile of old flipdot displays. It's unclear if they are reclaimed from old buses, or were new-old-stock. A lot of them are in pretty poor condition, but that may just be from how they were stored. Sam's plan was to turn them all into one huge display, but doing that would be a monumental amount of work, for reasons that will become clear in a moment. He was very happy to let me have a few panels to play with, with the promise of more if I could come up with a good way of driving them.

Here's one of the panels:

I kept this one for spare parts as it had some impact damage near the edge. At the top right I've managed to unsolder one group of dots.

The date code is 2007, which is younger than I expected.

The rear of the panel had a protective sheet, which I've removed here.

Note the resolution is 13 by 28. The dots on this panel were manufactured in groups of seven, which is where the 28 comes from. The 13 is probably a result of the designer's affinity for prime numbers.

Connect the dots

The number one problem with these panels is that the circuit board protrudes over the edges, which means we can't tile them seamlessly in that direction.

The number two problem is that the existing drive circuitry is quite slow, taking about one second to update the display. The whole panel is wired up as one big matrix, which puts a limit on how fast we can update it.

Sam was able to get a bunch of the panels working with their original drive circuits, first building an etch a sketch and then a bigger display using a few of the panels. He mentioned that at least one other person had got them working with the original circuitry as well. But building one very wide display isn't enough, we want to tile them vertically too!

The person who got furthest is Mike from mikeselectricstuff, and his flipdot video was extremely informative, acting as my primary reference on how the dots work. The most interesting point is that while it takes around 60 milliseconds for a dot to flip over, it only needs a very short pulse to polarise the cores, maybe a millisecond at most.

Longer and higher voltage pulses can get the dot to flip slightly faster, but not significantly. One thought is that by using higher voltages, we can use shorter pulses, which would let us scan through the matrix faster. Taken straight from Mike's video, the matrix layout is something like this:

To set the first dot, you'd set the first column either high or low, and then pulse the relevant row line. As I understand it, the original driver circuit would hold the column line in one state, pulse through each dot that needs to be set one way, then change the polarity of the column line and pulse all the remaining dots in the column, and then advance to the next column.

It may be possible to build a faster matrix by pulsing all of the dots in the column in just two goes. Hold the column line high, simultaneously pulse all the relevant rows, then hold the column low and pulse the others. If the pulses are one millisecond, we could potentially update all 28 columns in as little as 56 milliseconds, which is about as long as it takes for a dot to flip anyway.

There are two problems with this approach. One is that the power requirements are substantial. Each coil has a resistance of about 18 ohms, which would be 666mA at 12V. For faster pulses we might be looking at 15, 20 or 24V, with over an amp per coil. For these panels with just 13 rows, having a driver circuit which can push and pull 13 amps for every column is going to be tricky. The bigger panels would need even beefier parts.

We might be able to split it into multiple smaller matrices, but the next issue is that any matrix will have visible artefacts. Even though it takes tens of milliseconds for a dot to flip over, if different dots start flipping even slightly out of sync, there is a visual glitch as they progressively change. Even with small matrices of say 8x8, or even 4x4, when you flip the whole display, there would briefly be a kind of checkerboard as each matrix wipes over. For what I want to do, that's not acceptable.

Logically the next avenue to explore is desoldering all of the dots, and mounting them onto new circuit boards.

Closeup of underside of group of dots showing the fine magnet wire

We had some discussions about fast ways to do this but ultimately concluded that it's just not worth the time. The dots are so delicate, with tiny magnet wires and soft plastic that can melt, that it has to be done very carefully. Even with the very best desoldering equipment it would take forever. It's possible that pre-sawing the PCB into pieces would help; it's possible that a specific jig to melt a whole group of 14 pins at once would help; it's possible that we could try to abuse a wave soldering setup to speed it up... but remember, what we're trying to do here is build a fast flipdot display that's cheaper than the commercial offerings. After desoldering we've got the added labour of soldering them again to the new board. Unless our time has no value at all, this is going to end up very expensive.

Closeup of the desoldered group of flipdots, with diodes on the row connections visible

Mike's idea was to build a new circuit board that could solder directly onto the back of the existing boards. He found some very cheap H-bridge chips designed for driving small electric motors, which can run from 12V, don't need level shifting, and have built-in protection diodes. The parts have names like MX6208, BE6208, and LK6208, and one of them can drive a dot directly.

MX6208 diagram from datasheet

As you may expect if you've used motor driver chips before, the two inputs A and B are used to drive the outputs, with logic high and low corresponding to push and pull, but if A and B match then it either applies the brakes (setting both outputs to low) or allows it to freewheel (disabling both outputs). The datasheets conveniently give us the internal schematic and a truth table.

MX6208 internal schematic and truth table

These parts are SOIC-8 and rated for 0.5A continuous, so they can comfortably handle our pulses. The only real disadvantage of using them is that we'd need to cut all the column tracks on the existing PCB before soldering them down. Otherwise, the pulses would interfere with each other. (The diodes on the row connections mean we don't need to cut those.)

Mike was kind enough to give me his prototype boards as a starting point, and I spent a while studying them and thinking about our options.

Mike's PCB on the back of a flipdot panel

Typical of his designs it's filled with clever little details, like doubling up of the connector footprint so any of them can act as an input or output. Each shift register controls four dots, with the two inputs of each H-bridge directly controlled by the 8-bit shift registers. The latch and OE signals are tied together for simplicity.

The plan for these was to mount the whole panel in a small CNC mill, and use it to cut all the traces. Given that the edges need to be cut off too, this wouldn't be too inconvenient, if not for the fact I don't have a small CNC mill. Before cutting all the traces manually I tried to explore some more options.

Another technique for driving the dots involves using a series capacitor. There's a nice project here that uses the technique to build a new filpdot display. The advantage is that you don't need a full H-bridge per dot, only a single half-bridge.

Schematic of capacitor technique

When the bridge goes high, a pulse is sent through the capacitor which flips the dot. The capacitor then stays (or is held) fully charged, until the bridge pulls it low again, creating an inverse pulse that flips it back.

Of course it's not actually as simple as that, to control the high side from logic level signals we'd need another transistor.

Schematic of capacitor technique with level shifting

Naturally we'd use mosfets for the real thing. The linked project uses three mosfets and three pull resistors for every dot, in addition to the big capacitor. The biggest risk with building your own half bridge is the chance of turning both top and bottom switches on at the same time, which would cause it to explode. Another big downside is that having multiple discrete components takes up a lot of board space. The linked project uses a dedicated microcontroller for each group of seven, on a four layer board, and doesn't need to contend with soldering onto the back of an existing board.

In contrast the H-bridge chips are a self-contained solution, and combined with the very cheap 595 shift registers, Mike's board is much cheaper and simpler than the capacitor technique above.

But one thing about it really appealed to me: if one pin of each dot is grounded, we won't need to cut any traces. All of the column lines can be connected to ground. This also means we won't need to do as much soldering.

I did some test board layouts and found that we really don't have much space to work with. Certain areas of the PCB need to be empty to avoid shorting with pads on the original board, and we also need to leave space for mounting holes. Even if I wanted to do it with discrete components I'm not sure I could fit it. Thus began my search for a cheap, small half-bridge.

The first thing I looked at were gate drivers, these are dedicated devices for overcoming the capacitance of big mosfet gates. They usually have level shifting for us, and work like an isolated half-bridge that can momentarily deliver a huge pulse of current. Unfortunately, even the very cheapest ones are about twenty times the cost of those H-bridge chips. That'll compound when we scale up to thousands of dots.

Some of those gate drivers are available in SOT-23 packages, which would help a lot when it comes to board layout. There are also H-bridge chips available in SOT-23, but unlike the SOIC-8 parts they don't seem to have a standardised footprint. I compared dozens of them and found that nothing was perfect, nothing gave me confidence. I also looked at various "relay drivers" which had their own pros and cons.

Some shift registers have built-in output FETs that let them low-side switch a load directly. What I'd love would be a shift register with high-current push-pull outputs at a suitable voltage, but as far as I can tell nothing like that exists. There are high voltage shift registers if we go back in time to early CMOS technology like the 4000 series. A CD4094 shift register could work at 12V, but, the output capability is negligible, we'd be lucky to pull 10mA from it.

More options: there are buffer chips that could work in the ranges we expect. I found a few that would work electrically, but again, the price made them impractical, and most of them have footprints that'd be difficult to work with.

The idea of using one H-bridge to control a pair of dots was appealing but unfortunately not possible, because of their brake and freewheel conditions. However, given that the parts are so cheap, and we were planning on using one per dot originally anyway, would it be crazy to use them in a single ended fashion, ignoring one of the outputs?

Truth table for MX6208 again

It's not as easy as it sounds. Studying that truth table, if we send our signal to AIN, and hold BIN low, then AOUT would go high-impedance when AIN is LOW. If we hold BIN high, then AOUT would only ever be low. But if we send our signal to BIN, and keep AIN high, then the output will toggle high and low correctly (even if the part thinks this is a brake condition).

To see if this was even close to viable I built a test setup on some protoboard.

Prototype dots on a breadboard with protoboard shift register and H-bridge chips driving it

We have a 595 shift register, the H-bridge chips wired in single ended fashion wth AIN tied high. The capacitors are multilayer ceramic 100uF which were all I had to hand in the right ballpark. A nice bonus of wiring it up this way is that we can now drive eight dots from one shift register, so that's half the number of shift registers and half the data that needs to be clocked out.

Closeup of driver chips and capacitors

One of the lesser concerns with the capacitor method is that of self-discharge. I assumed this wouldn't be a problem because we can simply leave the outputs powered. No current should flow through the capacitor except enough to keep it at full charge. Unfortunately these chips are not built from FETs, but BJTs, so I soon realised that that was a mistake. When an output is enabled, even if unconnected, the chip consumes about 24mA simply because of the base currents through the transistors. Multiply that by a few thousand dots and we have a quiescent current that could kill the project.

Evidently we still need to pulse the signal then. If capacitor discharge is an issue, we can always periodically re-pulse it. Again looking at the truth table, we can use the AIN signal as an enable pin. I decided to tie this to the latch signal for the shift register. A minor benefit of the BJT inputs is that we don't need pulldown resistors: when the shift register's output is disabled, the H bridge inputs are effectively low.

Breadboard prototype with OE/latch signal added

With this I felt like we were finally getting somewhere.

A neat thing about the series capacitor is that it comes with some inherent safety. If a glitch occurred with the direct drive method, it could hang with full current flowing through the coils, eventually overheating, maybe combusting. This circuit comes with far fewer ways in which it could conflagrate.

The choice of capacitor is important, those big ceramic ones weren't going to fit on our PCB. I adapted the prototype so I could stick various different capacitors in and see how they compare. An interesting thing about the 100uF ones is that only after the fact did I notice they were rated for 6.3V, and I'd been pulsing 12V through them. Using some 10uF, 16V rated ones gave a noticeably worse performance. I can't say I really understand the physics of ceramic capacitors, I know that the capacitance is a function of the bias voltage so already it's a bit of a mind screw. If we pulse a capacitor above its voltage rating does it reduce its lifetime? Probably. Cost is a factor here, with standard sized capacitors being significantly cheaper than speciality ones, and it's really gonna add up later on. I concluded that two 10uF capacitors in parallel would probably be enough, and then we can use standard 0805 packages for them.

Another thought: does the series resistance of a ceramic capacitor depend on its physical dimensions? A 10uF ceramic is available in various footprints at different costs, so there must be some difference in the technology used to shrink them. Instead of idly wondering, it was time to throw our first PCB together.

To KiCad

These flipdot panels have a pin pitch of 15.24mm, or six tenths of an inch. The board supporting them has test pads or an unpopulated footprint in the corner next to each one, so we'll stick a keepout around that. I created a flipdot symbol and footprint which we can lock into position on the board.

But KiCad is really not very good at this sort of design, where we have lots of repetition. Unlike some other EDA software, there isn't a clear idea about forwards and backwards annotation. The original process was to create a schematic, export a netlist, and then import that netlist to the PCB editor. Newer versions have streamlined it to add an "update PCB from schematic" button, and in the latest versions there is even some attempt to add UUIDs to symbol-component pairs, but so far it's still a little janky.

The fundamental problem as I see it is that the auto annotation is useless. If you copy and paste some schematic symbols, you have a choice to leave them unannotated (so resistors have labels like R? instead of R3, etc) or to reannotate them based on their position (so resistors would be annotated based on the first free identifiers, in a left-to-right, top-to-bottom fashion). If you copy and paste parts on the PCB, you get the same options, but those are almost never helpful choices, because it's unlikely that the schematic and PCB will have the same physical layout. The effect is that if you copy and paste anything but the most trivial circuits, the schematic and PCB get annotations that don't match.

I did file an issue about this but I think I failed to convey exactly what I wanted and it got buried among a thousand other issues. The way it should behave – by default, if you ask me – is to re-annotate based on a predicable method that is independent of geographic layout. The most simple option would be to re-annotate in the same order as the source parts. So if you had R1, R2 and R3, they would become R4, R5 and R6 regardless of which one is to the left or the right.

There is a plugin for duplicating layouts, but the approach they use is different to what I wanted to achieve here. What I wanted, and what KiCad can do once it was fought into submission, is a single schematic file that describes one row of dots, and a single hierarchical file that links copies of that row together. If we edit the row, all of them change and the netlist used by the PCB editor reflects that.

Frustratingly, the newly added UUID system makes it even harder to get this to work, but with enough wrangling and a little bit of python I got there in the end. The overview looks like this:

Hierarchical arrangement of dot rows

And the dotrow.kicad_pcb is of this sort of persuasion:

Partial schematic for one row of dot drivers

There I've highlighted the latch signal going to AIN of each driver. The two 10uF capacitors are in parallel before each dot. If we open the same schematic for the next row, the layout is identical but the annotations are all different.

I based the PCB layout loosely on what Mike did, adding the dual footprints for serial data and some mounting holes, which I carefully checked would not interfere with any tracks on the underlying PCB. I extended this to 13 rows, so it would cover the full height of a panel.

KiCad Screenshot of first PCB design

Horizontally, we cover eight dots, so we'll need three and a half of these to drive one full panel of 28 columns. That's a bit awkward but we can deal with it later, for now I just wanted to confirm the ideas. While I waited for the circuit boards to arrive I made this 3D printed fence for a dremel with a diamond wheel, so we can efficiently cut away the old driver circuitry.

Using the dremel with 3d printed fence to cut off excess circuit board

The cut distance is different for the top and bottom of the circuit, which is slightly awkward but when it comes to preparing all of the panels, we can do all of the tops first, readjust, and then do all the bottoms.

When the circuit boards arrived, we held our breath as we soldered the 117 pins necessary (13 x 8, plus 13 grounds).

First PCB soldered onto the back of a flipdot panel

Initial results were mixed.

Flipping individual dots worked, but flipping all of them at once would fail. If each dot needs a pulse of around half an amp to flip, then flipping all of those would be about 50 amps, if only for a millisecond. As the power supply couldn't cope, the voltage drops, and the pulses become weaker. Instead of polarising the cores, they simply demagnetise the cores, and the dots go slack.

I noticed early on that the behaviour at lower voltages was significantly different with different series capacitors. Limiting our selection to 0805 ceramics, even doubled up, was probably a mistake, as they seem to suffer more from the undervoltage condition than a single big capacitor.

Another issue is that the inputs to the H-bridge chips draw significant current. The input is the base of a BJT, so unlike CMOS parts it will draw a few milliamps, and multiplied by a few hundred means the latch signal now needs more current than can be provided by a simple GPIO pin. I was driving this from a basic CH32V003 breakout board that was struggling to keep up. No worries, we can bodge in a mosfet to boost the current.

P-channel mosfet fitted to latch signal

That's a P-channel mosfet (FDN338P) on one of the unused connector footprints, with its gate connected to the OE signal. This suits me fine, we were already toggling the OE and latch signals alternately. This also saves us one pin on the connector.

I went with JST ZH connectors (1.5mm pitch) partly because I already had a pile of ready-made cables. These cables turned out to be just slightly too short, and I had to manually extend them...

First three boards soldered onto the back of the panel

The white wire is the former latch signal, which I've disconnected as it made sense to stick a mosfet on each of the panels.

I noticed an asymmetry: it's easier to flip in one direction than the other. You might think it's to do with the capacitor arrangement, but counterintuitively flipping from high to low is faster. Simply due to the physics of transistors, the low-side switching can sink more current than the high side can source. Another way of thinking about it is if we'd driven the coils directly from the H-bridge, the flipping would be symmetrical, but only because both directions would be limited by how much current can be sourced on the high side.

With more of the dots active, the dependency on a decent power supply worsened. With two benchtop supplies in parallel I could just about get all of the dots to flip at once. We can up the voltage (the drivers are rated up to 15V) but I really want the final thing to work with a 3S LiPo battery (from 12.6V down to 9.0V).

While experimenting with different power supplies, inevitably I ended up touching a 12V signal to one of the data lines, and one column of shift registers disintegrated. Nothing is more discouraging than destroying your dev board, so I decided to take a break.

Sad thermal image of the damaged panel, with one group of shift registers looking hot

Sad thermal image of the failed shift registers acting as a dead short

Second circuit

About six months later I came back to this and figured that power supply problems are best solved by flooding the thing with electrolytic capacitors. Some of the surface mount ones are actually cheaper than ceramics. What's more, the behaviour of electrolytics as a series capacitor for the dots seemed to be much nicer, so we might as well replace all of them with the biggest electrolytic caps that could fit.

I churned out another PCB design.

Screenshot of the second circuit design

The caps are 47uF, 5x5.4mm, with slightly shrunk footprints in order to fit them in. I added a 3.3V regulator and swapped out the superfluous connectors with test pads at the edge of the board, so we can do little jumps to daisy chain them instead of cables. The 12V is also taken to the JST connector, as we had a spare pin, and we can use it to measure the supply voltage and refuse to pulse if it's too low, both for battery protection reasons if needed, and to avoid failed pulses when the supply browns out. I stuck a polyfuse on that connection, as we might be using a really beefy power supply which got me a bit paranoid.

When they arrived I trimmed another flipdot panel and soldered up the result.

Second prototype PCB soldered to a panel

Immediately we got much more promising results, but I'd also ordered a 100-pack of 1000uF electrolytic capacitors, which I planned to solder in as many places as needed. It's only briefly that we'll need hundreds of amps, so with enough capacitance surely anything is possible.

Driving it is one of my CH32V003 dev boards, now pulling 3.3V out of the connector so it can run a test pattern without assistance.

Working my way through an audiobook, I soldered more of the boards into place, and also laser-cut some plywood panels to mount them. The bed of my laser isn't big enough to do it in one piece, but two pieces of 12mm plywood would hold two flipdot panels together. I mounted some brass standoffs strategically over the board.

Using brass standoffs to mount the panel to plywood

On the first PCB I'd aligned all the holes to avoid any tracks, but for the top edge it was very hard to fit the nut. Here, I moved the top standoff over, making it much easier to fit, but now it intersects one of the row connections. This did end up causing problems down the line, where a couple of them shorted that row to ground, but knowing that can happen is enough to avoid it, as we can peel away or cut that trace when we fit the standoff.

Without the edges, these boards are very difficult to hold without knocking out a bunch of the flipdots. I only realised how much I'd been struggling after I mounted it to the plywood and everything became much easier.

One PCB manages eight columns; there are 28 columns on the panel. I had wondered whether we could link two panels end-to-end, forming a single panel with a driver board straddling them, but given how unwieldy that would be I decided, at least for this prototype, to saw a driver board in half. I'd made sure to keep all of the important bits to the left of the centreline.

Driver board sawn in half

Another reason to cut the board is that we're probably close to the limit on how many shift registers can be chained together. There are 52 on the panel, and with a clock speed of a few megahertz I expect it wouldn't go much further than that. Best to stick a CH32V003 on each panel which can listen to serial data and drive its own group of shift registers accordingly.

Also visible above is a Deans- or T-connector, matching a 3S LiPo battery I'd acquired for testing. Even a cheap small one is able to deliver tens of amps continuously.

The panel comes to life slowly. Many of the dots are missing or don't work, but with gentle coaxing, re-soldering pins where needed, and fitting spare dots, we eventually approach full functionality. For one of the dots the pivot had broken off, but amazingly I was able to find the missing part at the bottom of the box it came in.

A dot missing one of the pivot points, with the broken plastic next to it

Some superglue set things straight.

Another board was sent through the same preparation process. Best part of an audiobook later and the exciting part was getting near.

Another panel with driver boards fitted, with the original panel in the background

At this point I'd started fitting the 1000uF caps, settling on eight of them per panel. With just a bit more tedious work, the standoffs were fitted, the panel was mounted, and finally with just a little bit of embedded magic, a fluid simulation was brought into being.

Two-panel display running a fluid sim

My FLIP implementation was just able to run at this resolution at about 40FPS on a Pico 2. Each panel has a CH32V003 listening to UART and squirting out SPI. The seam between the panels is vertical here, and almost invisible.

With that, we'd proved that we can drive the dots fast, and tile the panels seamlessly, and do it for less than a fortune.

It was pointed out to me that we could have stopped here. We got a fluid simulation running on flipdots, and hand holding the thing has amazing tactile feedback as the dots flip. No one would criticize me for arriving at EMF with just this. But my plan was always to go much bigger.

Mounting Dreams

The two-panel prototype has a resolution of 28 by 26, and physical dimensions of 396mm by 427mm (about 16 by 17 inches). Stacking more pairs in a 2 by 2 square would need eight panels, or 3 by 3 would need 18 panels. That would end up about 1.2m per side (roughly 4 feet). Anything that sort of size is going to be too heavy and unwieldy to manipulate in your hands.

Ways to arrange the rectangular panels

So, we need a mount that will allow the fluid to be manipulated. Initially I planned to build some kind of gimbal, maybe like this:

3d sketch of a possible gimbal

A key motivation is that I wanted the general public to be able to interact with it, and especially to do so in a way that's intuitive and doesn't need me to supervise it. If we welded up a frame from steel, we'd need to be careful that people don't spin it too fast or crush their fingers.

One method of adding damping is to use stepper motors. The motors wouldn't be driven, just connected at each pivot (possibly acting as the bearing) with the coil connections shorted or through a resistor. Using a variable resistance, we could control the exact amount of damping applied. It could even be coupled with series diodes, so that there's no damping at all below a certain speed.

But it might not even be obvious that the panel is something you can grab and manipulate. We'd need clear handles on it. Then again, if there are stepper motors on the pivots, maybe it should just be driven? We could have a control panel in front with arcade-style sticks.

How would it be powered? Adding brushes to each pivot makes it substantially more complex, so my main idea, at least if we forget motorised pivots, was to put a big battery pack on the panel. That would require frequent changes, fine if we just want timed and supervised performances, but no good for leaving it unattended.

A simpler option would be a single pivot, just mount the panel on the wall and let it rotate in one plane. It suffers from the same issues but would be easier to make, and much easier to give a motorised control.

Or perhaps it could be mounted on its back, like a table, with variable length legs?

Render of an arrangement like a table with adjustable length legs

This'd need pivots at the top and bottom of each support of course. Hydraulic cylinders could be unpowered and connected to a shared sealed reservoir, meaning they'd automatically spread the weight. If we're going to do that we might as well go full Stewart platform (six pistons in a delta configuration) and build a control stick mounted on six miniature pistons as encoders.

These ideas are all within my skillset but not really my budget or timeframe. How about something simpler: we could hang the panel from the ceiling (or a frame) by a rope, and then it could swing around. Anyone strong enough could pick it up and manipulate it, others could merely swing it and see the effect. It would probably still need a yoke of some kind rather than hanging it from an edge or corner. If the yoke itself was a pulley, that might give us reasonable range of motions.

Render of a possible mounting using rope and pulley

It would need to be perfectly balanced, and still depends on some kind of scaffolding to attach the rope to. And it still could be pretty dangerous if someone pulls it off to the side and lets go.

One final idea I had was to mount the panel on a big squishy ball. Sort-of the inverse of the pulley arrangement, just a soft and gentle blob that takes the weight of the panel as it's tilted about. It would at least be very easy to build.

At this point I wasn't even sure how big I could make it, as I needed to collect more of the panels from Sam. He's a very busy man at the best of times, but round about this point it was announced that he'd been selected as the UK's Eurovision entry. I didn't want to be pestering him in the run up to that, so I thought I'd wait till after, and then next I heard he's just had a baby. I really didn't want to be this annoying guy who keeps nagging for more flipdots, but in the end he was very amenable and enthusiastic about the project. Despite having his own EMF installation to worry about (the fire organ) he let me visit and helped me rifle through the pile of flipdot panels.

The idea of preparing and soldering eighteen panels was already pretty daunting, it'd probably take me most of a month, and with the donor pile being a varied mix it wasn't clear if there even were that many in the size I needed. We did a spot survey, and the majority of the panels were bigger than I'd what designed my driver boards for. There were also some at a different pitch, looked like 10mm instead of 15.24mm. Eventually we unearthed enough of the correct size for me to make my 8-panel display, and I embarked on the mammoth task of soldering them up.

Panel preparations

I had planned to update the design with another revision of the PCB, but what with my indecision about how it should be mounted (you might say I was "flipping" between options, hoho) and various other commitments I effectively ran out of time. The boards used for the two-panel prototype aren't perfect but they function, so I ordered more with virtually no changes. About the only thing I did was upgrade the testpads that propagate the serial data. On the two panel prototype, after swishing it about extensively, one of the pad connections had ripped from the board. For more mechanical strength I made them plated holes, into which I could shove bent bits of wire, shown here before soldering:

Closeup of linkages between adjacent panels

A number of dead pixels cropped up down the line after the prototype had been played with. Re-soldering the connections to them brought them back to life. It was only on the third panel I realised that the boards have some kind of conformal coating, which is why they're so hard to solder to (or desolder). In my attempt to work quickly, not all the connections had properly penetrated the coating and while they initially had continuity, some joints failed after being mechanically stressed.

There might be a solvent that could dissolve the coating, but simply scrubbing the connections with a brass wire brush beforehand worked wonders.

Brass wire brush scrubbing the back of a panel

I would have really liked to add a quick way to couple the power connections. There are certainly all kinds of high-current wire-to-board options, but most of them are pretty expensive. The few minutes of soldering we'd save probably isn't worth it.

Rear of panel with linkages soldered

The soldering was endless, but on the plus side here was the perfect time to use up all my neglected reels of solder, the ones with the wrong alloy, or wrong flux content, which annoy me enough that I don't use them for anything that matters.

For the end of each panel, this time I ordered some halfboards that just handle four columns, saves sawing them in half.

Each panel also got standoffs fitted, taking extra care for the top where we now have to drill through a row connection. I made a depth stop for the drill (just a sleeve so that only a bit of the drill sticks out). FR4 blunts drillbits quickly, and I'd bought a multipack of them in anticipation, but actually the first bit lasted the whole batch. To be fair it was only 64 holes in total.

Brass standoff fitted

I used tweezers to hold the nut, and a tiny socket driver to tighten the standoff. I possibly could have created a special tool to hold the nut in place. If we do another batch, I think maybe nylon snap-in standoffs would be better, the type you simply poke through a PCB and they lock in place. They'd need to cope with the double thickness PCB though, everything I could find was for 1.6mm boards.

Soldering all these panels took forever, and happened in parallel with the rest of the project. I think it was about panel three I concluded that eighteen panels wasn't going to happen. It doesn't sound like much, only 400-odd joints to solder per panel, but when you factor in all the little details, testing the panels, going back to fix the dead pixels, testing again etcetera, it was remarkably tedious. As I got more practiced, at my very fastest, I managed to prepare two whole panels in one day. All of that was negated when, for a later panel, I soldered the PCBs upside down, and had to spend a whole extra day desoldering them.

Prototype Polish

My mounting worries (as in, worrying about how it will be mounted, though the delays were starting to compound) highlighted the wisdom of having a backup plan, in case the big FLIP is a flop.

Dots easily get knocked out, so the prototype needs handles, protective side panels, and maybe a clear cover.

The laser at the hackspace has a bigger bed, so we could cut out the rear panel again in one go. Turning to FreeCAD we managed to cobble this together.

Freecad screenshot of case for prototype

This felt like a project within the capabilities of FreeCAD. It is, after all, just a few bits of plywood with holes in them. But coming from "real" CAD software it's an enormously frustrating process. We have to start using it if we want it to get better though.

There are some fundamental issues with Open CASCADE (which FreeCAD uses) that I understand are very difficult to solve. Things like the "topological naming problem" which is definitely still a problem. But there are also plenty of trivial frustrations that seem to an outsider as fairly easy to fix, like the minor UI problems, or the tedious steps you have to follow to achieve what should be a single click. I won't go on about it, hopefully these annoyances will evaporate soon.

The parts were laser cut...

Plywood panel laser cut for prototype

...sanded, given two coats of paint...

Plywood panel painted black

... and test assembled before we start transferring the gubbins.

Plywood panel test assembly

The holes in the sides are for airflow. This thing draws a few amps while running, and that heat has to go somewhere, presumably into the coils. It seemed prudent not to put them in an airtight box. For the same reason, I didn't want to seal up the back, even if the boards are a little exposed.

We also cleaned up the wiring, adding a power switch and nicely crimping and heatshrinking everything. I got some slick silicone cables, here we've soldered up the Deans connector.

Deans connector soldered and with heatshrink ready to apply

Some more painting, careful dismantling and re-assembly, and this prototype was starting to look ship shape.

Rear view of assembled prototype

The LiPo is held in place with velcro, and all the important parts are held in place with screws or wire ties.

Front view of assembled prototype

To finish I added some feet to the back so it can lay flat without crushing the electronics, and installed an acrylic sheet to the front, aligned to those protrusions in the corners. I spent ages fitting tiny magnets round the perimeter so the acrylic sheet is removable.

Magnets next to a hole in the acrylic cover

The magnets get glued into the holes, and matching holes with magnets are made in the plywood.

However the acrylic sheet ruins the sound of the flipdots, muffling them until the fun is gone. At least now we know not to bother when it comes to the big display. Another interesting effect, as I peeled off the protective film, the static charge played havoc with the dots, overpowering the magnetic attraction. Even after wiping it with a damp cloth it seemed to impact their performance.

The cover was very useful for protecting it in transport though. Pictured here upon return from EMF, with scuffed case after plenty of usage:

Fliptwo prototype on its back

Framework

I've had great success ordering parts laser-cut out of sheet steel. There's a minimum order cost, but beyond that it's very cheap, hardly more than the base cost of the steel. With some thought, we can add tabs to the pieces so they slot together neatly before welding. It's a very fast and accurate way of building really strong and durable constructions.

But it comes with a lead time, and with the drop to only eight panels, I realised I could do the whole thing in plywood in one go at the hackspace. I scaled up the design of the prototype.

FreeCAD sketch of the back of the eight panel frame

Ah, FreeCAD. So clear. The red cylinders mark a potential hole pattern for mounting it. Not shown are the imported PCB outlines we bind the hole patterns to. The grid pattern is a compromise between strength and exposing the circuitry, for air cooling and all those electrolytics we plan to install.

FreeCAD screenshot of eight-panel frame

It's much cheaper, and much much faster than sending off for it, but my plywood comes with the downside of being warped. In retrospect, I should have started with higher quality stuff. After cutting the stock to fit in the machine, I had to raid the welding area for as much scrap steel I could find, to weight it down in the corners. If I'd cut it a little bigger we'd have had an easier time, here I had to triple check the head wasn't going to collide with a pile of steel.

The big single cut on the hackspace laser

That's probably the biggest single cut I've ever done. The optics aren't great on this machine, and the progressive error of the beam means it cuts worse in certain areas. Couple that with the warped plywood (meaning inaccurate focus) and the only way to guarantee a complete cut is to set the speed real slow. I think it was 57 minutes of nail biting. If you nudge anything the whole thing is ruined.

I set the warp to be concave, so weights could hold it down at the edges, but I put no thought into how this would affect the flipdots. I should have set the design so the flipdot panels are crushed together, rather than spread apart which might lead to a visual discontinuity. Oh well. Fitting the side panels counters a lot of the warp anyway.

Test assembly of the big laser cut frame

The underside splodges are a sign the laser dwelt too long in those areas, but in other places it didn't fully cut and I had to finish off with a chisel, so overall I think I got the speed and power about right. It all gets hidden by two passes of sanding and painting anyway.

Parts painted

I still carried a vague hope of gimballing it all, but with only a few days till the event an articulated fixture was looking unlikely. Keeping my options open I added four M8 mounting plates in an approximate VESA pattern, to fit an off-the-shelf TV mount. The backup plan was to keep the panel static and put a joystick in front which would control the direction of gravity.

There are plenty of M8 plates available as table-leg attachments, but worried about the strength of the plywood I laser-cut spreader pieces that save us having to make big holes in the frame.

M8 mounting plate attached with spreader

With that we can start arranging all the prepared panels for the grand assembly.

Closeup of arranged panels for assembly

Too big for my desk, another not-quite-flat slab of plywood serves as the surface.

There's a reflective moment in projects like this, where the accumulated time sunk starts to become tangible, the hundreds of hours of soldering spread out on the table in front of you, and you wonder, is it going to work?

Panels arranged on extended desk

Thus began an extended session of awkwardly maneuvering the frame and endlessly installing M3 bolts with laser-cut washers. The side panels are then fitted, along with the VESA mount rails that make it easier to pick the whole thing up.

Frame fitted to panels

It weighs about 15kg at this point. The side panels are just a press-fit, though I'd planned to screw them in place.

While it appears to be a thing of beauty, there's a lot more electronics to do before we can test it.

Decoder Boards

On the prototype, the shift registers were driven by hand-soldered microcontrollers listening to UART from a motherboard. Scaling up, with more pixel data and physically longer cables, something more resilient is in order. Now that we've dropped to just eight panels, RS485 might be overkill, but with thousands of coils pulsing thousands of amps... a little bit of overkill can't hurt.

With asynchronous serial data, as a general rule, the faster the baud, the shorter your maximum cable length. RS485 is a differential signal that gives you vastly more headroom than a single-ended equivalent. The only downside is that you need encoders/decoders at each node. There are plenty of alternative ways of doing it, sometimes it makes sense to daisy-chain panels, with each one re-transmitting to the next, much like how addressable LEDs work. Or, have separate groups of serial data (referred to as a "universe" in the DMX world) so each group carries only a fraction.

My preferred method is to give each board an ID, and transmit all of the data to all of the boards. Not only does this mean we can route the cable in whatever order we like, it means we can easily sync the frames. The data consists of a header, a big block of pixel data, and a footer. Each board decodes its chunk of data as it arrives, but only latches the shift registers at the footer.

Schematic of decoder board

A DIP switch lets us set the ID (up to 32, you never know) and the 12V line is monitored, via a divider, so we don't pulse if the supply drops below 9V. This is both for protection if we do run it from a 3S LiPo, and to avoid failed pulses that demagnetise the dots. It's important to give this hysteresis, so the low-voltage cutout latches until the supply climbs above, say, 9.5V. The VCC line is from the 3.3V regulators on the panels. Note that there are variations on the MAX485 decoder chips, not all of them work at 3.3V, a lot expect 5V.

A full frame of data is only about 3000 bits, so 1 Mbaud is fine. A nice thing about CH32 parts is that they're pretty accurately trimmed from the factory, I've not had any problems decoding UART with them, unlike AVR chips, which often misbehave without a crystal. There's enough headroom that we don't even need proper twisted pair, my plan was to use a ribbon cable and clamp IDC plugs anywhere we need to tap into the data.

It's tempting to try to do the bit-shifting on the decoders, to offload whatever's driving it. It's particularly awkward as the second column of panels doesn't align to the 8-bit data. But trying to figure it out before the thing's assembled is far too mentally taxing so it's best if each decoder simply snorts a block of data and squirts it out unmolested.

In my haste I forgot to add a status LED and immediately lamented it. I hand soldered one onto one of the boards. I also failed to really think through where the boards would sit. The JST connectors were meant to make things simple, but not having cables of the right length complicated the situation, and eventually I cut some pre-made cables and soldered them in place at the decoder.

Decoder board connected to panel

It would have made more sense to connect the decoders to the panel before fitting the frame, and then skip the holes in the plywood entirely. I had thought about using surface-mount header pins, with matching socket on the decoder, but again, we had no time for such fancy solutions.

With the frame and decoder boards ready, we can start fitting the big 1000uF caps, tediously bending their legs to avoid obstacles.

Big capacitor ready to be soldered

The boards were then wiggled into place and stuck down with VHB tape.

Decoder board fitted to one of the panels

There was so much soldering still to do, that I transplanted my illuminated fume extractor to get full coverage.

Using fume extractor on big panel assembly

Power supply

There's a vast range of high current DC connectors available, but I limited the search to types commonly fitted to batteries in our ballpark. These 5200mAh 3S LiPos are fitted with XT60 connectors (There is an XT90 connector for even higher currents, used on things like electric bikes). 80C means they can deliver 416 amps continuously, or in other words, can be drained from full to empty in just 45 seconds. For testing our half-finished panel they were very useful. They even conveniently clip into the mounting rails, a pleasing coincidence.

LiPo battery powering half-assembled panel

A looming question was whether the inrush current was going to be a problem. With nearly half a farad planned in total, powering on will involve an enormous surge. Nothing that LiPo can't handle, but maybe an issue for our mains supply. Unprotected batteries won't care, but if there are fuses in the supply circuit the inrush current can cause them to blow before it powers on. There's ways to deal with it, usually by warming up the circuit first via a resistor, then switching in the full current. That could be via a relay, or a mosfet if it's suitable. You can even use a single mosfet and an RC circuit at the gate, so it switches slowly.

After some experimentation and extrapolation, I decided that at this scale we're probably fine. The average draw of the whole panel in use is going to be around 10 to 15A, and the boards won't start pulsing until the supply has stabilised. Another factor in our favour is that the capacitance is distributed, and the inductance of the power cables will filter a lot of the spike. Worst case, we can split the power rails into groups and give them separate switches.

I ordered a 12V mains power supply rated for 40A. The first thing I did was adjust the output up to 13V, for slightly better performance from the dots, especially if there'll be losses in the cable. And in case someone trips over the cables, risking a loose live wire, I wanted proper connectors instead of screw terminals. That means building a little box with panel mount sockets.

FreeCAD screenshot of power supply box

We're careful not to block any of the vents but otherwise cranked this out at breakneck speed. Even restricted ourselves to only one coat of paint.

PSU box parts ready for painting

Connections were soldered/crimped and heatshrinked.

Assembled PSU box with wires connected

I'd originally planned to sit this on the floor somewhere away from the installation, and make up a long DC cable. I bought male/female XT60 connectors and enough 12AWG silicone flex cable to keep it a few metres away, but then I realised we might as well just mount it to the back of the panel and save ourselves the bother.

PSU box with cover fitted

However, the PSU has a fan, and if we mount it right up on the panel, that fan is actually deeply annoying. It has a loud whine, and it spins up and down as the load on the display changes. Most people might not have noticed, but one of the main reasons we're doing this is for the sound of the flipdots, so it's not something I want to compromise on.

Pulling the supply to bits, the fan has a standard size, but not much wiggle room with where it sits. We can't fit a bigger fan, but we can replace it with a better one.

PSU dismantled with fan visible

I ordered a "Noctua" brand fan, very expensive, but supposedly the quietest you can get, with engineered blade design and superior bearings. Annoyingly, the old fan isn't driven by PWM but some kind of current control circuit, with the terminals reaching 30V with no load. The Noctua fan seemed to cope. I did consider if it might be better to run it straight from the 13V, then at least it won't have those noticeable speed changes.

While quieter, the new fan had perceptibly lower flow. It would be just my luck if I muck up the cooling, the power supply blows up and the whole installation is a dud.

New fan ready to fit to PSU

Whatever – I rolled with it and also stuck some passive heatsinks on the case where the main transistors are bonded.

Joystick

I'm aware of how long this writeup is getting, but the joystick could have been an entire project on its own.

Sketch of joystick and static panel arrangement

Statically mounting the panel as an easier option than the gimbal occurred to me only a few weeks before the event, which sounds like plenty until you realise nearly all arcade style joysticks are essentially D-pads, with just four microswitches instead of analog output.

I found a bunch of projects where people had converted off-the-shelf arcade joysticks to be analog. One of them looked promising but they hadn't released their CAD files before disappearing from the internet. Another was a commercial offering replacing everything with hall sensors and a big toroidal magnet attached to the underside of the stick. We might have been able to cobble something together, eventually, but the joystick was meant to be the "easy" option. Eventually I found a potentiometer-based joystick from the brand SJ@JX, from just one seller on Aliexpress. I ordered it right away, but postage from China meant a very stressful wait.

I was able to plan and order the rest of the pieces before it arrived, along with a backup idea we won't go into, in case the Aliexpress option failed to turn up or failed to work. When it finally arrived, we embarked on one of the maddest dashes yet.

Joystick parts ready to assemble

First thing is to mount the stick in the enclosure. The provided mounting plate was used to transfer the holes, which we then hit with the step drill.

Using a step drill to make the hole pattern in the enclosure of the joystick

Scribe lines are faint but we can check progress with the mounting plate again.

Checking progress of step drilled hole

It's unclear if this plate is meant to sit underneath the mounted surface, or if we can forget it. A test fit shows the full range of motion, so we move on.

Test assembly of joystick

The underside gets a hole pattern for some rubber-coated magnets. I hadn't decided on how to make the pedestal, but magnets keep our options open.

Hole pattern on underside of joystick enclosure

Convenient we have all these scrap bits of plywood around eh?

Feet mounted to underside of joystick, test assembly

For connecting this to the motherboard, instead of a cable gland I had the idea to use a MIDI cable and a 5-pin DIN connector. They're cheap, available, and shielded. We can also get nice cables with industrial-looking metal plugs.

Hole drilled for 5-pin DIN connector

Out comes the little M3 tap wrench.

Tapping holes for the DIN connector

With the socket pins bent, there is just enough room for the joystick to be centred, almost as if we planned this.

Test fit of DIN connector and joystick

We then solder and heatshrink some wires to it.

Preparing DIN connector connections

It would not be enough to wire the joystick to this connector and call it done. Noise and cable losses would ruin it. The plan was always to stick a microcontroller in here, which will read the joystick and transmit the digitised values. I couldn't start on this until the joystick was in my hands, right at the last moment, so this was cobbled together on protoboard.

Circuit board for joystick

The joystick has JST XH connectors so I used one for the DIN connector too. We'll send 5V down the cable, then linear regulate that to 3V to provide a nice stable reference for the potentiometers. The regulator is a tiny SOT23 part in the corner. Another CH32V003 dev board uses its ADC to read the values when one of its pins is pulled low, and then transmits the raw numbers over UART (at, say, 31250 baud...?). We'll need a deadzone but that can be set on the motherboard. The 2-pin connector is for the LEDs on the joystick.

If you're interested in the dual-footprint dev board for the CH32V003 the design files are here.

The underside:

Underside of joystick circuit board

This gets a strip of velcro for a removable attachment inside the enclosure. Overall a pretty neat job eh?

Parts connected inside of joystick enclosure

While the software on that micro is trivial, there's still something thrilling about writing it without testing, sealing up the box and hoping for the best.

Motherboard and power bus

At the heart of the installation is an STM32H7R3, a chip I've used a lot lately even if I've not published much with it. Don't be fooled by the H7 badge, that's just a marketing thing, the H7R3/H7S3 are kind of a different family to the others. All that matters is it's very fast, reasonably cheap, has a smidgen of onboard flash and is available in TQFP so we have a hope of soldering it ourselves. Base clock speed is 600MHz, and the only supporting parts we need are a few capacitors.

One thing I've learned is to always break out the BOOT0 pin. The memory protection stuff on these chips is aggressive, even when it's supposedly disabled, and the slightest problem causes it to lock down the debug interface for security, effectively bricking the chip until you use the system loader to restore it. This has caught me out so many times it's worth installing a switch on the BOOT0 line. The rest of the board is simple, just a 3.3V regulator, the MAX485 to transmit to the panels, and a secondary UART to talk to the joystick. Instead of linear regulating directly from 12V, I stuck a buck converter on the board, but rather than do it with discrete components, it's much faster to grab a ready-made step-down module.

Partially assembled motherboard showing step-down module

We set the trimpot to 5V before we soldered it down.

Finally there's some LEDs and, just in case we need it, an accelerometer (U2). I wish you could get them in bigger footprints, LGA is a right pain to solder.

Motherboard assembled and powered up

Until now, we'd been testing the panel in chunks with a USB-RS485 adapter. Before full scale testing we need to sort out the power distribution. Instead of a big busbar it was quickest to use wagos. Velcro will keep them tidy.

Routing power with wago connectors and velcro

Apparently you can get a 10-way wago now, but 5-way is all I had, so we'll do two groups. This will also help if we need to split the power into separate domains later.

I made some three-way splitter cables to direct power to the groups and also bleed off for the motherboard. The trick is to smoosh the wires together, wrap tightly in more wire, paint it with flux and then flood it with solder. I used two irons simultaneously to give it enough heat.

Soldered joint in power cable

Meanwhile the motherboard got a backing of EVA foam and was screwed into place. There's motivation to get the accelerometer centred on the panel, or we'll get spurious readings when the panel is rotated. Not that I expect the panel to see much spinning.

Power lines mostly connected, motherboard attached

If the pace were more sedate I'd have stuck a polyfuse on the cable where the gauge changes (the loose motherboard connector on the left in the image above). I think it's fine, if a serious fault develops on the motherboard, that short bit of wire will probably burn itself out as a poor man's fuse.

Running first from our insane LiPos, and then from our customised power brick, the first full test patterns flipped to life. A nice thing about the DIP switches, we can give some panels the same ID for testing purposes.

Test pattern repeated four times

The next few hours were spent shooting test patterns out my laptop down the USB-RS485 adapter and figuring how our bits should be mangled. The pixel mapping would have been so much simpler if the panels had even one dimension divisible by eight.

A handful of newly dead pixels arose. A bunch of dots were stuck along the bottom edge simply by getting caught on the frame, I should have given a few mm more tolerance. Others failed for more complex reasons, but the first concern is whether power distribution is safe under maximum flippage.

Thermal image of power distribution, wagos look cold, power switch is at 42.9C

Man I love wagos. They're still at ambient temperature, you can barely make them out in the thermal image. To be fair they're only taking half the load. The soldered joints too are at room temperature. In contrast the spade terminals on the power switch are 43°C, and I don't think that's the switch contacts, that's the actual spade terminals being resistive. The XT60 connector is also warmer than I'd like, though we're nowhere near its current rating.

Thermal image of XT60 connector at 36.3C

We plugged in the motherboard, and the LDO gets hot, but less hot than if it were dropping from 12V.

Thermal image of motherboard operating, regulator is hottest

Once the pixel mapping was established, coaxing the motherboard to talk to the panel was gruntwork. Our fluid sim has already been ported to this chip, so the only remaining task was to read the joystick. Delightfully, all was well, we even avoided the traditional mistake of swapping RX and TX.

Finally I chucked a baby heatsink on the processor, shown here with the panel now mounted to the stand:

Motherboard fitted with heatsink

Supported by the stand, we slightly loosened the frame and the dead pixels on the bottom row came back to life. Some of the pixels were properly dead though. With very little time remaining, I made the dubious decision to dismantle everything and investigate the problem. I expected soldering mistakes, but to my horror, near the standoffs on the last few panels we built, the coils themselves were damaged!

Closeup of a coil near a standoff with visible damage

I must have been overconfident with the tweezers towards the monotonous end. Our options here are limited, we have spare parts, but desoldering the full group of seven with the double circuit board behind it is going to be dastardly difficult. I opted to repair the coils, gently unwinding them with tweezers, removing the broken sections, identifying the loose ends and soldering them together.

Repaired coil damage

The coil may have lost a few turns but it still functions. Afterwards we wrapped this in paper to protect it from the sticky tape that normally lines the panel, then tested and reassembled.

One of the fiddliest bits of soldering I've done in a long time. I then repeated this repair four times.

Is our flipdot display finished?

Full view of rear of panel mounted on stand

Logo

It's important! We can't skip putting a pretty sign on it. I laser cut a design once in 3mm plywood, painted black, and once in 5mm painted silver.

Laser cut backing of logo

The silver positives were glued into the black negatives, with the whole thing mounted on brackets velcro'd in place.

Logo finished and mounted

And in case anyone wonders what the joystick is for, a nicely laminated label sets things straight.

Joystick labelled as Gravity Control

A slightly overpriced industrial-looking "speaker stand" serves as the pedestal.

The Event

Chatting with the Arts team, it was quite hard to decide where the flipdots should go. A lot of the interactive art ends up in Null Sector (a kind of cyberpunk rave area built from shipping containers) or the bar, or the arcade, all of which are such noisy environments we wouldn't be able to enjoy the flipping noise. I was offered a dedicated half-dome, but the weather forecast predicted thunderstorms with torrential rain and I was worried the dots would be too exposed.

About the only place guaranteed to be quiet was the "lounge" tent, where noisy exhibits are outlawed. Not everyone was pleased about me setting up there, but it was hard to judge until the thing was turned on. I figured I'd set up and if it's deemed too loud we'll move it elsewhere. First task was to level the feet and secure it with these 10" pegs.

Feet levelled with wood and pegged down

I wrapped the (extra long) joystick cable around the legs to reduce the chance of it yanking the thing over.

Ultimately the weather turned out swelteringly hot and dry. So hot that I had a genuine concern the thing would catch fire. To my delight, on the first day someone donated a pile of old server fans to the EMF swap shop! I chose the quietest of them and rigged them to the back, shown here after they'd been running for a few days, hence the dust:

Fans fitted to back of installation, dusty

Even the adjustable power supply driving the fans came from the swap shop. In return, I donated a bunch of soldering equipment that got gobbled up right away.

The installation ran without fault for the full four days of the event. A bunch of dead pixels appeared, mostly from people poking it I think, or just dust and dirt jamming it.

I was able to walk away, to see the rest of the festival and worry about my talk, and almost every time I returned people were playing with it.

Finished view of installation in lounge, in use

Early morning or late at night, where the joystick was untouched for an hour or more, it would spread out into these patterns with the fluid in zero gravity. I left the viscosity at zero, so the only way for the fluid to lose energy was by colliding with the walls. It takes a long time to settle.

Finished view of installation in lounge tent with the joystick to one side

The joystick saw so much use that a grimy wear pattern appeared. Additionally, by the end of the event the screws on the underside of the enclosure had loosened. It might have been some kind of precession effect, where people spinning the joystick in one direction caused the screws to unwind.

Joystick on last day, showing lots of use

I regret not making a plaque explaining the pun and how the flipdots work. I did write a blurb but it never got printed out as the Arts team were somewhat overwhelmed. EMF is extremely popular, tickets sold out almost immediately, and one of the only ways to guarantee a ticket as a newcomer is to have an artwork or talk accepted. This year there were more proposals (and acceptances!) than ever before.

EMF is a magical place, it's one of the rare situations in the UK where people can be whoever they want to be, without fear of being judged. You can be as weird as you like, and apparently, what I wanted to be, was someone taking their flipdots for a walk.

photo by tef

Photo by tef. Hokusai would be proud.

I hadn't abandoned my gimbal dreams though. Remember that last idea to use a big squishy ball? It wouldn't be perfect, but when I realised how cheap and easy it was to try, I was sold.

Gymball gimbal

The fact that inflatable exercise balls are referred to as a "Gym ball" was icing on the cake. Gymball gimbal!

Gymball gimbal

Conclusion

The length of this writeup conveys only a fraction of the tedium in all that soldering.

The total cost for the big panel, not including the donor parts or my time, was under £500, or around £0.17 per dot. If you factor in my time, well... it's clear why the commercial offerings will run you five or six figures.

If you happen to have a 13x28 Hanover flipdot display, I've put my PCB designs for driving them up on github. Sam has a load more panels in storage, but they're all different sizes, so the next step is to adapt the design, and then convince some volunteers to do all the soldering for us.

As for this panel, I have a few ideas. Since they're non-volatile, it would have been great to build the circuitry in a way that could read out the state of a dot. We could use the panels as core-memory with visible state. Or we could turn the panel into any number of video games, I'm sure Tetris or Snake would be great fun. A flipdot arcade machine.

I have another idea I think would be great, but I'll keep it a surprise until it's posted, hopefully soon.

Hello! Recently I needed bike lights for my bike. And I remembered that I already had rechargeable bike lights that I bought ten years ago, that I hadn’t tried in a long time. I tried to recharge them, but after fully charging them, they only worked for maybe 5 minutes before they turned off again.

I don’t know much about electronics, but I’ve been curious about whether it’s possible to fix old electronics for a long time, and this seemed like the perfect repair project because I might just need to replace the battery.

So I went to the local queer makerspace where I’m a member to use the soldering iron and try to do it! I don’t know much about electronics and this post does not contain any safety advice because I don’t know much about safety. I think it’s nice to do projects in a community space where you can get help.

step 1: cut it open

The bike light felt like it was made of silicone, so I cut open the silicone in a haphazard way along something that vaguely looked like a seam.

I definitely ripped some silicone in the process and it was pretty messy but I got it open and found the circuit board.

I don’t know the model number of the bike lights but there’s a photo of them at the end of the post.

step 2: remove the screws

There were some screws attaching things together so I removed them so I could get the circuit board out.

Mostly I tried to remove as few screws as possible because I was worried about losing them or not being able to put them back after. I probably put the screws in a bag or something.

step 3: get the circuit board out

I took out the circuit board. Here’s what it looked like:

You can see where the battery is attached, I think it’s left of RI3 and above Q2.

Here’s what the battery looked like:

step 4: desolder the battery

I’d never desoldered anything before, so I found the iFixit guide to desoldering and read it. Also I asked my friends Lee and Lauria for advice.

Here were the steps I ended up following based on the guide & the advice I got:

1. Use a desoldering pump to remove most of the solder
2. Once most of it is gone, kind of pull them apart to try to separate them
3. Also try to avoid getting the battery too hot in the process by taking breaks to let it cool down. I’m not very good with a soldering iron so it took a while.
4. The battery has an attachment that is welded to the top. For a while I thought I needed to remove this and it seemed impossible, but it turned out the replacement battery comes with that part so actually I was supposed to leave it alone.

step 5: identify the battery

In the picture of the battery in Step 3, you can see it says something like “3” and “LI???77”. There’s a piece of metal that I think is welded or something to the top of the battery. It seemed impossible and also maybe not smart to try to remove so I wasn’t sure how to find out what an “LI????77” was or how to order another one.

I’ve been trying to avoid using LLMs (though I will not get into that because I am exhausted by LLM discourse and I’m sure you are too), but I really had no idea how to figure out what the battery was so I asked an LLM. It gave the response “LIR2477”, which (when I looked it up) looked exactly the same as my battery so I figured that was plausible.

I would be interested to learn non-LLM ways to figure this out though. There must be a way. Lauria showed me how to use DigiKey’s search which was very cool though DigiKey didn’t have that part.

step 6: buy the battery

I went to AliExpress and ordered:

1. 2 batteries (I had 2 bike lights and I wanted to fix them both)
2. some silicone glue to glue things back together

I think the batteries were $3 each and the glue was $8.

step 7: solder the new batteries in and glue it back together

The parts took maybe 2 weeks to arive, and once they arrived, I went back to the makerspace and:

• soldered in the new batteries
• put the screws back in. The screws were very small and hard to hold, so at this point I dropped some screws on the ground and couldn’t find them because they were too small. So I just used fewer screws and hoped for the best.
• used the glue to try to put everything back together.
• Make a somewhat halfhearted attempt to clamp the parts I was gluing together

Then after waiting some amount of time for the glue to dry I took it home and waited 24 hours for the glue to cure.

Also I took the old batteries to somewhere nearby that accepts old batteries.

it works!

The lights work! I have used them to bike at night! I still haven’t needed to recharge them (and tragically I had to order a new Mini USB cable because I got rid of all my Mini USB cables, so I’m still waiting for that), so I still don’t know for sure how long the lifetime of the new battery will be.

Here’s what the light looks like after re-gluing. You can see that I didn’t glue very carefully. It didn’t really go back together that well but I’m hoping it’ll be good enough.

I thought it was really cool that I was able to do this with extremely minimal electronics skills! It cost about $20 CAD to buy the parts, and (whether or not the repair holds up, I’ll try to update this post in the future!), it was fun to try to repair something and learn something new.

About Lofi Cities

Lofi Cities is a free web app that pairs animated pixel-art city nights with endless lofi music generated live in your browser. Pick a city and let it run: every scene is a 480×270 pixel animation that loops seamlessly every four minutes, with its own weather, landmarks and city sounds, while new lofi tracks are composed on the fly.

There is nothing to install, no account and no pop-ups. It works on computers, phones and tablets, and makes a calm background for studying, working or winding down.

Every city has a billboard for rent, with a made-up local ad on it until a business takes it. Put your business on one ↗

What's new

• 28 Sep Emoji in the chat: the smiley button beside the text box opens a small set of cozy ones, and typing a colon and a name, such as :coffee:, lists the matching emoji as you type.
• 28 Sep Chat with the people on your city: the Chat button in the player (or Enter) opens short messages from the last hour, under a random two-word name such as "sleepy fox". No links, and you can hide anyone.
• 28 Sep The city picker (/) features the city with the most people right now, in a wide card above the others, and lists the cities with the most people first.

All updates

Cities

• Paris Rain on the attic window

  A painter's attic under the zinc roofs, rain on the window and the Eiffel Tower sparkling far off.

• Tokyo Neon rain by Tokyo Tower

  Neon signs, cherry blossoms and wet asphalt, with Tokyo Tower glowing and a train crossing the viaduct.

• New York Snow over Brooklyn Bridge

  A snowy winter night by the Brooklyn Bridge, with the Manhattan skyline across the East River.

• London Fog on the Thames

  Fog and drizzle on the Thames, with Parliament, the London Eye and a Victorian pub on the South Bank.

• Rio de Janeiro Copacabana summer night

  A balcony above Copacabana, with Christ the Redeemer, Sugarloaf and fireworks over the bay.

• Istanbul Bosphorus by night

  A rooftop tea garden facing the historic peninsula, with ferries and gulls on the Bosphorus.

• Hong Kong Victoria Harbour after dark

  A crowded Kowloon rooftop facing the Hong Kong Island skyline, with ferries, a red-sailed junk and a light show on the harbour.

• Sydney Jacaranda night on the harbour

  A harbourside park under a blooming jacaranda, looking across to the Opera House and the Harbour Bridge.

• San Francisco Fog over the Golden Gate

  Fog pouring through the Golden Gate, seen from a steep Russian Hill street with a cable car and painted Victorians.

• Hamburg Harbour lights on the Elbe

  Wind-driven rain over HafenCity, with the Elbphilharmonie, the Speicherstadt and container cranes across the Elbe.

• Amsterdam Bridge lights on the canal

  A mild spring night on a canal quay, with the Westerkerk tower, gabled houses, a houseboat and a bulb-lit bridge.

• Dubai Fountain night at the Burj

  A mild winter night on a café terrace by the Burj Lake, with the Burj Khalifa, the dancing fountain, a lit souk and moonlit dunes beyond.

• Madrid Summer night on Gran Vía

  A warm June night on a wrought-iron balcony above the fork of Gran Vía and Calle de Alcalá, facing the floodlit Metrópolis and a rooftop neon for churros.

• Rome Summer night by the Colosseum

  A warm August night in a cobbled Monti lane running down to the floodlit Colosseum, with a trattoria and shooting stars.

• Prague Lamps on Charles Bridge

  A mild June night on the Old Town embankment, with Prague Castle lit gold above Charles Bridge, swans on the Vltava and a red tram rattling past.

• Munich Christmas market in the snow

  A snowy December night at the Christmas market on Marienplatz, with the Neues Rathaus, the Frauenkirche domes and a giant lit tree.

How the music works

Every track is composed on the spot with the Web Audio API: a chord progression, a drum groove, a bass line and often a melody, finished with tape wobble and vinyl crackle. Everything is synthesized in real time, with no samples or recordings, so the stream never runs out.

The Vibe panel picks the style: jazzhop (the default: a Rhodes, jazzy chords and a swung boom-bap beat at 68 to 88 BPM), piano (a sleepy felt piano with soft drums), ambient (slow warm pads with barely a beat, for sleep and deep focus), bossa (nylon guitar over a bossa nova groove), synth (80s city-pop synths), house (lofi deep house on a soft four-on-the-floor kick), guitar (a clean electric guitar, fingerpicked over a lazy beat), sad (slow minor chords and a lonely muted trumpet) or medieval (a lute and a recorder over hand drums). A new style starts with the next track.

It also sets the mood, chill (slower and darker) or upbeat (faster and brighter), and plays the full band, no drums, or the chords only.

Track titles are inspired by the city on screen, and a separate ambience layer mixes that city's sounds under the music. The Up next list shows the coming tracks so you can pick one to play right away, and music and city sounds each have their own volume slider. The city sounds mixer goes further, with a slider for each sound of the city: turn the rain up, the traffic off or the bells down, and the setting carries over to every city with that sound.

The Weather chip (W) brings rain, snow, autumn leaves or a clear night to any city, and the city sounds and track titles follow it: rain brings the sound of rain in, snow a hush of wind.

How the music on Lofi Cities is made

Keyboard shortcuts

← → 1–9 0

Switch city

/

All cities: a preview of each with its local time, and a search

C

City tour on or off: a new city every few minutes, or follow the night

W

Weather: the city's own, rain, snow, autumn leaves or clear, on every city

Space

Play or pause the music

N

Next track

Q

Up next: see the coming tracks and pick one

A

City sounds mixer: a slider for each sound of the city

V

Vibe: the style, chill to upbeat, full band, no drums or chords only

T

Timers: focus (25/5 or 50/10, on repeat) and sleep (fade out and pause)

G

Together: reactions and lanterns

L

Release a lantern over the city

Enter

Chat with the people in the same city (in its text box, : and a name adds an emoji, such as :coffee:)

M

Mute

F

Fullscreen (or double-click the scene)

P

Mini player: the scene floats in a small window over your other apps

H

Hide or show the interface

S

Save the current frame as a 1080p PNG

?

Open or close this panel

Questions

What is Lofi Cities?

A free website at loficities.com that shows animated pixel-art city nights (Paris, Tokyo, New York, London, Rio de Janeiro, Istanbul, Hong Kong, Sydney, San Francisco, Hamburg, Amsterdam, Dubai, Madrid, Rome, Prague and Munich) with endless lofi music composed live in your browser, plus each city's ambient sounds.

Is Lofi Cities free?

Yes. It runs entirely in your browser, with no download, no account and no pop-ups.

Where does the music come from?

It is generated in your browser while you listen. Chords, drums, bass and melody are synthesized with the Web Audio API, without samples or recordings, so every track is new and the music never runs out. The Vibe button (or V) picks the style (jazzhop, lofi piano, ambient, bossa nova, synth city pop, lofi house, chill guitar, sad lofi or medieval lofi), makes it chill or upbeat, and plays the full band, no drums, or the chords only.

Why doesn't the music start right away?

Browsers block sound until you interact with a page. Click, tap or press any key and the music starts; the play button pauses and resumes it.

Can I use it as a background for studying or working?

That is what it is made for. Press F for fullscreen and H to hide the interface, or P for the mini player: a picture-in-picture window that keeps the scene floating over your other apps (on a computer, in browsers that support it). The controls also fade out on their own when you stop moving the mouse.

Can it switch cities by itself?

Yes. The Tour chip next to the city picker (or the C key) moves on to the next city every 5, 10, 15 or 30 minutes, or follows the night: it shows whichever city is closest to 1 am local time, moving west from Sydney to San Francisco as the hours pass. Cities on the same clock, like Paris, Hamburg and Amsterdam, take turns.

Can I change the weather or the season?

Yes. The Weather chip next to the Tour chip (or the W key) brings rain, snow, falling autumn leaves or a clear night to any city, or goes back to the weather the city was drawn with: winter snow in New York, autumn in London and Hamburg, spring blossom in Tokyo and Sydney. The city sounds follow it, so rain brings the sound of rain in, and your pick stays for the next visit.

How many people are listening right now?

The glowing pill under the city name shows it, for example "23 here now · 5 in Tokyo · 480 today · 12,345 all time": people with Lofi Cities open right now, how many of them are on your city, browsers that came by today, since midnight where you are, and every browser that has come by since the launch. When nobody else is on, it shows the rest ("480 here today · 12,345 all time"). The city picker (/) shows how many people are in each city right now. You can say hello to the people on your city too: pick a heart, note, star or moon with the Together button in the player (or G), then tap the scene to float it up from that spot on everyone's screen there, or release a paper lantern (L) that rises over the city for about a minute, in the same place for everyone, including people who arrive while it is in the air. The Chat button beside it (or Enter) lets you talk with the people on your city: short messages of up to 80 characters, without links, under a random two-word name such as "sleepy fox" that stays the same for your browser. The smiley button beside the text box adds emoji, and so does typing a colon and a name, such as :coffee: for a cup of coffee. There are no accounts: a random id in your browser's local storage (not a cookie) makes your tabs count as one person, the server forgets it about 25 hours after your last visit, and other people only ever see your two-word name. The all-time count is a single number: your browser notes once that it was counted, and the server keeps no list of who. Your page also tells it when your day began, so "today" starts at your midnight, and that is forgotten when you leave. Besides that it keeps the lanterns still in the air and each city's last 30 messages, for an hour at most. You can hide anyone's messages on your screen; when a few people hide the same person, their messages go for everyone and they can't chat for an hour.

Is there a Pomodoro or sleep timer?

Yes. Press T or the clock button. The focus timer runs 25 minutes of focus and 5 of break (or 50 and 10) on repeat, with a soft chime at each break and a bell to call you back. The sleep timer fades the music and city sounds out after 15 to 90 minutes, then pauses.

Can I use Lofi Cities as a stream background in OBS?

Yes. Add a Browser source set to 1920×1080 with a city page's address followed by ?obs, for example https://loficities.com/tokyo/?obs. The interface stays hidden and the music starts on its own; tick Control audio via OBS to bring it into your mix. Interact with the source to switch city with the arrow keys, skip a track with N or change the weather with W, or add &weather=snow (rain, leaves or clear) to the address.

Does it work on phones and tablets?

Yes. In portrait orientation every city's landmark and billboard stay in view, you can drag the scene sideways to look around it, and swiping past its end switches city. All controls work with touch.

Does it work offline?

Yes. Open it once while online and it keeps working without a connection, every city and the music included, since both are generated in your browser. You can also install it as an app with your browser's Install or Add to Home Screen option (on iPhone and iPad: Share, then Add to Home Screen).

Can I save a scene as a wallpaper?

Yes. The save button in the player (the down arrow) saves the current frame as a 1080p PNG, and S does the same with a keyboard.

Can my business appear in Lofi Cities?

Yes. Every city has a billboard for rent, where your café, shop, studio or brand can be drawn in pixel art. Fill in the short form at tally.so/r/rj7bY5 to get in touch.

Who made Lofi Cities?

Lofi Cities is made by Safa Elmali. You can support it on Buy Me a Coffee using the coffee link at the top right.

Can I request a new city?

Yes. Send your idea to Safa Elmali on X at https://x.com/SafaElmali (Request a city, at the end of the city picker, or the X link at the top right of the site). New cities are added over time: Hong Kong, Sydney, San Francisco, Hamburg and Amsterdam joined the first six, then Dubai, Madrid, Rome, Prague and Munich.

Can I buy the city videos?

Yes. Get the 16-city collection on Gumroad through the Get video loops link or the save menu. Each AI-assisted video is a four-minute 1920×1080 MP4 loop at 60 fps, without audio. Personal use and video or livestream backgrounds, including monetized content, are allowed with the Lofi Cities credit specified on the product page. The music and city sounds you hear on this site are not included in the downloads. The website stays free.

Does Lofi Cities use cookies?

No. There is no sign-in and no cookies. Anonymous usage statistics, such as which cities are viewed, are collected without cookies, and browsers that send Do Not Track are left out of them (the live count still counts the tab, anonymously). Your settings (such as the last city, volumes and vibe) and anonymous random ids for the statistics and the live count are kept in your browser's local storage instead.

Planetary scientists from Freie Universität Berlin have published two new studies in Science Advances with promising discoveries about the potential for life on one of Saturn’s icy moons. New evidence indicates that Enceladus might be more supportive of life than once assumed. In fact, spacecrafts already in planning could be used to identify evidence of life there much easier than previously thought, should it exist.

A photograph of the icy moon Enceladus taken by the space probe Cassini.
Image Credit: NASA/JPL/Space Science Institute

What are the chances of finding extraterrestrial life in our solar system? Professor Frank Postberg, planetary scientist at Freie Universität Berlin, has just published a study in Science Advances with a team of international researchers, in which they present new evidence that it is easier to determine the constituents of the ocean hidden under Enceladus’s icy surface than previously assumed. Postberg was also part of a second study published on the same day, in which he and another planetary scientist from Freie Universität, Dr. Nozair Khawaja, contributed to research that reveals that certain microorganisms could actually tolerate the conditions in Enceladus’s ocean better than was previously thought. These results increase the probability of finding evidence of life on Saturn’s moon.

What Makes Enceladus so Attractive in the Search for Life in Space?

Enceladus is considered to be one of the most promising places to search for extraterrestrial life in our solar system. Researchers suspect that under the moon’s icy crust there is a global ocean of liquid water and a rocky core further below. Due to cryovolcanic activity, gigantic plumes break through cracks in the crust at the moon’s south pole, ejecting ice particles hundreds of kilometers into space.

NASA’s Cassini spacecraft passed through these plumes multiple times to analyze its composition. Enceladus’s ocean is the only extraterrestrial “body of water” from which scientists have been able to directly analyze samples. The samples revealed traces of various salts and organic compounds. Furthermore, previous analyses conducted by Cassini gave indications of hydrothermal processes on the seafloor and other conditions that are significant in supporting life.

Enceladus Ice Plumes Separate and Sort the Ocean’s Components

Postberg’s study, “Cassini CDA Observes Compositional Segregation of Enceladus’ Ice Grains from Slow Freezing and Fragmentation of Oceanic Spray,” includes surprising findings about what happens to the ocean water on its way into space. The international team of scientists working on the study used Cassini data, long-term laboratory experiments, and theoretical models to reconstruct the process.

Droplets form at the ocean’s surface as bubbles filled with gas float up and pop. Water vapor then carries the droplets through cracks in the ice shell out into space. Until now, scientists believed that they froze instantaneously, however these new findings reveal that they freeze slowly. Due to this slow process, most components (including dissolved ones) separate from each other. Salts and organic materials are thus distributed at different locations inside each freezing droplet. Various types of previously dissolved salts also get segregated in the process; for example, sodium chloride (table salt) separates from sodium carbonate.

Enceladus ejects ice particles hundreds of kilometers into space. This results in salts and organic materials being separated and distributed at different locations inside each droplet.
Image Credit: Professor Frank Postberg

On their way up, the frozen droplets are accelerated to speeds of up to 1,000 km/h. If they smash into the walls of the icy cracks, they break into fragments only a few micrometers in size before they shoot into space. This results in the ice particles often consisting of just one highly concentrated, previously segregated, substance.

“Enceladus actually does a lot of the work for us in preparing samples for analysis that usually take a lot of effort in chemical labs on Earth,” says Postberg, who led the study. “The oceanic constituents are separated from each other and simultaneously concentrated into individual ice particles.”

Making It Easier to Find Signs of Life

When probing Enceladus’s plumes, this mechanism is not just helpful in characterizing the ocean as a potential habitat for life, it is also particularly interesting in the search for biosignatures, i.e., measurable indications of life. If one of these ocean droplets contained components from alien microbes, they could be segregated from other components in the freezing process. After fragmentation, the microbial material would potentially only be contained in a small fraction of ice particles; however, in those ice particles it would be found in high concentration and in relatively pure form.

“That is great news in the search for life,” says Postberg. “Future spacecrafts will have to analyze many individual ice particles in the plume. But if they come across one with microbial material in it, they could identify biosignatures in the particle relatively easy with already available technology.”

This discovery could have important implications for future space missions to Enceladus, such as the ESA’s L4 mission, which is currently in planning. The mission will be specifically looking for signs of life on Saturn’s moon. Postberg’s lab at Freie Universität Berlin has previously conducted laboratory studies that demonstrated the capacity for specialized instruments to detect microbial cellular material in individual particles from the ice plumes. 

Could Life Really Exist on Enceladus?

A recent study is also providing new insights into this question. On the same day Postberg’s article appeared in Science Advances, another article in the same journal was published by scientists at Ludwig-Maximilians-Universität München (LMU): “Enceladus-Like Geochemistry Fuels Methanogenesis under Extreme CO₂-Limitation.” Postberg and another colleague from Freie Universität Berlin, Dr. Nozair Khawaja, contributed to this study. In it, scientists were able to reproduce the conditions of Enceladus’s ocean in a laboratory. The ocean has a very low concentration of oxygen, a very high concentration of carbonate, and is very alkaline (with pH values of 10 or 11). After recreating these conditions, including its hydrothermal interaction with the rocky ocean floor, they introduced Methanothermococcus okinawensis into the contained environment. This microorganism is a methane-producing archaean that normally lives near deep-sea hydrothermal vents on Earth. They do not need oxygen, which is rare on Enceladus, to survive. Instead their metabolism requires only hydrogen and carbon dioxide.

The results were surprising: The organism failed to grow in an optimum laboratory medium at such a high pH, lacking dissolved carbon dioxide. By contrast, in the Enceladus simulant it continued to grow, producing methane using hydrogen generated by water-rock reactions. Under the simulated Enceladus conditions, the microorganisms were even able to adapt their metabolism to the low amounts of carbon dioxide. “This was really a surprise to us,” Khawaja said. “This was an experiment for which we did not expect such a successful outcome.”

Taken together, the two studies in Science Advances shed new light on the search for extraterrestrial life. “On Enceladus the specific geochemical conditions might allow one of the oldest known metabolic systems on Earth to work, even in very alkaline environments,” Postberg says.  “While that doesn’t mean that there is life on Saturn’s moon, our first study shows that – in the event that there is – future space missions might have a good chance of finding traces if they analyze individual ice grains from Enceladus’s plume.” 

Freie Universität Researchers Investigating the Emergence of Life in Space

What are the prerequisites for planets and moons to provide suitable conditions for the formation and preservation of life? This is a central research topic for planetary scientists at Freie Universität Berlin. A new Collaborative Research Center funded by the German Research Foundation (DFG), CRC 1759 “Habitability as a Fundamental Planetary Process,” was launched in July 2026. The research center is led by Professor Lena Noack with Professor Frank Postberg as her deputy, both of whom are planetary scientists at Freie Universität Berlin. The research center will study the processes that could have enabled life to emerge in space in the first place, on Enceladus and other celestial bodies, both in our solar system and beyond.

Further Information

Publications

• Frank Postberg et al.: “Cassini CDA Observes Compositional Segregation of Enceladus’ Ice Grains from Slow Freezing and Fragmentation of Oceanic Spray.” Science Advances (2026): https://www.science.org/doi/10.1126/sciadv.aee7256
• Vanessa Helmbrecht, Frank Postberg, Nozair Khawaja et al.: “Enceladus-Like Geochemistry Fuels Methanogenesis under Extreme CO₂-Limitation.” Science Advances (2026): https://www.science.org/doi/10.1126/sciadv.aei0167
• CRC 1759 website: https://www.geo.fu-berlin.de/en/geol/fachrichtungen/planet/projects/SFB1759/index.html

Uncovering a century-old misinformation

Have you ever asked yourself, “What is the size of Yemen?” Yeah, me neither. At least, not until I received this peculiar comment under my video.

I became intrigued, so I googled the size of Yemen, and voila, he wasn't lying. 555,000 sq km – as the first Google response with the same number used by Wikipedia. I wanted to see what went wrong, if this person was right or not, so I tried replicating the borders by drawing them on Google Maps. And…. I arrived at 456,000 km2.

I included all the islands, including Socotra, to ensure I didn't miss anything. There was no possibility of some small border error – the missing area was the size of a quarter of the country.

Where 555k km2 came from

So how did the 555 claim originate? With the help of Google Books, I managed to find the oldest mention possible:

Source: Statistical Year-book - Yemen Arab Republic, Central Statistical Office, 1991

This is a statistical yearbook published by the Central Statistical Office of Yemen in 1991, the first one after the unification of the country. When I read the paragraph carefully, I noticed one slight problem. Yemen does not stretch into the 20th North latitude. In fact, the northernmost point of Yemen is exactly at the 19th altitude, as per the border agreement with Saudi Arabia.

So what happened here?

Before 2000, the border of Yemen with Saudi Arabia was not clearly demarcated. So, it’s quite possible that some official drew the northern border up to the 20th altitude for propaganda purposes to make the country as big as possible. When I plugged the new borders into Google, my assumption seemed to have worked out.And yes, this area includes the Rub-Al-Khali desert despite what the text says.

Where 527k km2 came from

"Well, it's a case of Wikipedia using bad sources – a very common occurrence indeed. Surely, other sources must have gotten it right?" So I looked into the most reliable source: the CIA World Factbook, which claimed that the size of Yemen is 527,968 km2. The same number can be found in the World Bank database, the United Nations website, as well as an old geography book that I found in my closet. But it still diverges from the Google Maps number. Why? I delved into previous editions of the CIA World Factbook and found the earliest editions where the number appeared – 1991, exactly a year after the unification of Yemen happened.

I looked into previous editions and found separate entries for North and South Yemen – with the first one being 195,000 km2, and the second one being 332,968 km2. Both numbers were summed up to form the total size of United Yemen. But were they correct? I plugged the borders used in the textbook into Google Earth, and to my surprise, I found out that none of them were correct – North Yemen only had 136,000 km2, and South Yemen was only 288,000 km2. There was no possibility of some minor border or rounding error. They were both wildly off. What was happening here?

North Yemen’s size

I started with North Yemen. I was looking for the earliest mention of the number 195,000 sq km2 in relation to North Yemen. And… after a lot of digging, I found it – the 1964 edition of the Statesman Yearbook described it as 75,000 sq. miles (195,000 sq. km). This gave me a hint to search for the area in square miles. After a long search with the help of Google Books, I found the earliest mention.

The 1904 edition of the New International Encyclopedia described the size of the Yemeni Vilayet, a province of the Ottoman Empire. When I plugged its border into Google Earth, it fit perfectly into 195,000 km2. Despite the fact that independent North Yemen was half the size of the vilayet, every book until 1990 continued to show its size from the 1904 source, repeating it without any fact-checking! The number has appeared on every single mention of North Yemen I have found online, despite being based on a completely different entity..

South Yemen’s size

So after clearing that one up, I started looking into South Yemen. I found a peculiar pattern. Every source up to 1976 showed South Yemen’s size as 287,000 km2 (here, here, here). CIA Factbook changed it to 332 in 1986. Encyclopedia Britannica did it in 1981. The United Nations did it in 1977. What happened in 1976 to cause such a sudden expansion? Nothing about South Yemen’s size changed. After much searching, I found it: Buried among other documents was an obscure World Bank report about South Yemen’s economic situation from October 1976.

Buried on the 51st page was a table with South Yemen’s governorates and their areas summing to 338,100 km2. I used the administrative map contained in the annex to the report and Google Earth to calculate their sizes on the map when I found a problem. The seventh item on the list, Thamud, wasn't a governorate… It was a part of the Fifth Province, as could be seen in the map. Its area had been counted twice in the table and led to a massive overestimation of South Yemen’s size. And since then, every source has repeated this erroneous claim without double-checking.

Closing thought

To wrap it up - for two unrelated reasons, the sizes of both North Yemen and South Yemen have been exaggerated, and during their unification, sources simply combined them to create a new, even more inaccurate size.

I wish I had made it all up; to call it embarrassing is an understatement. Because for 100 years, no one bothered to fact-check their geographic tables. Even every single online statistic that contains the size of the country is… wrong.

• No, Yemen's population density is not 65/km2. It’s 75.
• No, Yemen is not the 49th biggest UN member state. It's the 54th.
• No, Yemen is not “slightly larger” than twice the size of Wyoming as CIA claims; it is smaller, in fact.

At this point, I don’t know how a mistake of this size could be fixed. Maybe someone watching my video could get this story famous and get all these institutions to finally correct this century-old error? Imagine how cool that would be.

Archaeologists excavating in the ancient Hittite capital of Hattusha have uncovered a fragment of the world's oldest peace treaty, signed around 1269 B.C.

Archaeologists have found a fragment of cuneiform recording the Treaty of Kadesh, which was signed by Egypt's Ramesses II and the Hittite king Hattusili III around 1269 B.C.

Archaeologists have discovered a 3,400-year-old fragment of the world's oldest known peace treaty, the Turkish Ministry of Culture and Tourism has announced.

The agreement, known as the Treaty of Kadesh, was signed around 1269 B.C. by Egyptian pharaoh Ramesses II and Hittite king Hattusili III. Other parts of the treaty have already been found, but the new fragment was unearthed during an excavation at Hattusha, in north-central Turkey.

"We have found traces of peace, thousands of years old, in the soil of Anatolia," Mehmet Nuri Ersoy, Turkey's minister of culture and tourism, said in a Sept. 21 translated statement. The newly discovered cuneiform tablet fragment contains text from a section of the treaty related to the treatment of refugees, according to the statement. "It reminds us of our responsibility to protect human life and dignity," Ersoy said.

Archaeologists unearthed the fragment from a building in the ancient Hittite capital of Hattusha. The Hittite Empire controlled central Anatolia from about 1650 to 1200 B.C. during the Bronze Age. Archaeologists have discovered numerous examples of diplomatic documents at Hattusha that were written in Akkadian, the earliest documented Semitic language, including other fragments of the Treaty of Kadesh.

The royal leaders signed the treaty around 1269 B.C. to mark the end of a lengthy war between the ancient Egyptians and the Hittite Empire, and it is regarded as the world's oldest known peace treaty. Although known as the Treaty of Kadesh, the document does not actually mention the Battle of Kadesh, which occurred several years earlier, so it is also called the Egyptian-Hittite peace treaty, the Eternal Treaty or the Silver Treaty.

The treaty was preserved by inscriptions in two ancient cities. In the early 19th century, archaeologists first discovered the treaty inscribed in hieroglyphics on the walls of a temple dedicated to Ramesses II in Thebes (modern Luxor). Fragments of the Akkadian cuneiform text of the treaty were first identified at Hattusha in the early 20th century.

Marking the end of a two-century-long war over lands in the eastern Mediterranean, the Treaty of Kadesh marks a military détente between Ramesses II and Hattusili III. According to a translation of the treaty provided by the Turkish Ministry of Culture and Tourism, it includes lines like, "He is a brother to me and he is at peace with me; and I am a brother to him and I am forever at peace with him," referring to the two leaders. "The country of Egypt and the country of Hatti [Hittite Empire] will be forever in a state of peace and of fraternity as it is with us."

The treaty has further provisions for coming to a country's aid if it was invaded and on how to treat refugees. The text mentions that any refugee from Egypt to Hattusha will be returned to Egypt, and any Hattusha refugee to Egypt will be returned to Hattusha, as fleeing was considered a crime.

However, the treaty specifically forbids punishing refugees: "Their tongue and their eyes are not to be pulled out; their ears and their feet are not to be cut off; their houses with their wives and their children are not to be destroyed." Part of this text appears on the newly discovered cuneiform fragment. Ongoing excavations may help uncover the remaining parts of the treaty, according to the statement.

A copper replica of the Treaty of Kadesh is displayed prominently today in the United Nations Conference Building in New York City. According to the U.N., "the treaty pledges eternal friendship, lasting peace, territorial integrity, nonaggression, extradition, and mutual help. The pledges in this treaty are similar to the United Nations ideals."

---

Are you a fan of mummies and hieroglyphs? Find out with our ancient Egypt quiz!