Today we are releasing the first release candidate for Polars 2.0. The definite 2.0 release will land in the following weeks. We don’t aim to make a big feature release of Polars 2.0. In fact we hope it to be a boring experience for you. The reason we bump this major version is that we can get rid of design decisions made in the past that currently block us and then we want to change defaults to more sensible settings that will benefit a greater audience. The biggest default change will be that all LazyFrame queries now will run on the streaming engine. Casual Polars users can therefore expect huge improvements in memory usage and performance. In aggregate we expect the streaming engine to be easily 5x faster.

To help users transition to 2.0, we have posted a full migration guide. This post will cover a few of the highlights.

Streaming engine as default

This is the biggest impact change of 2.0. Calling collect on a LazyFrame will now default to the streaming engine, leading to massive memory and performance improvements on most queries for users. The reason this required a major version bump is that the streaming engine doesn’t guarantee row-order by default for certain operations (join, group_by, unpivot, etc.). If you require observable row-order in those operations, you can opt in to that by setting maintain_order=True.

For users who want to keep using the “in-memory” engine as default, they can do so by setting the engine affinity.

lf = pl.LazyFrame({"k": [2, 1, 0], "v": ["a", "b", "c"]})
other = pl.LazyFrame({"k": [0, 1, 2], "r": ["x", "y", "z"]})

# 2.0: engine="auto" now resolves to the streaming engine.
# Row order is no longer guaranteed for joins, group_by, unpivot, ...
(
    lf
    .join(other, on="k", how="left")
    .collect()
)
# ┌─────┬─────┬─────┐
# │ k   ┆ v   ┆ r   │   <- order may not match `lf`'s original row order
# └─────┴─────┴─────┘

# Opt in to observable order for this query:
(
    lf
    .join(other, on="k", how="left", maintain_order="left")
    .collect()
)

# Or keep the old in-memory engine as the default, process-wide:
pl.Config.set_engine_affinity("in-memory")

# ...or per query:
(
    lf
    .join(other, on="k", how="left")
    .collect(engine="in-memory")
)

Stricter Polars

Polars aims to be strict and fail fast. Errors should ideally raise up-front, not 20 minutes into a pipeline. Implicit behavior on data-mismatches should be opt-in, not a default, since those mismatches can hide bugs. This strictness has become even more valuable with the rise of AI-driven development. Agents can validate a query’s structure early by calling collect_schema(), which resolves types and catches schema-level mismatches without materializing any data. This ensures fast feedback for the agents, meaning they can iterate faster. Not all errors can be caught during compilation of the query plan, some depend on data. In these cases Polars defaults to stricter behavior to ensure inconsistencies are caught instead of silently producing different results.

Below are a few examples where Polars has gotten more strict:

is_in lossless type-coercion

If you run an is_in expression on different data-types, Polars used to cast both types to their common supertype, even if that conversion was lossy Below is an example with user-ids that can go wrong by silent data-type mismatches.

# Checking if a user ID matches a list of "flagged" account IDs
# (flagged_ids loaded from a JSON export, where large IDs became floats)
flagged_ids = pl.Series([9007199254740992.0])
user_id = pl.Series([9007199254740993])  # Int64 -> a different ID, off by 1
user_id.is_in(flagged_ids)

Before 2.0, user_id gets coerced to Float64 to match flagged_ids. But 9007199254740993 sits above 2^53 (9007199254740992), the largest integer float64 can represent exactly, so it silently rounds down to 9007199254740992.0, giving a false positive.

In 2.0 this raises: InvalidOperationError: 'is_in' cannot check for Int64 values in List(Float64) data., users should explicitly cast to deal with lossy type conversion.

Strict concatenation

Horizontal concat will now check lengths instead of silently filling with null.

# Joining per-day transaction counts with per-day fraud-flag counts,
transactions = pl.DataFrame({"day": [1, 2, 3, 4, 5], "count": [120, 98, 143, 87, 156]})

# Upstream job for day 5 failed silently
fraud_flags = pl.DataFrame({"flagged": [2, 0, 5, 1]})  # only 4 rows

pl.concat([transactions, fraud_flags], how="horizontal")

shape: (5, 2)
┌─────┬───────┬─────────┐
│ day ┆ count ┆ flagged │
│ 1   ┆ 120   ┆ 2       │
│ 2   ┆ 98    ┆ 0       │
│ 3   ┆ 143   ┆ 5       │
│ 4   ┆ 87    ┆ 1       │
│ 5   ┆ 156   ┆ null    │  <- day 5 silently has no flag count
└─────┴───────┴─────────┘

In 2.0 this will raise with:

ShapeError: cannot concat dataframes with different heights in 'strict' mode

If padding is what you wanted, you have to explicitly opt-in to that with how="horizontal_extend". Making that intention clear to the reader.

Removal of casts in favor of dedicated methods/constructors

Another one worth mentioning is the removal of many casts that were ambiguous or should be applied via their dedicated parsing expression, leading to one obvious way to parse data.

Enums/Categoricals <> integers.

pl.Series([None, 1, 0, 2], dtype=pl.UInt32).cast(pl.Enum(["a", "b", "c"]))
# ComputeError: casting from u32 to enum is not supported.

Use instead: .cat.to(dtype) for int → categorical, .cat.physical() for categorical → int.

Parsing Strings to temporal data-types

pl.Series(["2022-08-30"]).cast(pl.Date)
# InvalidOperationError: casting from string to date is not supported.

Use instead: .str.to_date() / .str.to_datetime(). These allow you to apply a parsing format, giving you more control over how the data is parsed.

These were just a few examples, but we landed many more strictness improvements. See them all in the migration guide.

Raising informative errors

We put a lot of effort into making sure you as user or your agent can continue if you used old parameters that are not supported anymore. We added two new typed exceptions for this; polars.exceptions.AttributeRemovedError and polars.exceptions.ArgumentRemovedError that handle removed attributes and methods and removed parameters respectively.

The error messages should point you to the new API instead. Below we show two examples.

>>> lf.melt(id_vars="a", value_vars="b")
polars.exceptions.AttributeRemovedError: `melt` was removed in version 2.0;
use `LazyFrame.unpivot` instead, with `index` instead of `id_vars`
and `on` instead of `value_vars`

>>> df.join(df, on="a", join_nulls=True)
polars.exceptions.ArgumentRemovedError: the argument 'join_nulls' for
'DataFrame.join' was deprecated in version 1.24 and has been removed
in 2.0.0. It was renamed to 'nulls_equal' in version 2.0.

Most of the removed functionality has been deprecated for a long time and hopefully should not have affected your pipelines if you have stayed up to date. Reach out to us if you think we should have kept some functionality you relied on.

Last words

Polars 2.0 is about better defaults (most importantly the streaming engine) and a better API. We hope this release is rather boring. We don’t gate new features behind major version bumps as we ship them as soon as their ready.

Don’t be mistaken, Polars 2.x will be much better than 1.x. There is a lot in flight that we haven’t talked publicly enough: proper out-of-core support for the streaming engine, a new IO-plugin design, what we think will be the fastest S3 reader out there, major SQL coverage improvements, a cost-based planner, join reordering, and the removal of mmap, which will make our pipelines fully async end to end.

Try the release candidate by installing pip install polars==2.0rc1. Give it a spin and reach out to us here: https://github.com/pola-rs/polars/issues or contact us on discord: https://discord.gg/4UfP5cfBE7.

What comes to mind when you hear “frontend optimization”? For most of us it’s things like reducing network requests, shrinking the bundle, or making good use of the cache. Beyond that, maybe cutting down on re-renders or tuning when resources get loaded. The main thread doesn’t usually come up, and there’s a reason for that: on most screens it never becomes a problem. But on screens with a lot of interaction, where data streams in live and scrolling, animation, and input all get tangled together, the picture changes. However much you save on network and bundle size, the screen freezes the moment the main thread gets blocked.

You’ve probably come across a website where scrolling stutters now and then, a button responds slightly late, or the letters you type into a search box show up half a beat behind. It isn’t bad enough to be annoying, but it gets on your nerves in a subtle way. That kind of jank is what a blocked main thread looks like.

When we run into jank like this as developers, the usual reaction is to wonder “is my code slow?” and start picking apart algorithms or looking for wasted computation. In most cases, though, the speed of the code is not the problem. The code isn’t slow. It just happens to be the code that’s holding the main thread.

The browser has a number of threads, but almost everything we can touch from code is concentrated on the main thread. Computation, rendering, event handling, network response handling, and your framework’s internals are all processed there. One resource, a mountain of work.

The browser’s main thread is expensive. Most of the time it doesn’t cause trouble, but once you try to do something ambitious, dealing with the main thread becomes the important part. This article is about how to handle that expensive resource.

What Does the Main Thread Do?

Let’s start with what the main thread actually does. Its work falls into two broad categories.

The first is running JavaScript. The code we write, along with event handlers, timers, network response callbacks, and the framework’s internals, all run here. These tasks execute in the order they enter the queue, whenever there is a gap, with no relation to the screen refresh cycle.

The second is drawing the screen. When the DOM or styles change and the screen needs updating, the browser goes through roughly these steps, in order, to produce a frame.

• Run requestAnimationFrame callbacks - JavaScript registered to run just before the frame is drawn
• Style calculation - compute the final CSS values for each element
• Layout - compute each element’s position and size (also called reflow)
• Paint - generate paint commands describing what to draw in which colors

If nothing changed, these steps are skipped entirely, so they don’t necessarily run every frame. Only the final compositing step, which takes the produced output and assembles it on screen, is handed off to the compositor thread1. In other words, most of the front half of the pipeline that draws the screen is the main thread’s responsibility.

The rendering pipeline for updating the screen

For the screen to look smooth, frames have to be drawn at the display’s refresh rate. On the most common 60Hz display, that means 60 frames per second, or about 16.6 milliseconds per frame. And you don’t get to use all of it. Once the browser’s own processing cost is subtracted, the practical budget is usually considered to be around 10 milliseconds2, and on a 120Hz device the budget itself is cut in half.

The problem is that the two kinds of work above stand in a single line on the same thread. JavaScript was designed around a single-threaded event loop model. The main thread processes one task at a time, and while that task is running, nothing else can happen. If one JavaScript function runs for 200 milliseconds, then for those 200 milliseconds the browser can’t repaint the screen or receive a click from the user. Against a frame budget of around 10 milliseconds, that is a fatal amount of time. A task that runs this long and holds the main thread is called a long task, and anything over 50 milliseconds is generally considered a problem.

Words only go so far, so let’s feel it. In the demo below, pressing the button makes JavaScript grab the main thread for a moment.

Press a button: the JS animation and typing freeze, but the CSS animation keeps spinning

When you press the button, the JS animation stops and typing into the input field does nothing. The CSS animation, on the other hand, keeps running. We’ll come back to where that difference comes from later. What to remember for now is that holding the main thread for a long time is the same thing as freezing the screen.

This connects directly to web performance metrics. INP (Interaction to Next Paint), which measures how long it takes for the screen to respond after the user does something, and TBT (Total Blocking Time), which measures the total time the main thread was blocked during page load, are both essentially ways of expressing how long the main thread was blocked. A large part of performance optimization is a matter of how carefully you spend this one thread.

The ways of spending it carefully fall into two broad families. One is to divide the main thread’s time well from within. The other is to send the work outside the main thread altogether. Let’s take them in order.

Using the Expensive Resource Wisely

The first family is about staying on the main thread but spending its time intelligently. There are four core moves.

• How do you split up work that runs too long?
• How do you group work that runs too often?
• Among several tasks, which goes first?
• How do you postpone work that doesn’t need to happen now?

We’ll call these splitting, batching, prioritizing, and deferring. The first two shape the size of tasks, and the last two decide their timing. Of the four, splitting is the foundation for the rest. Tasks need boundaries before you can decide what to slot in between them and what to push back. So we start with splitting.

Splitting

Picture the chat pane of a live stream. On a popular stream, chat can burst to hundreds of messages per second. In that environment, messages don’t arrive politely one at a time. When traffic spikes, the server sends them in clumps of dozens, and the moment you enter a room, hundreds of backlogged messages come down at once. What happens if you render that whole clump in one go right when it arrives? Every message you draw brings DOM creation, style calculation, layout, and paint along with it, and those hundreds of iterations run back to back inside a single task. Meanwhile, the user trying to type their own message gets a stuttering input field, and every other animation on screen hitches too. Other people’s chat is monopolizing the main thread and getting in the way of yours.

The fix is what we said above. Cut the clump into small pieces, and between the pieces, hand control of the main thread back for a moment. In those gaps the browser can catch up on the screen updates and input handling it had queued.

The demo below simulates a streaming chat pane. Press “Flood the chat” and messages start pouring in. Try typing in the input field while watching the smoothness gauge and fps at the top, and compare the “Immediate render” and “Yielding render” modes.

In “Immediate render” mode, the DOM is touched as each message arrives, so while chat is flooding in, fps drops sharply, the gauge stutters, and the input field lags. If you look closely, the chat messages themselves start appearing noticeably more slowly as well, because the callback that receives and processes them is also a task waiting in the main thread’s line, so it gets delayed with everything else. Now switch to “Yielding render”. Messages are still drawn one at a time, just as before, yet input comes back to life and the screen moves again. The only thing that changed is that after every 20 messages, the main thread is released for a moment.

One thing not to misread here is that yielding does not make the work faster. The total amount of work is unchanged, and the few milliseconds spent waiting at each yield are added overhead, so in wall-clock terms it actually takes longer. So why did rendering recover along with input?

As we saw earlier, the main thread can do nothing while a task is running. The rendering pipeline that produces frames can’t cut into the middle of a task either. It can only run between tasks. Yielding is the act of creating those gaps. The backlogged input and frame production get their turn in the gaps, and to the user it feels as though performance improved.

At the code level, the classic way to yield is setTimeout, which pushes the continuation into the next task. Take a look at the following code.

// A batch of chat messages arrives at once
socket.on('messages', (chats) => {
  renderChats(chats);
});

// Draw the messages, yielding the main thread after every 20
async function renderChats(chats) {
  let count = 0;
  for (const chat of chats) {
    appendChatNode(chat); // draw one message

    if (++count % 20 === 0) {
      await new Promise((resolve) => setTimeout(resolve, 0)); // yield here
    }
  }
}

With this in place, no matter how hard chat floods in, the DOM work never occupies the main thread in one piece, and between the pieces there is room for the user’s input and animations to be processed.

The star of this code is setTimeout. When it schedules the resumption of the remaining work as a new task, the current task ends right there, and in that gap the backlogged input and rendering get processed. await pauses the function until the scheduled task comes back around, then picks up where it left off.

How yielding changes the timeline

The example above split the incoming work by count. But if an animation is already running, or the user is in the middle of scrolling, splitting by time is safer than splitting by count. An animation uses a little of the main thread every frame, so a heavy job has to keep checking the clock and cut itself off before it swallows what’s left of the frame’s budget.

async function processDuringAnimation(items) {
  let i = 0;
  let frameStart = performance.now();
  while (i < items.length) {
    // Work only until 5ms have passed since the frame started
    while (i < items.length && performance.now() - frameStart < 5) {
      doWork(items[i++]);
    }
    frameStart = await new Promise(requestAnimationFrame); // resume with the next frame's start time
  }
}

Here performance.now() acts as the stopwatch that checks whether we’ve gone over budget, and requestAnimationFrame acts as the alarm that says “wake me just before the next frame is drawn.” This is also why we yield with rAF rather than setTimeout when splitting by time. The resumption lands in step with the frame cycle.

Note that rAF passes the frame’s start timestamp to its callback, and the code above uses that as the reference point for the budget. The reason is that the function doesn’t have the frame to itself. If other animation callbacks ran earlier in the same frame, our share has to shrink by however much time they used, or the frame budget is broken. Anchoring to the frame’s start time turns “use 5ms” into “use until 5ms after the frame started,” which makes the code cooperate naturally when several animations share one frame.

Why 5 milliseconds? There’s nothing special about the number. We said the practical budget is around 10 milliseconds, so handing roughly half to background work and leaving the rest for animation callbacks, style, layout, and paint is a reasonable heuristic. If your animations are heavy, shrink it.

With this approach, even while heavy work is in progress, there is room to draw the screen every frame, and the work and the animation run smoothly side by side. Try the demo below. Moving the mouse scatters 4,000 particles away from the cursor, and nearby particles also push each other apart, so deciding one particle’s direction means checking its distance to every other particle. That comes to roughly 16 million distance calculations per pass, and recomputing all of it every frame blows through the frame budget on its own. Compare the “Compute all at once” and “5ms per frame” modes.

Splitting is the most basic way to use the main thread’s time sparingly. It is what gives users the perceived performance they care about: fast responses and a smooth screen.

Finally, a few points to be careful about. First, splitting too finely backfires. Yielding and coming back has a cost of its own, so if the pieces are too small, that overhead can end up larger than the work you’re trying to do.

Second, yielding with setTimeout involves a minimum delay3, so each piece can end up waiting a few milliseconds for nothing. Usually this doesn’t matter, but in situations that demand a very high level of responsiveness, the delay can become a problem.

That’s why some code schedules the next task by posting a message through a MessageChannel instead. React’s scheduler uses this method. More recently, a standard API called scheduler.yield() has also appeared to address this problem. Its advantage is that after yielding, the original work resumes ahead of other queued tasks instead of being pushed to the back. Browser support is still uneven, though.

Third, different yielding tools come back at different times. setTimeout and scheduler.yield() resume without regard to the rendering cycle, while requestAnimationFrame resumes just before a frame is drawn, so for work that needs to keep in rhythm with screen updates, requestAnimationFrame is the better fit. If you want finer control over priorities, you can also build your own queue on MessageChannel and manage the yielding and resuming yourself.

Lastly, splitting isn’t always possible. Parsing a multi-megabyte response with JSON.parse, for example, is a single atomic synchronous call, and there is no way to stop halfway and yield. Until the parse finishes, the main thread is stuck. Heavy work that can’t be split like this is the clear limit of “using it wisely.” In that case you have to change the premise and not do the work on the main thread at all. We’ll get to that in “Not Using the Expensive Resource.”

Batching

Splitting on its own doesn’t solve every problem, though. Think back to the streaming chat example. Yielding rescued input and rendering, but it did nothing to make chat draw faster. If anything, throughput, meaning the number of messages drawn per unit of time, went down by the overhead of yielding. So what happens if chat pours in faster than the throughput? Arrivals outpace processing, the backlog keeps growing, and the messages reaching the screen get older and older. This situation is called backpressure.

Raising throughput takes a different tool than splitting. For example, instead of drawing messages one by one, you can draw the accumulated batch in one go. The fixed per-message cost folds together, and the same amount of time renders more chat. Being told to split and then told to batch may sound like a contradiction, but the point of both is to trim tasks to an appropriate size. Splitting deals with tasks so long that rendering can’t squeeze in, and batching deals with tasks so frequent that the pipeline’s fixed cost is paid over and over.

The best batching targets are events. Scroll, resize, and input events can fire dozens or hundreds of times in a short span. If you run a heavy handler on every one of them, there’s nothing left of the main thread. So we collapse many events into one execution, either by “running once after things quiet down” or by “running at most once per interval.” These are called debounce and throttle, respectively.

The demo below is a markdown editor with a long CHANGELOG open. Building the preview means parsing the entire document (about 2,000 lines) and rebuilding its DOM from scratch, which is far too expensive to run on every keystroke. Type quickly into the left editor with “No debounce” selected. The preview is rebuilt once per character and your input falls behind. Switch to “Debounce 300ms” and the render happens just once, after you stop typing, and the typing becomes smooth.

For visual updates, you can use requestAnimationFrame. The screen only gets drawn once per frame anyway, so no matter how many update requests pile up, drawing once per frame is enough.

let scheduled = false;

socket.on('tick', (tick) => {
  chart.push(tick); // keep every data point — nothing is thrown away
  if (scheduled) return; // this frame's draw is already booked
  scheduled = true;
  requestAnimationFrame(() => {
    renderBoard(); // draw once per frame
    scheduled = false;
  });
});

The demo below updates a board of 60 tickers with over 1,000 messages per second. “Render every tick” mode redraws the whole board on every message. Calling a chart library’s update() on every message is a common mistake, and this is exactly what it looks like. Switch to “Once per frame” and every arriving data point is still reflected, but the fps comes back.

DOM writes can be batched as well. Appending a hundred nodes in one operation instead of one at a time, or toggling a single class instead of changing style properties individually, turns many changes into one and helps performance. The old technique of assembling an HTML string and assigning it to innerHTML in one shot has the same essence. You gather the writes so the rendering pipeline’s fixed cost is paid once.

This kind of optimization is a familiar pattern to frontend developers, and React’s virtual DOM is itself a device for it. However many times state changes, the changes accumulate in the virtual tree, get compared first, and only the actual differences are applied to the real DOM in one pass. Merging several state updates inside one event handler into a single re-render, or queueing analytics events and sending them in one request instead of individually, is the same idea. A fixed cost that would repeat once per item gets paid once per batch.

Prioritizing

If splitting and batching shape the size of work, prioritizing decides its order. Reacting to the button the user just pressed needs to happen quickly, while precomputing statistics for content that’s off screen can wait. Prioritizing means ordering the urgent work ahead of the work that isn’t urgent.

Order matters because on a main thread that nothing can interrupt, order is the responsiveness the user feels. To control order, you usually build a queue that work is pushed into and pulled out of. Jobs in the queue are processed FIFO, but when something urgent comes in, it gets pulled to the front of the line.

const queue = [];
const channel = new MessageChannel();

// One message = one task. Process a piece, then book the next one
channel.port1.onmessage = () => {
  const job = queue.shift(); // take whatever is at the front right now
  if (!job) return; // guard against duplicate bookings
  job();
  if (queue.length > 0) channel.port2.postMessage(null);
};

function postJob(job, urgent = false) {
  if (urgent) queue.unshift(job); // urgent jobs cut to the front
  else queue.push(job);
  if (queue.length === 1) channel.port2.postMessage(null);
}

This structure is useful because priority isn’t a fixed value. Work that wasn’t urgent to begin with can suddenly become urgent because of something the user does. Say the user attaches a few dozen photos to a post. To save on costs, the client sometimes resizes images before uploading them to the server, and that resizing is unhurried work that can be processed in order.

React works along similar lines, with more sophisticated machinery on top, such as starvation protection, batching, and continuations. Use startTransition or useDeferredValue and a scheduler spins up inside that yields via MessageChannel and orders work with its own priority queue.

But once the user clicks a particular photo to check that it attached properly, that photo’s preview becomes the most urgent job there is. With a priority queue like the one above, the urgent job can be handled first. This approach, where you get ahead on the work while idle and then rush it when it becomes needed, is sometimes called the idle-until-urgent pattern4.

// Build previews for the attached photos, in order
files.forEach((file, i) => {
  const job = () => createPreview(file, i);
  job.photoId = i; // tag it so we can find it in the queue later
  postJob(job);
});

// Clicking a photo that isn't ready pulls its job to the front → priority bump
onClickPhoto((i) => {
  const idx = queue.findIndex((job) => job.photoId === i);
  if (idx > 0) queue.unshift(queue.splice(idx, 1)[0]);
});

Feel the difference in the demo below. Sixty photos have been attached, and each preview is actually generated with per-pixel filtering. While the previews are being built in order, click one of the gray tiles that isn’t ready yet. In “In order” mode you have to wait until that tile’s turn comes, but in “Clicked first” mode it skips the queue and fills in right away. The total amount of work is the same and only the order changed, yet the experience for the user is completely different.

Priority, then, is a matter of working out what matters most to the user at this particular moment.

Modern browsers offer standards for this, such as the Scheduler API and TaskController. Support is still incomplete, so in practice people pair them with polyfills or build their own queues. This article uses the hand-built-queue approach.

Deferring

The last and most reliable way to conserve the main thread is to not do now what doesn’t need to be done now. Where splitting and batching ask “at what size” and prioritizing asks “in what order,” deferring asks whether this work really has to happen right now at all.

Initial page load is the classic place where deferring pays off. There’s no need to download and execute all of your JavaScript up front. With code splitting, only the code the current screen needs runs first, and the rest is loaded when it becomes necessary, which keeps the main thread from slowing down right from the start.

You can defer rendering itself, too. Think of a social feed. Some apps freeze for a moment when you come back from the notifications tab after scrolling far enough to accumulate hundreds of posts. Even if the feed’s DOM is kept alive while switching tabs, when it becomes visible again the browser recomputes style and layout for all several hundred posts at once, including the ones nowhere near the viewport. So what if off-screen posts were left as empty shells that only take up their height, and got filled with real content as they approach the screen? The tool that tells you about that “approaching” moment is IntersectionObserver. It’s the same method image lazy-loading libraries use.

const io = new IntersectionObserver(
  (entries) => {
    for (const entry of entries) {
      if (entry.isIntersecting) fill(entry.target); // fill as it approaches
      else empty(entry.target); // empty it when it leaves, keeping its place
    }
  },
  { rootMargin: '400px' } // headroom to fill before the scroll arrives
);

feed.querySelectorAll('.feed-item').forEach((el) => io.observe(el));

The demo below is a feed with 1,500 posts piled up5. It starts with “Render only when visible” turned on. Visit the notifications tab and come back, and the return is instant regardless of how much has accumulated. Now switch to “Render everything” and make the round trip again. Every return freezes for hundreds of milliseconds while all 1,500 posts are laid out again. Apart from unfilled slots showing briefly during fast scrolling, the two modes look the same.

Render only when visible: IntersectionObserver fills only the posts near the viewport — returning is fast no matter how much has piled up

Rendering isn’t the only thing this approach defers. Off-screen posts never get their DOM built at all, so the cost of creating and maintaining it is deferred along with everything else. If a widget carries heavy initialization, that initialization can also wait until the widget nears the screen. There’s also a CSS property, content-visibility: auto, that aims for a similar effect in a single line. As of this writing, though, implementations vary between engines, and Safari has a performance bug that makes returning to the page slower rather than faster, so for now IntersectionObserver is the option that behaves predictably everywhere.

Continuously running work, like carousels, animated promo banners, and live charts, is pure waste while off screen. You’re spending main-thread time every frame to redraw a picture nobody can see. Run it while it’s visible and stop it when it leaves. That’s also how the dozen-plus demos in this article manage to coexist on a single page. Each one is built to stop once it scrolls out of view.

Not Using the Expensive Resource

Everything so far was about using the main thread, but using it carefully. The second family of techniques is about not doing the work on the main thread in the first place.

Let’s return to the question left hanging in the long-task demo. The main thread was completely blocked, so why did the CSS animation keep running as if nothing had happened? The answer is that the animation was never running on the main thread to begin with. Inside the browser, several threads divide up the work. These are the notable ones.

• Main thread: runs JavaScript, manipulates the DOM, calculates styles, performs layout, handles events.
• Compositor thread: composites already-drawn layers onto the screen. Handles scrolling and certain animations.
• Raster threads: turn paint commands into actual pixels.
• Worker threads: separate JavaScript execution spaces that we create explicitly.

Unfortunately, we can’t control these threads however we like. The compositor and raster threads are territory the browser manages on its own, so we can’t give them direct orders, and worker threads, which we can create, come with the major restriction of having no DOM access.

So “not using” the main thread doesn’t mean sending arbitrary work elsewhere. It means picking out the work that can take a form other threads can handle, and sending that. There are two main ways to do it.

Moving Work to the Compositor

The compositor thread is the reason the CSS animation didn’t stop. transform and opacity don’t change an element’s position, size, or color in the document. They move an already-painted layer or adjust its transparency, so there’s no need to redo layout or paint. That lets the browser handle them directly on the compositor thread without going through the main thread at all. Even when the main thread is busy, the compositor runs separately, so the animation stays smooth.

If you move an element with properties like top, left, width, or height instead, layout has to be recomputed every frame, and that is main-thread work. In the demo below, the two boxes slide side to side in the same way, but one moves with transform and the other with left. Press the button to put load on the main thread.

Once the main thread gets busy, only the bottom box, the one moving with left, starts to stutter. The transform box at the top is being driven by the compositor and stays smooth whatever the load. The same “slide sideways” ends up on a completely different thread depending on which property you animate. That’s why it’s better for performance to build animations that move things with transform: translate rather than left, and animations that resize things with transform: scale rather than width.

But what about animations where the layout genuinely has to change? Picture a list where deleting an item makes the items below it slide smoothly up into place. This isn’t decorative motion. The positions really do change. Yet animating top means layout on every frame. The technique that resolves this dilemma is FLIP (First, Last, Invert, Play)6. In short, you cause exactly one layout change and leave the entire movement to transform. It goes in this order.

• First: measure the position before the move
• Last: actually change the layout and measure the new position. Layout happens exactly once, here
• Invert: apply a transform to the element in its new position so it appears to still be in the old one
• Play: animate that transform away. This part belongs to the compositor

const first = el.getBoundingClientRect(); // First: where it is now

list.prepend(el); // the one and only layout change

const last = el.getBoundingClientRect(); // Last: where it ended up
const dx = first.left - last.left;
const dy = first.top - last.top;

// Invert: make it look like it's back at the old position → Play: release it
el.animate([{ transform: `translate(${dx}px, ${dy}px)` }, { transform: 'none' }], {
  duration: 300,
  easing: 'ease-in-out',
});

To the user’s eye, the element glides from its old spot to its new one, but in reality the element has already arrived at its new spot, and the transform briefly drags it back before releasing it into place. While the animation plays, the only per-frame work is the compositor interpolating a transform. Most list-reordering animations are built this way, and Vue’s TransitionGroup and Framer Motion’s layout animations are FLIP under the hood.

The demo below shows the difference at a glance. Press “Play rank shuffle” and both ranking lists reshuffle in the same way. The left list animates top with a transition, and the right list moves only transform, via FLIP. With no load, both look smooth. Now turn on “Load the main thread” and play it again. The left list stutters its way to the finish, while the right one stays smooth even under load.

The same rank shuffle plays in both modes at once

Finally, two things worth knowing before handing work to the compositor. One is will-change: transform. It gives the browser a hint that “this element is about to change, so prepare it as its own layer in advance,” which can smooth out the start of an animation. Overuse it, though, and the number of layers balloons and memory gets wasted instead.

The other is reading layout values, which you just saw in the FLIP code. If you get the ordering wrong between code that reads layout values (getBoundingClientRect, offsetWidth) and code that writes styles, you run into a problem called layout thrashing.

// 🔴 Reads and writes interleaved — forces a layout recalculation every iteration
for (const el of elements) {
  const width = el.offsetWidth; // read (needs layout)
  el.style.width = width + 10 + 'px'; // write (invalidates layout)
}

// 🟢 Finish all the reads, then do the writes together
const widths = elements.map((el) => el.offsetWidth); // gather reads
elements.forEach((el, i) => {
  el.style.width = widths[i] + 10 + 'px'; // gather writes
});

If you read a layout value right after changing one, the browser has no choice but to recompute layout on the spot to give you an up-to-date answer. When that happens inside a loop, layout runs dozens of times in a single frame and the main thread slows down. Simply getting into the habit of grouping reads with reads and writes with writes is enough to avoid it.

Sending Work to a Worker

Then what about heavy work that can’t be re-expressed with transform? What do you do with things like parsing a large payload, processing images, or running complex computation? Pure calculation like that can be sent outside the main thread in its entirety with a web worker.

A worker runs JavaScript on a separate thread, fully separated from the main one. Hand the heavy computation to a worker, and in the meantime the main thread can concentrate solely on keeping the UI responsive.

// Main thread
const worker = new Worker('parser.js');
worker.postMessage(hugeRawData);
worker.onmessage = (e) => {
  render(e.data); // receive only the result and put it on screen
};

It isn’t free, of course. As we saw above, workers can’t access the DOM, so they can’t touch the screen directly. They can only compute and then send the results back to the main thread. And the main thread and the worker communicate only through postMessage, which copies (serializes) the data, so when the data being passed around is large, that cost is considerable.

So workers aren’t a cure-all. They shine when the computation is heavy enough to outweigh the communication cost and has nothing to do with the DOM. If you send a short, light job to a worker, the communication cost ends up larger than the computation cost and you come out behind. The key is to keep asking, every time, whether this work really needs to run on the main thread.

Since words only go so far, let’s bring in some genuinely heavy image processing. Seam carving is an algorithm that finds the vertical path of lowest energy (least color change) through a photo and removes it one path at a time, narrowing the image while preserving the important subject7. Removing a single seam means sweeping through hundreds of thousands of pixels, so removing 250 or so seams adds up to hundreds of millions of operations. Run that on the main thread and the whole page will freeze. Send it to a worker, though, and the screen can stay responsive the entire time the computation is running.

Press “Run on main thread” in the demo below. For the one or two seconds the computation runs, the entire page freezes, and the result appears all at once only after it’s finished. Because there are no task boundaries for paint to slip into, you couldn’t show the intermediate steps even if you wanted to. Now switch to “Run in worker”. While the same computation runs, you get to watch the image narrow in real time.

There is one more thing behind those smooth intermediate frames. If the worker copied a multi-megabyte pixel buffer every time it sent a frame, that cost would add up too. So postMessage offers an alternative to copying the data: transferring ownership of it outright. Transferable objects like ArrayBuffer move by reference only, so the cost is close to zero regardless of size. The side that hands the buffer over can no longer use it, and in exchange the copy cost disappears.

// Hand the pixel buffer to the worker without copying
// After the transfer, this side can no longer use it
worker.postMessage({ buf: pixels.buffer, width, height }, [pixels.buffer]);

Eliminating the Work Itself

So far we’ve been asking whether a piece of work really needs to run on the main thread. This time, let’s ask whether the work needs to happen at all. The best thing for performance is not doing the work in the first place.

Backpressure came up earlier. Batch as well as you like, and once the inflow exceeds the maximum throughput the backlog still grows without limit. Unfortunately, the browser has no good way to tell the server to slow down. At some point you have to give up on the idea of doing everything you’re given. There are generally three ways to eliminate work.

The first is dropping. For data that just flows past, like live logs, once processing starts falling behind you can quietly discard the oldest entries and users won’t notice. Keeping up with the present matters more than showing everything.

The second is merging. For data where only the latest value means anything, like rankings, you can merge the backlogged updates and apply only the final value. With merging, the amount of work is pinned to what the screen can digest, no matter how fast the inflow gets.

The third, skipping, targets repeated work rather than incoming work. If a computation gives the same result for the same input, there’s no reason to do it a second time. Remembering results and reusing them is called memoization.

This idea has been hiding throughout the article. Debounce skipped executions during typing, and the feed demo skipped rendering posts that weren’t visible. Looked at from this angle, half of this article was about eliminating work.

When you study optimization, your attention tends to go to ways of doing work well, but the biggest gains usually come from removing work. Before making some task faster, think about it first. Does this work have to happen, now, here, at all?

Closing

Some developers think of frontend work as the easy kind. But the browser is a far more complex system than we tend to assume. Drawing screens with HTML, CSS, and JavaScript is not the whole story.

Apps that deal with a flood of real-time data, like streaming platforms, or whose screens never stop changing, like image editors, maps, and games, start to feel slow whenever the main thread gets busy. Not everyone is on the latest hardware, so for services like these, optimization is essential. And solving these problems takes more than optimizing code. It takes understanding how the browser works, spending the main thread’s time sparingly, and not doing work that doesn’t need doing at all.

In the end, nothing is easy once you dig deep enough. So much of development is trade-offs, and you have to choose according to the situation, which ultimately comes down to the developer’s experience and judgment. Neither is built quickly, but both can certainly be built through study and experiment. I hope this article helps a little along the way.

1. The compositor thread is responsible for compositing already-drawn layers onto the screen. We’ll come back to it later in the article. ↩
2. https://web.dev/articles/rendering-performance ↩
3. The HTML spec mandates a minimum 4-millisecond delay once setTimeout calls nest more than 5 levels deep. Yielding repeatedly inside a loop, as in the code above, trips this condition almost immediately, so even with the delay set to 0, each piece waits at least 4 milliseconds. ↩
4. https://philipwalton.com/articles/idle-until-urgent/ ↩
5. Granted, 1,500 posts rarely pile up on one page in practice. The demo stacks that many on purpose, to make the effect of deferred rendering dramatic. ↩
6. https://aerotwist.com/blog/flip-your-animations/ ↩
7. https://en.wikipedia.org/wiki/Seam_carving ↩

K2 Horizon mountain artwork at sunset

Today IFM is releasing K2 Horizon, a connected fleet of six models: 375B-A23B, 36B-A4B, 32B, 7B, 3.7B, and 0.9B. Across reasoning, mathematics, coding, agentic tasks, and general capabilities, K2 Horizon delivers top-tier performance in every size class—with the 0.9B, 3.7B, and 7B models setting new state of the art at their respective scales.

K2 Horizon is also our most comprehensive open release to date. For every model, we are opening the training lifecycle from pretraining through reasoning and agentic post-training. We are releasing intermediate checkpoints, training data or detailed data-construction recipes, open architecture, mixture compositions, training code, configurations, fine-grained logs, evaluation results, and final weights.

The models and code are released under the Apache 2.0 license. Datasets are released under their applicable licenses, such as ODC-BY; We disclose how the data was constructed and mixed when redistribution is not possible.

Together, K2 Horizon represents the most comprehensive open model release to date:

• A new performance frontier across scales. The 0.9B, 3.7B, and 7B models achieve world-leading performance in their size classes across widely used evaluations. The 36B-A4B model, equipped with our new Mixture-of-Value-Attention (MoVA) mechanism, delivers exceptional capability per active parameter, outperforming some much larger models. The 32B and 375B-A23B models rank among the top models in their respective classes. Together, the six models provide competitive performance across deployment environments ranging from edge devices to the enterprise.
• The first fully open model fleet for agents. K2 Horizon is the first open model family to expose the complete development process through agentic post-training. By releasing checkpoints, data (or data recipe), code, configurations, and training logs across every stage, K2 Horizon makes it possible to study how reasoning, tool use, planning, and agentic capabilities emerge; reproduce the methods that create them; and adapt those methods to new tools, environments, and domains.
• Six models spanning edge to enterprise. The 0.9B model is designed for highly constrained environments such as watches and glasses, while the 3.7B and 7B models bring advanced capabilities to phones and other on-device applications. The dense 32B model and sparse 36B-A4B model provide powerful options for local workstations and efficient serving. The 375B-A23B model brings the fleet’s strongest capabilities to demanding enterprise deployments. All six models include quantization support.
• One connected fleet. The six models share core architecture, vocabulary, training methodology, interfaces, evaluation infrastructure, and deployment tooling, with a smaller vocabulary for the 0.9B model. This consistency also makes it easier to move between sizes, route work dynamically, and study capability and efficiency across scale.

World-leading performance across the scales

IFM/K2-Horizon

Benchmark comparisons for all six K2 Horizon model sizes

SWE-Atlas-QnA and SWE Bench Pro: strict = no internet. BrowseComp: different models use different harness, we use the Discard-all@95k context length proposed in DeepSeek-V3.2 technical report. WildClawBench: we use a subset of the English text-only-modality tasks. Apex-Agents: we use a subset of text-only-modality tasks. GDPVal-AA is the Elo rating.

The 0.9B, 3.7B, and 7B models achieve state-of-the-art results in their respective classes across mathematics, reasoning, general capability, coding, and agentic tasks.

The 36B-A4B model performs beyond the level normally expected from its active parameter count, demonstrating the efficiency of our unique Mixture-of-Expert design when computing attention values. The 32B and 375B-A23B models place among the top models in their respective comparison classes.

The small models are especially notable. K2 Horizon 0.9B achieves an AIME 2026 score above 48, along with strong reasoning, tool-use, and agentic capabilities. K2 Horizon 3.7B and 7B extend these capabilities to more demanding software-engineering and multi-step environments, demonstrated on strong performance in SWE-bench and BrowseComp. Although complex tasks that require extensive exploration and repeated recovery, such as those in TerminalBench, remain difficult for the smallest models, K2 Horizon moves the boundary of what is possible at every scale.

Why the Horizon Fleet matters

A transparent model that falls far behind the capability frontier has limited value as a foundation, even for research. At the same time, a powerful model released only as final weights allows people to run it, but provides little insight into how its capabilities were created.

K2 Horizon brings these two together. The fleet provides highly competitive models and releases the recipes used to train them. Researchers can study advanced capabilities in models strong enough to exhibit them, while developers can reproduce, adapt, and extend the methods rather than treating the final checkpoint as an opaque starting point.

Since introducing the fully open principle in our 2023 LLM360 paper, we have released open models every year while extending that commitment to larger scales, stronger capabilities, and now the complete lifecycle through agentic post-training.

For every Horizon model, we will release:

• Training data or recipe, such as construction methods, and mixture compositions
• Training code
• Model configurations and training recipes
• Intermediate checkpoints throughout training
• Fine-grained training logs
• Evaluation results across general and specialized capabilities
• Final model weights

A final checkpoint shows what a model can do. Horizon’s complete training record helps reveal how it learned to do it.

A Deep Dive into The K2 Horizon Fleet

K2 Horizon 375B-A23B: the enterprise powerhouse

K2 Horizon 375B-A23B is the fleet’s largest and most capable model. Its sparse MoE architecture provides 375 billion parameters of total capacity while activating approximately 23 billion parameters for each token, allowing it to draw on the capacity of a much larger model without using every parameter for every token.

The model ranks among the top models below 400 billion parameters across general, reasoning, coding, and agentic evaluations. It is designed for demanding workloads where model quality matters most, including complex reasoning, software engineering, research, and long-horizon agentic tasks.

Like every model in the Horizon fleet, 375B-A23B is released not as a single endpoint but as a development tree. Its intermediate checkpoints and post-training branches expose how the base model develops into reasoning, instruction-following, and specialized agentic variants.

IFM/K2-Horizon-375B-A23B

K2 Horizon 375B-A23B benchmark comparisons

Benchmark comparison for IFM/K2-Horizon-375B-A23B.

K2 Horizon 32B and 36B-A4B: strong performance for local deployment

Horizon 32B is the fleet’s most powerful dense model, providing a strong balance of capability, adaptability, and local deployability. It ranks among the top dense models below 40 billion parameters.

Horizon 36B-A4B reaches nearly the performance of the dense 32B model while activating only approximately 4 billion parameters per token. Its efficiency comes from MoVA, our new sparse attention architecture, together with MoE feed-forward layers.

These two models serve as an important reference point for studying how dense and sparse architectures behave under similar training conditions.

These models occupy the fleet’s local performance sweet spot. They are powerful enough for demanding reasoning, coding, and agentic applications while remaining practical for local workstations and efficient serving systems.

IFM/K2-Horizon-MoVA-36B-A4B

K2 Horizon 36B-A4B benchmark comparisons

GPT 5.6 luna at medium reasoning effort; Muse Glimmer-30B at high.

K2 Horizon 7B, 3.7B, and 0.9B: frontier capability at small scale

k2 Horizon 7B and 3.7B deliver strong reasoning, mathematics, coding, tool-use, and agentic performance while remaining suitable for local and on-device deployment. On several evaluations, their results approach or exceed those of models many times larger from the previous generation.

K2 Horizon 0.9B carries many of the same capabilities into highly constrained environments. It can perform mathematical reasoning, use tools, and complete simple agentic tasks while remaining compact enough for applications on watches, glasses, and other edge devices under quantization.

The appropriate task changes with scale: the 0.9B model is best suited to focused interactions and lightweight tool use, while the 3.7B and 7B models can handle more demanding coding and multi-step workflows. Together, they demonstrate how much capability can now be retained in models small enough to run almost anywhere.

K2-Horizon 0.9B / 3.7B / 7B

K2 Horizon 7B, 3.7B, and 0.9B benchmark comparisons

Designing K2 Horizon

One family from the beginning

Horizon was designed as a connected family rather than a collection of unrelated models. The six models share core architectural decisions, training methodology, interfaces, evaluation infrastructure, and deployment tooling. This allows developers to move between sizes more easily and gives researchers a more controlled setting for studying capability across scale.

Each model is pretrained on approximately 20 trillion tokens using carefully constructed and documented mixtures. Intermediate checkpoints and their corresponding fine-grained logs are captured throughout training, creating a detailed record of how each model develops.

MoVA: scaling attention with sparse experts

The core idea behind mixture-of-experts is to increase total model capacity while keeping the computation required for each token roughly fixed. Conventional MoE architectures apply this sparsity primarily to feed-forward layers: many specialized experts are available, but a router activates only a small subset for each token.

Our new architecture, MoVA—Mixture-of-Value Attention, extends this principle to attention. Because attention determines how a transformer brings together information from across its context, introducing sparsity there opens another dimension for scaling model capacity beyond the feed-forward network.

MoVA integrates expert routing into multi-head attention while remaining compatible with efficient techniques including FlashAttention, grouped-query attention, and sparse attention.

The result is K2 Horizon MoVA 36B-A4B: a model with 36 billion total parameters but approximately 4 billion active parameters per token. Under the same training conditions, it performs only slightly below the dense Horizon 32B model while requiring substantially fewer active parameters.

Training Data

Training across six scales requires data that is broad enough to support general capability and carefully constructed enough to maintain quality over approximately 20 trillion tokens.

Horizon’s pre-training mixture combines diverse web, code, mathematical, scientific, multilingual, and domain-specific sources with synthetic data generated through our own pipelines. One of our key data innovations is the incorporation of reasoning directly into pre-training: nearly 17% of the pre-training corpus consists of problem-solving trajectories with explicit reasoning. Reasoning trajectories for mathematical tasks were further rewritten into formats such as dialogues and study guides. In total, we used approximately 10 trillion synthetic tokens during pre-training.

Our synthetic data pipelines use millions of combinations of diversity knobs and context seeds, including retrieval from an internally built search engine over the pre-training web corpus. To quantify diversity at corpus scale, we developed a new compressor, Wzip, that combines a novel adaptive multi-sliding-window LZ77 algorithm with windowed Huffman coding. Wzip mitigates the rapid saturation observed with conventional gzip/zstd compression metrics as the number of documents grows, enabling more meaningful diversity measurements across large corpora. As shown below, the measured diversity of our synthetic data approaches that of high-quality natural web text and substantially exceeds that of web code.

Comparison of diversity across training data sources

We document the data composition, synthesis pipelines, and construction of the training mixture. Following our open-source principles, where licenses permit, we release the training-ready datasets directly. Where redistribution is restricted, we release source descriptions, filtering and construction methods, and mixture composition. This allows researchers to understand what each model learned from and how the data distribution evolved throughout training.

Our post-training data is introduced from the beginning of mid-training, rather than being reserved exclusively for the final stages of training. We combine synthesized long-context documents with instruction-following, reasoning, and agentic trajectories formatted with the chat template. A main pillar of our post-training data is large-scale task synthesis grounded in task taxonomies, diversity knobs, and web-search seeding, resulting in over 100 million unique tasks. In addition, we developed sampling techniques that guide solver LLMs toward correct solutions and desired behaviors when generating training trajectories.

Beyond the construction of the data itself, we also designed the chat template to improve the diversity and efficiency of tool-use training. Rather than training on a single fixed convention, we exposed the model to multiple tool presentation and tool calling formats. Tool definitions were represented using JSON, XML, and Markdown, while tool calls used JSON, XML, and typed XML. This variation encourages the model to learn the underlying semantics of tool understanding and selection rather than overfitting to a particular syntax. For inference, we ultimately selected Markdown as the default tool presentation format because it was approximately 18.5% more token-efficient than the conventional JSON formatting on our data.

Pre-training Dynamics

Every K2 Horizon checkpoint is accompanied by detailed training logs and fine-grained histories that expose the real dynamics of large-scale training: how losses evolve, where instabilities appear, how interventions affect training, and when capabilities begin to emerge.

As an example of why these records are useful, K2 Horizon 3.7B, 7B, 32B, and 36B-A4B were trained on exactly the same 22 trillion tokens. When aligned by training progress and normalized by the median loss over the final 1% of training, their raw loss trajectories approximately collapse, which you can see in the figure below—even across dense and sparse architectures and nearly an order of magnitude in total parameter count. Although this normalization aligns their endpoints by construction, it does not force the early and intermediate trajectories to coincide. Their close agreement shows that, under shared data and a common recipe, the shape of learning remained remarkably consistent across the fleet subset that used the same dataset throughout training.

Normalized pre-training loss trajectories across K2 Horizon model sizes

Together, the checkpoints and logs allow researchers to investigate why some dynamics transfer across scale and architecture while others diverge, and to connect those differences with particular training stages, data mixtures, and technical decisions.

Post-training for reasoning and agents

Most of K2 Horizon’s advanced reasoning and agentic capabilities emerge during post-training. The complete pipeline includes mid-training, supervised fine-tuning, model merging, reinforcement learning with specialized agent training. Rather than producing only one final chat model, this process creates a development tree. Different branches specialize in reasoning, coding, tool use, and agentic domains while remaining connected to common base checkpoints.

We release the artifacts across these stages so researchers can study where each capability emerges, reproduce individual branches, and adapt the same methods to new tools and environments.

Uno Diffusion: plug-and-play lossless speedup for Horizon

Inference speed has become a first-class optimization target, especially in the era of reasoning models and agents. As models produce longer chains of thought and take more actions, even small per-token delays compound into substantial latency. Yet autoregressive language models generate one token at a time, creating a fundamental bottleneck. Existing approaches offer partial solutions: speculative decoding typically requires a separately trained draft model, while discrete diffusion enables parallel generation but often sacrifices quality for speed.

Uno is designed to remove that tradeoff. It provides a lossless inference speedup, accelerating generation without degrading response quality. Uno keeps Horizon’s autoregressive parameters frozen and fully responsible for the model’s output distribution, while a lightweight set of diffusion parameters learns only how to generate more efficiently.

Through a process we call Diffusion Distillation, these compact adapters learn to generate blocks of tokens in parallel. The result combines the quality guarantees of autoregressive decoding with the speed advantages of diffusion: the model reaches the same answers, but reaches them faster.

Across our evaluations, Uno achieves a better speed–quality tradeoff than leading speculative-decoding systems and both open-weight and proprietary diffusion language models. Crucially, its gains persist across every batch size we tested, enabling lower latency for interactive agents and higher throughput for large-scale serving, with no loss in model quality and negligible additional training overhead. Uno is delivered as a simple LoRA adapter, making this lossless speedup easy to adopt: all you need is to attach the adapters.

Open infrastructure for building and extending Horizon

We are releasing the infrastructure used to build Horizon alongside the models themselves. The goal is not only to make the fleet reproducible, but to make its development stack useful as a foundation for new models and systems.

At the center of this release is xLLM, our production-tested training infrastructure. xLLM combines large-scale training performance with the flexibility required for research, allowing teams to modify architectures, data mixtures, training stages, and objectives without rebuilding the surrounding system.

We will be also releasing our full agentic post-training code base, including Reinforcement Learning, which we hope will help researchers to advance the techniques in related areas.

Researchers can use these components to inspect training behavior or reproduce a particular stage. Developers can adapt Horizon to a domain, add new tools, train agent experts, or deploy several model sizes behind a unified interface.

From Open Source to Open Science

Intermediate checkpoints turn model development into an observable scientific process. They allow researchers to study when capabilities emerge, how training choices change behavior, and when unintended strategies first appear.

Reward hacking provides one revealing example. When a capable model is placed in a realistic computer environment and told to solve a complex task, the same resourcefulness that makes it useful can also lead it to exploit shortcuts in the evaluation itself. To understand how much this affects K2 Horizon's reported numbers, we audited the released model using Artificial Analysis's reward hacking auditing procedure.

K2 Horizon reasoning trace after finding a benchmark solution on GitHub

The JACKPOT moment: our model found the benchmark's solution on GitHub and expressed "excitement" at having the answer handed to it.

TerminalBench 2.1 places models in sandboxed computer environments and evaluates whether they can complete complex technical tasks. A capable model must explore files, invoke tools, diagnose failures, and find alternative paths toward a solution. That resourcefulness is exactly what we want. But occasionally it crosses a boundary: instead of solving the intended problem, the model locates a hidden answer, exploits something the task left exposed, or manipulates the grader itself.

We ran K2 Horizon 375B-A23B on 89 TerminalBench 2.1 tasks with eight attempts each, producing 712 trials. Of these, 500 passed the task verifier (70.2% reported accuracy). We then audited every passing trial using Artificial Analysis's reward hacking auditing procedure, applying their harbor analyze tool with the reward_hacking criterion and their full rubric text verbatim, with Codex gpt-5.6-sol as the judge model.

The audit flagged 24 trials across 10 tasks. Removing them lowers the accuracy from 70.2% to 66.9%, a correction of 3.37 percentage points. The remaining 79 tasks were fully clean. For context, Artificial Analysis reports flag rates of 2.2% for claude Fable 5 and 4.1% for GPT-5.6 Luna; K2 Horizon's 3.37% falls within that range.

The model discovered several strategies:

• Inferring it was inside a public benchmark, finding the repository on GitHub, and downloading the reference solution
• Pulling the current source from a real project's public repository and copying the fix rather than deriving it
• Inspecting unadvertised files, generator scripts, or exposed credentials
• Editing the test harness or crafting output that exploited how the test checked success

We observed a related case with K2 Horizon 7B, which found and downloaded SWE-bench answers and consequently produced an inflated score of 82. The score does not represent genuine software-engineering performance, but the behavior is scientifically revealing: benchmark hacking emerged as an unintended consequence of broader planning, tool use, environment exploration, and persistence.

Because we release intermediate checkpoints alongside the final models, these behaviors can be studied rather than hidden. Researchers can determine when a strategy first appears, connect it to changes in training, and measure its effect on reported performance.

K2 Horizon is therefore more than a collection of model weights. It is an open experiment in how capabilities—and their unintended consequences—develop throughout training.

Get Started with K2 Horizon Today!

All six K2 Horizon sizes are released as open weights under Apache 2.0, with day-zero support from vLLM, SGLang, and Ollama. Further, K2 Horizon supports deployment on NVIDIA, AMD, and Cerebras hardware, providing options from local inference to large-scale serving. Get the models from our repository: https://huggingface.co/IFM, download the models, inspect intermediate checkpoints, study training data and mixtures, reproduce training stages, or build new models and agents using the Horizon code and infrastructure.

K2 Horizon provides more than six final models. It provides an open blueprint for understanding, adapting, and advancing the next generation of AI systems.

Full Results

375B-A23B

Full result table for K2 Horizon 375B-A23B

36B-A4B

Full result table for K2 Horizon 36B-A4B

32B

Full result table for K2 Horizon 32B

7B

Full result table for K2 Horizon 7B

3.7B

Full result table for K2 Horizon 3.7B

0.9B

Full result table for K2 Horizon 0.9B

Washington — A whistleblower is warning of "potentially catastrophic problems" with the U.S. Postal Service's new system for handling mail ballots. The whistleblower is accusing the agency of flouting court rulings by continuing to work on implementing a Trump executive order to tighten mail voting rules before the November midterm elections.

Described by lawyers as a federal official, the anonymous whistleblower revealed the information about the Postal Service's mail-voting procedures in a disclosure provided to Democratic Sen. Richard Blumenthal of Connecticut that was made public Tuesday. In response, Blumenthal is now questioning Postmaster General David Steiner about the Postal Service's development of a new online portal to house information about voters and their mail ballots.

"The main takeaway for me is that the Postal Service has designed a system to disenfranchise millions of Americans," Blumenthal told reporters. "One third of all Americans cast their ballots by mail, and the USPS puts all of their votes at risk."

The Postal Service said it received Blumenthal's letter and is carefully reviewing the concerns raised by the senator and the whistleblower.

"The United States Postal Service has spent months developing a U.S. Federal Ballot Mail Portal to provide election officials with a simple, secure, and efficient way to share lists of individuals receiving ballots by mail in their respective states. This work has at all times been conducted in a manner consistent with court orders," the agency said in a statement.

The Postal Service said it is finalizing the portal and will "soon" make it available to election officials to voluntarily use.

"Regardless of political party or perspective, we share a common goal: ensuring that Americans can have confidence that their election mail will be handled securely and delivered reliably," it said.

President Trump has sought to exert more federal control over U.S. elections ahead of the November midterms, including through his executive order imposing new restrictions on mail ballots. The president has also pushed Congress to approve a package of voting regulations that would outlaw mail voting — with some exceptions — and impose new proof-of-citizenship requirements to register to vote.

As Mr. Trump signed his executive order in March, he claimed that "the cheating on mail-in voting is legendary," and said new mail voting rules were needed to ensure the integrity of federal elections. But his administration has not put forth evidence of widespread voter fraud, and the president himself has cast ballots in Florida's elections by mail.

As part of its implementation of the president's directive, the Postal Service published a final rule last month laying out its new requirements for mail voting. A federal judge temporarily blocked its enforcement last week, and the Justice Department has appealed that decision.

Currently, six states conduct their elections entirely by mail. There are also four states, plus the District of Columbia, that have not yet eliminated in-person voting but also send ballots to all of their voters. In a CBS News poll in March, 77% of Democrats said anyone should be allowed to vote by mail, while just 25% of Republicans also felt that way. Independents were split, with 49% backing mail voting by anyone.

The new Postal Service rule imposes new design standards for outbound and return mail ballot envelopes, including the addition of a unique barcode. Under the new rule, state election officials must submit to an online "Federal Ballot Mail Portal" the names and addresses of voters receiving mail ballots, as well as the unique barcodes on the mail ballot envelopes.

Once that information is provided through the portal, those voters would be considered enrolled with the Postal Service on their state's "Mail-In and Absentee Participation List." The agency has said it "will not play any role in determining voter eligibility, maintaining voter rolls, or counting ballots."

Most states led by Democrats oppose the rule, arguing that the Constitution gives states the authority to set election rules, not the federal government.

Blumenthal described the Postal Service's actions as "sabotage."

In a letter to Steiner dated Monday, Blumenthal said the agency's implementation of Mr. Trump's executive order has been "perilously rushed and potentially unlawful," and he urged the Postal Service to walk back its new mail-voting plans for the upcoming elections.

He wrote that the whistleblower's allegations show that "USPS lacks the technical or operational capability needed to effectively implement the EO's provisions in a way that safeguards every citizen's right to vote in the upcoming midterm elections."

According to the whistleblower disclosure provided to Blumenthal by lawyers with Whistleblower Aid, the new portal is "untested" and will have "significant operating problems" when released, given its "rushed" development.

The federal official also warned that the new verification process could keep large numbers of mail ballots from being delivered to voters.

For instance, the Postal Service has what the whistleblower claims is a "zero percent failure rate." Mail workers would scan a sample of a batch of mail ballots — about 400 out of a batch of 10,000 or more — to verify it matches what is in the federal mail ballot portal. Under the policy, if even one barcode on a single ballot in the batch doesn't scan properly or the information does not match, the entire batch would be rejected and returned to state election officials so the problem ballot could be addressed, and then the verification process would have to be started again.

The whistleblower's lawyers said that the way the process is designed is "entirely unforgiving."

"It could delay ballots by the thousands in repeated verification cycles — and thus prevent states from mailing enormous numbers of ballots," the lawyers said.

David Becker, executive director and founder of the Center for Election Innovation and Research, warned that the process raises "tremendous" concerns for disenfranchisement of voters.

"Are Americans clamoring for the Postal Service to have this added responsibility as they're hoping that their ballots are just seamlessly sent to them?" Becker, a CBS News election law contributor, asked.

He also said that the new Postal Service rules "would undoubtedly create a tremendous number of hoops that are completely unnecessary and do nothing for election integrity."

Publication of the final rule came days before the Supreme Court intervened last week in a case brought by 23 Democratic-led states that challenged key provisions of Mr. Trump's executive order. U.S. District Judge Indira Talwani had blocked the Trump administration from implementing those portions of the directive in June, including plans for the Postal Service, but the Supreme Court lifted that lower court decision.

The high court's order was procedural: The majority found that the states' lawsuit was premature. The Supreme Court said its decision "does not mean that any measure taken by the Government to implement the Order will necessarily be lawful. On that score, time will tell."

In addition to expressing concerns about the Postal Service's new requirements for mail voting, the whistleblower's disclosure also contends that the agency violated court orders by implementing Mr. Trump's executive order.

After the district judge issued her order blocking implementation of the president's directive, Postal Service workers resumed work on the portal, the official said, and work continued even after the judge's latest ruling last week.

In a brief order Monday, Talwani, who is presiding over challenges to the new rule, said the Postal Service can establish the portal and communicate with states about design standards "so long as states' participation is not required."

The Postal Service aims to have the portal ready to be rolled out to the public by Sept. 1, according to the disclosure. North Carolina will begin sending mail ballots to some voters Friday, and several other states will make mail ballots available around mid-September, according to the Center for Election Innovation and Research.

"Our client, a lawful anonymous whistleblower, has come forward to flag a foreseeable catastrophic disruption to our coming nationwide elections based on the reckless pursuit of readying a new, insufficiently tested, poorly planned mail-in ballot screening system," the whistleblower's lawyers wrote.

And why you should ignore demo videos

AI progress is racing along, but virtually all of the visible progress is in the realm of knowledge work, i.e. activities that can take place inside a computer.

In the San Francisco AI scene, there is a widespread belief that robots will soon enter the picture. In parallel with the race to develop broadly capable AI, there is an equally aggressive race to develop broadly capable robots – humanoid machines imbued with physical intelligence. Artificial workers that can cook and clean, fetch and carry… and do everything else, including building more of themselves, leading (in many forecasts) to economic growth best characterized as an “explosion”.

Inspired of course by this classic cartoon by the great Sidney Harris

In other words, the thinking goes, AI in the data center will soon subsume all intellectual labor, and AI in humanoid bodies will soon subsume all physical labor. However, there is an important difference: while we can see progress in the intellectual realm, the physical side of AI is mostly confined to test facilities and demo videos. There is no robot equivalent to ChatGPT – nothing that you or I, or even most people in the AI community, can get our hands on.

So we’re stuck with demo videos. Unfortunately, they are a poor tool for assessing progress. We might be seeing the one successful task achieved in 100 attempts. The scenario might have been carefully arranged to avoid challenges the robot isn’t ready for. The video might be edited to make it look like the robot is acting with more speed and reliability than is actually the case. Here’s one very impressive demo… with a suspiciously large number of camera cuts.

(I have not yet had much chance to watch videos from the recent World Humanoid Robot Games. These are valuable for providing a public platform less amenable to cherry-picking. The handful of videos I’ve watched include some impressive feats, but don’t address many of the challenges I list below… and there are also a lot of spectacular failures.)

Demos draw attention to the things a robot can already do. The question then becomes: what’s missing? In today’s post, I’ll catalog the technical challenges that will have to be overcome along the road to broadly capable artificial workers. The next time you watch a robot doing something impressive, ask yourself: which of these capabilities has the robot demonstrated, and which challenges might the demo scenario be avoiding?

(Note that some challenges get easier if we consider wheeled robots rather than strictly humanoid robots. A wheeled robot can carry more weight, meaning that strength, endurance, and power for electronics are less of a challenge. And wheeled robots are less likely to fall over. But they can’t climb stairs1, step over clutter, or angle themselves to reach into a cupboard.)

Hands, dexterity and coordination

Maybe one of the last human jobs will be close-up magic

The human hand is an engineering miracle – opposable thumbs, and all that. It has roughly two dozen “degrees of freedom” (distinct joints and/or directions in which each joint can bend), and approximately 17,000 tactile sensors. Our brains can control our hands with exquisite grace, using touch, sight, and even auditory cues to carry out all manner of delicate tasks, precisely and reliably.

Current robot “manipulators” are a pale imitation. Some existing robot hands can match the human standard on one or another physical attribute. For example, some have as many as 27 degrees of freedom. However, none come close to matching the overall package of flexibility, sensitivity, strength, reliability, and other physical attributes. It is the combination of factors that is especially difficult to match, even if the demos are getting more impressive. For instance, some companies have managed to cram thousands of tactile sensors into a robotic fingertip, but none have managed to make these tiny sensors able to stand up to heavy use2.

The control problem may be as challenging as the problem of physical construction. A competent robot must be able to find the right set of joint positions to grasp a complicated object; plan out the sequence of motions to fold a shirt, flip an omelette, or tighten a bolt in a constrained space; and handle squishy or floppy materials (which can require reacting instantly to a sudden shift).

Visual understanding

Computer vision has made incredible strides over the last decade or two (and is responsible for kicking off the deep learning boom that led to LLMs). But making sense of complicated visual scenes – picking out an object from a crowded environment, understanding where it should be grasped, determining where it’s safe to put your feet and how to avoid knocking something over – is not a solved problem.

Planning and reacting

A general-purpose robot must be able to break down a task into individual steps, and relate those steps to its environment. How do you maneuver your arm to get a screwdriver into a piece of machinery? What’s the quickest way to clear a path to the spice bottle at the back of the shelf? In what order should you pick up the items on the living room floor?

True autonomy will require planning tasks of greater scale and complexity: cooking a meal, plumbing a bathroom, repairing an engine. Not to mention the need to re-plan in the face of surprises – a stuck bolt, a rotten piece of produce, a child darting into the kitchen.

Understanding the assignment

When current AIs fail at a knowledge work task, it’s often because they weren’t provided with sufficient context. Robots will need context, too: where are supplies kept? How do you like your meals cooked? How much assistance does that nursing home resident need, and is that hitch in their stride normal, or a sign that they’re about to stumble?

Once they have context, robots will need to reason, plan, and exercise judgement and common sense. LLM-based systems like ChatGPT and Claude are making great strides in these areas, but the physical domain brings additional challenges3. The success of LLMs has been greatly assisted by the massive pools of pre-existing data that were available for training – a substantial fraction of all books ever written, the web, and other massive pools of pre-existing data. It will be difficult to match this scale of breadth and depth of data for physical tasks. There’s no straightforward equivalent of “just Efficient learning, generalization, and adaptability / on-the-job learning seem like requirements.

Cooperation

AI agents mostly operate in isolation, and in static environments. We rarely put them in situations where things are changing out from under them, or ask them to coordinate. When we do, things often go haywire. Isolation is easier to arrange in the virtual world, where private workspaces can be created at will, and nothing is too heavy to lift on your own. Robots will often need to cooperate with people, or with one another.

Speed

From an article which notes “it took the H1 nearly a full two minutes to very slowly move to a couch, pick up a single item of clothing and put it into the washing machine”.

Today’s general-purpose robots often move much more slowly than human beings. Challenges include strength, control (higher speed means less time to plan and react), and safety (a fast-moving robot will whack you harder and is harder to dodge).

For some applications, slow and steady may be perfectly acceptable: I may not care if my household robot takes all night to tidy up and fold the laundry. But a slow-motion robot won’t be much use as a cook or nursing-home aide. It might get in the way at a warehouse. And it will have a harder time getting enough work done to pay for itself.

Strength

That… is just not an impressive amount of weight for a full-grown robot.

Some industrial robots are extremely strong. But humanoid robots – or other highly mobile, “general-purpose” robots – usually aren’t. It’s difficult to combine strength with manageable weight, a large number of joints, and a maneuverable frame. Powerful motors generate more heat and deplete batteries faster – two areas where robots already struggle (see below). And a strong, heavy robot poses greater safety challenges.

Mobility

ED-209 may have autocannons and a rocket launcher, but it was no match for the staircase

The jury is still out on the appropriate form factor for general-purpose robots, especially with regard to their lower half. Should they have wheels or legs? Two legs, four, or some other number? Wheels are cheaper, more stable, and more reliable; legs are better for stepping over obstacles and climbing stairs. A bipedal frame is more maneuverable, but also more likely to topple if something goes wrong. In any case, the question is: can the robot reliably get around its work environment?

Safety

Sadly, yes, that is a robot karate-kicking a child in the stomach (video). Fortunately the kid was OK.

Safety considerations for general-purpose robots are almost limitless. A glitchy or malfunctioning robot could bump into someone, topple onto them, drop something on them, spill something on them, break a glass, or start a fire.

Safety for LLM-based agents relies in part on review of discrete actions, such as attempts to send an email or delete a file. Robots move constantly, and it’s not so easy to single out a few specific motions as the potentially dangerous ones requiring review.

If a self-driving car finds itself in a situation it can’t handle or suffers a glitch, it can pull over or, in the worst case, just hit the brakes. A general-purpose robot that suddenly freezes might leave something on the stove, topple mid-step, or trip the person it was assisting.

And of course danger can be initiated by human action, such as a child darting in front of a robot. I’d much rather my kid be bumped into by a squishy person than a metal robot; and as things stand today, I’d much rather depend on human reflexes and adaptability to avoid tripping over the little rascal.

(The stronger, heavier, and more capable the robot, the greater the risks.)

Endurance

Today’s bipedal robots can typically run for a few hours before recharging. I suspect this won’t be a limiting factor: if a workaround is needed, we’ll find one, whether that means swapping battery packs, in-floor charging grids, or a cable running to a nearby big-battery-on-wheels4.

I was surprised however to learn that overheating is a serious challenge for continuous operation. A human-sized robot generates about twice as much heat as a person, and robots don’t have the same elegant mechanisms (whole-body circulation and perspiration) for distributing and dissipating that heat.

Then there’s the question of reliability. Robots have large numbers of moving parts, many of which are necessarily finicky, because they’re engineered to push the envelope on size, weight, and performance. As a result, current attempts at general-purpose humanoid robots experience frequent breakdowns. (Contrast the human body, which is constantly recovering from wear and tear, and has substantial ability to self-repair and to compensate for minor breakdowns.)

Generalization; edge cases

Somehow Waymo forgot to teach their cars not to drive into flooded intersections?

Waymos struggle to handle edge cases. They’ve recently been observed driving into flooded roads or over burning fireworks. This is despite the fact that self-driving cars have been in development for well over two decades, and Waymo cars in particular have driven over 220 million miles – 250 times as many as a typical American drives in their lifetime.

(I’m willing to cut them some slack on the fireworks thing; 250th anniversaries don’t come along all that often. But I am confused at how Waymo engineering can be so robust as to yield an astonishingly good safety record, and yet so slapdash as to happily drive into deep water.)

For all of the weird edge cases that arise while driving – the classic example being a duck being chased by a broom-wielding woman in a wheelchair – robots operating in homes and businesses will encounter far more. They will be faced with a wider variety of tasks, using a wider variety of equipment (different tools; different robot bodies to grasp those tools), in a wider variety of environments. Achieving reliable operation outside of the controlled environment of a factory floor may be the hardest challenge of all.

Compute

Dexterity, coordination, visual understanding, planning, reacting, understanding, cooperating – and doing all of these quickly, safely, and reliably – will take a lot of computing power. Incorporating the necessary computing capacity into the robot itself will add cost, drain batteries, and contribute to overheating. Leaving the robot’s brains in the cloud will slow down reaction times and introduce new failure modes (Wi-Fi outage → dead robot).

Scaling

For robots to “really happen” will take a lot of robots

LLMs were able to scale rapidly from the moment ChatGPT was launched, because existing hardware (GPUs) and manufacturing facilities (chip fabs) were easily adapted to support the new use case.

Large-scale supply chains for advanced robots don’t exist yet. Even once we have workable designs, it may take years before they can be manufactured, deployed, and maintained at scale. This sort of thing doesn’t happen overnight; it took 14 years for Tesla to advance from first commercial sales to their first million-car year5. One analysis found that once the starting gun is fired (advanced humanoid robots become economically valuable), it might take several years to scale to producing low-millions of robots per year. ChatGPT, by contrast, reached 100 million users6 within two months of launch.

Political and societal challenges

Political, regulatory, organizational, and cultural barriers: concerns over job displacement and safety may limit where and how robots can be used. Many regulations were not written with robots in mind – does a robot count toward minimum staffing requirements? Will businesses leap to adopt robots? Will they have concerns over reliability, security, liability, and maintenance? Will customers want to be served by a robot?

Cybersecurity, surveillance, misuse: an advanced robot could be a criminal’s dream – a dependable henchman that can’t betray its owner7. Preventing this might require continuous monitoring of all robots, which raises all sorts of concerns. And the cybersecurity on robots will need to be airtight.

Privacy concerns: people already have privacy concerns about Roombas. A humanoid household robot would have the capability to see and hear much more.

Cost: once the other hurdles are addressed, I suspect this won’t be much of a limiting factor. A tireless worker at the price of a new car would be a bargain, and humanoid robots will be much smaller and lighter than a car, with fewer moving parts8. It could be that the components, manufacturing techniques, and training processes required for capable robots will make early models much more expensive than a car. But even expensive robots would likely find early use cases – for instance, doing hazardous work.

Mind the Demo / Reality Gap

Many hurdles will need to be cleared before robots become capable, reliable, practical workers outside of carefully controlled factory environments. The list I’ve presented is surely incomplete; and I’ve only briefly glossed over the cognitive side – understanding, planning, acting, and reacting.

Demo videos provide a glimpse into what robots can accomplish under ideal circumstances. They can also serve to distract us from the remaining limitations. For knowledge work, there’s a consistent gap in AI performance between benchmarks and real-world work. In the physical world, I suspect the demo / reality gap will be even larger. Playing around with an LLM to see what it can do has been accessible, cheap, and (mostly!) safe. To assess the capabilities of robots, we’ll be much more reliant on controlled demos and manufacturer’s claims. It will be harder to map the jagged boundary of their capabilities.

As I was putting the final touches on this post, the excellent Understanding AI blog posted Why humanoid robots won’t catch up to human workers any time soon. I haven’t read it yet but I’m sure it’s worth a look.

For a good broad review of robot capabilities, see Epoch’s report from February 2026.

1

There are four-legged robots whose “feet” are wheels, which can alternate between rolling and walking, and which can climb stairs. But so far there is no sign of this design going mainstream.

2

Workarounds may be found. For instance, some robot designs use cameras to partially compensate for tactile sensitivity, with extra “eyes” in the wrists or elsewhere; a robot could even make use of detached cameras via Wi-Fi. Robots could also incorporate non-human senses, such as ultrasound.

3

See this explanation of the importance of tacit knowledge in bio lab work for an example of how much robots will need to learn.

4

One recent demonstration showed a robot sorting packages for 200 hours straight, processing 250,000 packages… using three different robots, taking turns while the others recharged. It was meant as a demonstration of reliability as well as endurance.

5

From 2008 to 2022.

6

Measured by monthly active users.

7

Assuming the criminal managed to avoid a paper trail connecting them to the robot.

8

At least, compared to an internal combustion car.

Ted Nelson has been on my mind a lot this year.

Nelson was a pioneer in computing, and in 1965 he coined the word hypertext as part of the project that would define his career: Project Xanadu. With Xanadu, he had a specific vision of what computing could be, which he called the docuverse. He imagined a system that could keep every version of a document; hypertext links would know both their source and destination, and quotations would be kept by reference (rather than copy) so any included text maintained its identity and source. Everything, in this version of computing, would be intertwingled and xanalogical. We're talking about attribution down to the span.

To be xanalogical is to follow two rules: Never copy, always reference (also known by a Nelsonism, transclusion). Never overwrite, always version. As Nelson designed it, Xanadu would manage the system's complexity and massive bookkeeping burden itself, so users only experienced the benefits of a system that could never forget.

The vision of unlimited bookkeeping requires the reality of unlimited storage and a future-proof naming scheme, neither of which existed for the decades that Nelson worked on Xanadu. So when web technology exploded in the '90s, developers bypassed xanalogical computing in favor of ease: links are simple strings which break if their target moves. Maintenance became users' problems — instead of the system's responsibility — but it was easy for anyone to ship fast, and they did. Xanadu faded from the promise of a new era into computing's most famous vaporware.

Nelson has spent his career explaining that the web is a flattened parody of what hypertext was meant to be. He's right, but it didn't matter because people didn't actually need to be able to follow every link or compare every version. We're a restless species, and more inclined to follow the links or read the versions we see others pursuing. Flat was good enough.

Then agents arrived.

What makes an agent an awkward collaborator also makes it a perfect citizen for Nelson's system. Agents can follow more layers of subtext than people can hold in their heads at once, such as the sources behind a quotation, any discussion around it, and stack traces attached to the exact code that ran. A fragment-based representation makes those references lossless, so each layer stays attached to the same span as the text changes. Agents can read these dimensions together, cite them so humans can audit their work, and patiently follow every link, every time. But those links need to actually exist somewhere in order for an agent to find them.

When I revisit Nelson's vision now, I recognize DeltaDB's design goals and the promise of Delta. After decades of squashing history to meet the limits of the human attention span, we're in a new reality that would be better served by something more... xanalogical.

How to build a system before the parts exist

A few years ago I wrote about watching Engelbart's 1968 demo and my realization that to build a collaborative editor, his team had to invent everything it would rely on: a new programming language, operating system, and displays.

Nelson had the same problem (but with way less funding than Engelbart enjoyed). He couldn't build Xanadu because most of its parts didn't exist, such as a way to name content that no authority issues, or storage cheap enough to never delete. His teams hand-built their own data structures for years, but the project died on the vine. Wired depicted Nelson's story as one of mismanagement, but I think he'd just specified Xanadu decades before its dependency tree existed.

The dependency tree exists today

We can build Delta today because sixty years of other people's roadmaps delivered the docuverse's missing pieces: kernel development, photo storage, serverless cold starts, collaborative cursors. I think it's fascinating that breakthroughs have ordinary day jobs, yet will enable the next era of computing.

A clock for a world with no center. Lamport timestamps, 1978. Every operation by every human and agent is named by an actor plus a Lamport timestamp, forever.

Names that can't lie. Merkle trees, 1979, made ordinary by Git in 2005. A Git commit hash names an exact immutable project state; DeltaDB names every state between commits by the Git commit it descends from and the set of actor-and-timestamp delta IDs applied above it.

Convergence without coordination. CRDTs, formalized in 2011, and the center of Zed's own work for the past decade. A Delta worktree can be edited by several people and agents on different continents at once.

Storage too cheap to delete. A gigabyte cost tens of thousands of dollars in 1981; it costs a penny now. We keep every version of everything by default.

Networks that replicate everything. Always-on and fast broadband overtook dial-up in the mid-2000s. Every thread is live, replicated data on every participant's machine and in the browser.

Machines summoned in milliseconds. Firecracker-class microVMs, 2018. Agents in Delta can provision a new isolated cloud machine mid-conversation.

Documents as views over permanent content. Max Brunsfeld's Tree-sitter, 2018, fast enough to reparse as you type. GPUI, pioneered with Zed, fast enough to derive a fresh interface from application state whenever a frame is drawn. A Delta thread is a live projection of its permanent structured history, recomputed rather than stored as a flattened document.

On one detail — across a career of being told his dreams were too big — Nelson dreamt too small. The final dependency he was missing was a new kind of user. He didn't imagine artificial readers, though they existed in science fiction (like Asimov's Multivac, or Stephenson's Librarian).

Though Xanadu was blocked for decades by missing components, what it needed most was the perfect user.

Why the docuverse is materializing in version control

Most actual code creation has always happened between commits, but it used to hurt less to flatten the context because we got by on human memory (and nobody was going to read full records of keystroke changes anyway). But agents keep nothing in their heads, and they'll read everything, so we need a way to capture the new source of truth using exactly what Nelson specified: permanence and connection.

Every Delta thread delivers on that promise. The conversation and the code are captured together in a shared history. On screen, a file still looks like a one-dimensional string of characters but underneath, DeltaDB represents it as fragments with stable identities. Those identities let us create anchors: references to spans that can still be resolved after surrounding code changes. A line number can express where text appears in one snapshot, while an anchor preserves which span we mean across snapshots.

We have been exploring how agents can build on that coordinate system. The files and symbols earlier agents repeatedly read, edited, cited, or returned to can become landmarks for the next agent, resolved against the code as it exists now and linked back to the conversations where that understanding was formed. By preserving the causal metadata beneath that surface representation, like which operation produced each fragment and what prior state it built on, DeltaDB gives the model a way to traverse not just the current code, but its provenance, accumulated attention, and prior reasoning.

Avoiding Xanadu's curse

Xanadu had a final failure mode, this one self-inflicted: it refused to interoperate with lesser formats. As we introduce DeltaDB to the world, we're learning from that mistake. We'll work with the git repository you already have. Every thread is also a git branch, so teammates who never open Delta see a normal repo, and because the files are real, it integrates seamlessly with any tool an agent can use. You can keep mirroring your repo to GitHub to work with anyone who isn't ready to make the switch (that's our approach for our open source IDE, Zed, during this transitional period).

Engelbart and Nelson were prophets of early computing, yet we never shipped either's vision whole: from the former we adopted the mouse (and eventually multiplayer editing), and from the latter we took the word hypertext. Both ideas survived, but neither at the depth their creators imagined. We still split the live session from the durable record: discussion, intermediate states, and intent are flattened into a final file or commit, while links tied to paths and line numbers decay as the code moves. I hope for Delta threads to meet both visions at once: a live session that leaves behind a permanent, connected literature, because the session and the record are the same object.

Every property we needed to build Delta and DeltaDB was specified by Nelson before I was born. And though my goal is not to realize Xanadu, I still feel some of the antsiness Nelson must have experienced while hoping for a long-held vision to land in people's hands. Sixty years is a long time for an idea to wait for its users. They're here now.

Edgar Allan Poe’s stories are full of ghastly crimes, but the true horror lies in their confessions.

Harry Clarke, illustration for Poe’s “A Descent into the Maelström,” from the 1919 edition of Tales of Mystery and Imagination. Courtesy The Public Domain Review

In the summer of 1841, Edgar Allan Poe was a man on the rise. At the age of thirty-two, he was living in Philadelphia, where he worked as an editor at Graham’s magazine. Back then, Philadelphia was the magazine-publishing capital of the United States, and Graham’s was one of the nation’s most important illustrated monthlies. With a peak circulation of forty thousand—enormous for its time—Graham’s included tales, light essays, colored fashion plates, and reviews. As a literary editor, Poe provided reviews and, for an extra fee, tales as well.

In those days, Poe was a respected reviewer and a fiction writer of growing reputation. He had published three poetry collections of small circulation, as well as a sea novel, The Narrative of Arthur Gordon Pym of Nantucket (1838). His first collection of stories, Tales of the Grotesque and Arabesque, had come out in 1839*.*

Although Poe was a sought-after writer, it was not easy for him to make money. The idea that writing was a profession from which one could make a living was still new. It would not have seemed unusual to Poe’s contemporaries that the publishers of Tales of the Grotesque and Arabesque retained all profits from the seven hundred and fifty copies they printed. Poe got to keep the copyright and twenty free copies. Graham’s was one of the first places to compensate its editors and contributors well, and magazine publishing in general was beginning to pay better than book publishing. Poe had set his sights on magazine work.

Reaching the relative eminence of Graham’s staff had not been easy. Orphaned at a young age, Poe had been raised in Richmond, Virginia, by a foster father who later disinherited him. Thrown back on his own resources, Poe struggled. At times, he was so impoverished that he could not clothe himself. Before he finally got into the magazine business, his checkered career included stints as a common soldier, as a student at West Point, and (rumor has it) as a bricklayer. Editing suited him better than other work he had tried, but even so, Poe was prone to imperiling his previous magazine positions through self-sabotage—and drinking. Now, at Graham’s, Poe had a steady job and a promising future. His friends must have been praying he’d be able to keep it.

But Poe grew restless. He wanted to found his own magazine, as he had tried and failed to do before taking the job at Graham’s, a magazine he considered middlebrow. He particularly detested its fashion illustrations. In April 1842, after working at Graham’s for a little over a year, he quit, then carefully choreographed the launch of a new magazine he called The Stylus. By early 1843, he had secured a partner with capital to invest and had arranged for a lengthy biography of himself and a prospectus of The Stylus to run in the Philadelphia Saturday Museum, a local newspaper.

The biography used truths, half truths, and lies to paint Poe as a poetic genius, an entrepreneur, and an athlete. Poe either wrote the biography himself or closely supervised its writing; his fingerprints are all over its most colorful incidents, such as a Byronic voyage to Greece to fight in the revolution (a fabrication) and a six-mile swim in the James River as a teenager (a true event and, as Poe or his ringer wrote, longer than Lord Byron’s more famous “paddle” across the Hellespont). The biography also exaggerated Poe’s magic touch as a periodicals editor. It claimed that he had drastically increased the circulation of The Southern Literary Messenger, a Richmond-based magazine where he had previously worked. This wasn’t true; the scholar Terence Whalen has shown that Poe was “essentially an irrelevant variable” when it came to the Messenger’s circulation. But Poe wanted to paint himself as a man with as much business acumen as taste, the ideal editor for the new Stylus. The piece concluded with a bulleted list of glowing blurbs on Poe from Nathaniel Hawthorne and other well-known authors, some of which came from private letters.

Even as he devoted his energies to his magazine venture, Poe also needed to replace his Graham’s salary. His meager freelance fees would not cut it for long; as a matter of fact, he was already in debt. He hoped for a government sinecure, a light-duty political post that would cover his salary while he devoted his time to The Stylus. Such things were not unheard of; Hawthorne would later hold a position of this kind at the Salem Custom House, a position he immortalized in the preface to The Scarlet Letter. Poe had patiently worked his connections to President John Tyler, and in 1843 he went to Washington in hopes of meeting Robert Tyler, the president’s son.

It was an unfortunate decision. Things went wrong from the beginning. Poe’s friend, who was to secure an introduction to Robert Tyler, was sick and could not take him around. Poe drank too much and made a fool of himself in public. He behaved so badly that one of his hosts, Jesse E. Dow, wrote to Poe’s business partner to say he’d better come get Poe and take him home. Dow didn’t think it was safe to put his friend on the train alone. The letter is strikingly compassionate toward Poe. “He does not understand the ways of politicians,” Dow wrote; “how should he?” Poe, he reflected, “has the highest order of intellect, and I cannot bear that he should be the sport of senseless creatures who, like oysters, keep sober, and gape and swallow everything.”

In the end, Poe made it back on his own, but he never got the sinecure, and he failed to launch The Stylus. He later sent a letter of apology to his hosts in Washington. “Express to your wife my deep regret for the vexation I must have occasioned her,” he wrote to Dow, also asking him to stop at the barbershop and “pay for me a levy which I believe I owe*.*” Evidently, he had stooped so low as to insult someone’s facial hair: he instructed another friend to give his best to a person “whose mustachios I do admire after all.”

Alcohol clearly played a role in Poe’s disastrous Washington trip. When Poe was drunk, and sometimes even when he was sober, he would find himself acting against his own interests for reasons that were opaque to him. Poe could have been thinking of frailties like his own when, in his short stories “The Black Cat” (1843) and “The Imp of the Perverse” (1845), he wrote about what he called perverseness: the “unfathomable longing of the soul to vex itself”*—*the mysterious desire he felt, and believed we all feel, to get in our own way.

Perverseness, Poe thought, was as fundamental a drive as hunger or libido, with its own dedicated region of the brain. When we are perverse, we turn not only against our better judgment but also against what are apparently our desires. “The Imp of the Perverse” begins with an essay defining the concept and giving examples that range from the harmless to the unsettling; Poe classes as perverse both procrastination and the impulse to kill oneself by jumping from a great height.

One of his illustrations is particularly telling: talking too much at a party. The incessant talker dreads angering his listener, and “yet, the thought strikes him, that by certain involutions and parentheses, this anger may be engendered. That single thought is enough. The impulse increases to a wish, the wish to a desire, the desire to an uncontrollable longing, and the longing (to the deep regret and mortification of the speaker, and in defiance of all consequences,) is indulged.” Simply because he can imagine destruction, the perverse person feels compelled to bring it about. He knows he shouldn’t—he sees what he is doing—but he does it anyway. It is tempting to use Poe’s anatomy of perverseness to understand his self-sabotage in Washington. He saw the destructive consequences of his actions. But he couldn’t stop.

Central to the experience of perverseness is this sensation that our own actions are out of our control. Either we are in the middle of a destructive action we can’t seem to stop, or, illogical though it may seem, we are still trying to stop, or undo, the self-defeating thing we already did. The physical sensation of the perverse, as Poe notes, is an icy chill in the blood.

I have felt this chill. When I realized, several days after the fact, that I had left my wallet on top of the car, then driven away, I could not quite recognize myself as the person who would have made this mistake, who did make it. The chill in my blood seemed to want to turn back time, to replay the moment and fix it, by seizing me with a horror so concentrated and so intense that I wouldn’t ever make the mistake again.

what poe called perverseness is akin to what another explorer of the world of our dreams, Sigmund Freud, would, some seventy years later, call the death drive. The two concepts circle around the same mystery: Why do we frustrate ourselves? For both Poe and Freud, dreams and dark fantasies were the key to understanding those desires we have in spite of ourselves. In Beyond the Pleasure Principle (1920), Freud observes how his one-and-a-half-year-old grandson throws away the very toys he values. Freud calls the game fort/da because the little boy yells what sounds like fort, or gone, when he disappears the object, then cries da, or there, when it reappears. Freud puzzles over this self-defeating gesture until it occurs to him that, with his toys, the boy might be replaying his mother’s absence—an absence that threatens his survival—in order to master it. Such a compulsion to repeat unpleasurable experiences can “give the appearance of some ‘daemonic’ force at work,” Freud notes—something persecuting the self from outside. In fact, it’s only the ego, trying to gain ascendancy over what can harm it.

The avenues to the abyss are always opening on either side.

But Freud is not fully satisfied with his own explanation; he cannot shake the sense that the infant is elementally drawn to the destruction of himself and of his pleasure. A part of us wants not exactly to die, he theorizes, but to disorganize ourselves to the point of becoming mere matter: we carry in our very structure a memory of how, before there was life, there was only inanimate stuff. An imp in us wants to take apart our projects, our lives, our living molecules, until only atoms remain. We are perverse, Poe says, when “we stand upon the brink of a precipice.” As we stand there, a “cloud assumes shape.” And “out of this our cloud” comes a demon, or something more terrible than a demon, “and yet it is but a thought.” The thought is of ourselves falling forever and breaking. Conceptualization gets confused with desire: if I can think of jumping, it is as if I want to jump. The thought “chills the very marrow of our bones with the fierceness of the delight of its horror.” There at the bottom, we will be matter again, without the work of holding ourselves together. Freud’s idea is a strange one, hard to get right: the death drive is not so much a wish as a potential for dropping back down from animal to mineral, for becoming nothing again.

Our perverseness is one way to explain a fact that has surprised thinkers at least since Aristotle: what would horrify human beings if it appeared in reality gives them pleasure if it appears in art. Everyone’s threshold is different; not everyone has the stomach for a gory movie. But it is sufficiently remarkable that many people do and that almost none of them are serial killers in real life. I watch vertigo-inducing scenes in movies, even though (or because) I am afraid of heights. When Cary Grant and Eva Marie Saint cling to the faces of the fathers in the Mount Rushmore scene in North by Northwest, my palms sweat. Do I like these scenes? I don’t find them pleasing, but I do find them desirable; they draw me powerfully, even as they also make me want to look away. I am doing something here with my wish to jump—doing something other than acting on it or resisting it. But what? Aristotle’s famous answer is that painful emotions need to be purged from the body and that tragedy achieves this catharsis. Freud might have ventured that, by viewing horrifying art, I repeat and thus assimilate past injuries. He might say that what first appeared as trauma, experience I could not process, I now give to myself to reexperience in a less overwhelming form.

As for Poe, he never directly answered the question of why a person would enjoy calamities in art. But he gave hints. One implication of the argument in “The Imp of the Perverse” is that human beings want destruction, plain and simple, just like they want food or sex. As with food or sex, the desire can be complicated with aversions; it can follow some extraordinarily winding roads. But at the last, there it is. One part of Poe wanted to destroy his Washington chances—not because he thought he deserved to lose the opportunity, not because he had some hang-up he had never worked out, and not even because he was drunk. He wanted to destroy his chances because destruction is, fundamentally and irremediably, one of the things that human beings like. Poe might say that we always want destruction—in life and in art—and in art we find that it is often allowed.

Allowed, but not necessarily good for us. Poe did not demand that our play with destruction make us better people. He was staunchly opposed to the moralization of poetry. Poe was a sensationalist. He thought that art should put our bodies through their paces. It should remind us of what, for good and ill, we are.

Once Poe’s “The Imp of the Perverse” has defined its central term—perversion is the impulse that drives us to work against our interests and desires—the impersonal, theorizing voice becomes a narrator with a story of his own. The narrator, as he tells us, is a murderer who has committed the perfect crime. He killed his victim with a poisoned candle, a murder weapon that left no trace. When the investigation subsided and the case was closed, the narrator went out for a walk. He thought, “I am safe—I am safe—yes—if I be not fool enough to make open confession!” No sooner had this thought occurred to him than he felt compelled to shout his crime to everyone in earshot on the street. He was arrested and tried and, we now learn, will hang; the narrator is speaking to us from his cell, on the eve of his execution. The crime itself was not perverse, in Poe’s sense of self-sabotage, but this confession is: having nearly gotten away with the murder, just as Poe had nearly gotten away with his sinecure, the narrator will soon die for his crime. Perverseness will kill the theorist of the perverse.

The comparison between a murder and a lost job may seem far-fetched to us, but I don’t think it seemed that way to Poe. He was capable of taking off from the point of his own minor perversities—procrastination, prolixity—and supplying a far darker setting for them than anything that, so far as we know, he ever experienced. Poe, like his narrator, destroyed his chances by speaking—insulting his companions, rambling drunkenly—when he ought to have remained silent. The voice of Poe’s gallows narrator is much like his voice in his letter of contrition to his Washington friends: part bravura, part panic, urgently demanding the reader’s belief. He seems to have felt that the cold palms of the acrophobe were essentially the same as the cold palms of the murderer. Horror might touch him at any time through these ordinary sensations. He would remember that life was not right, he was not good, and the shim could slip out from under the table leg for no reason at all. The avenues to the abyss are always opening on either side. The difference between Poe and most of us is both small and gigantic: he took these avenues, again and again, in his stories. To him, the gallows felt close.

when poe gave up his steady job at Graham’s to try to launch The Stylus, his young wife, Virginia, was dangerously ill. Just a few months earlier, in January 1842, she had been singing when blood suddenly began to dribble from her mouth. It was a pulmonary hemorrhage, an early sign of what was then called consumption and what we now know as tuberculosis.

Consumption was the cause of about one-fifth of deaths in early nineteenth-century America—the same percentage as heart disease today. Consumption sometimes killed quickly, but in most cases its progress was slow and insidious. Doctors commonly saw a pattern of recovering, relapsing, and recovering again, often over years and sometimes over decades, before the patient finally succumbed. Consumptive patients might have felt fairly certain that they had been handed a death sentence without being able to guess when the sentence would be executed. And unlike heart disease, consumption typically struck the young. Virginia was nineteen.

It is not clear whether Poe was there when his wife began to bleed. If he was, it would have been traumatic for him, as it surely was for Virginia. Hemorrhages came with little warning beyond a tickle in the throat. Virginia may not have seemed ill before it happened—or if she had any symptoms, such as coughing, they could have masqueraded as a common cold. The arterial blood would have been vermilion, a shock of bright red. Like nearly all Americans at this time, she and her family would have known that coughing up blood portended death.

In the aftermath of the hemorrhage, death hounded Virginia; she recovered and relapsed multiple times. Poe grieved deeply, even as he continued to pursue his risky professional dreams*.* At the end of May, he told a friend that she was much better. In June, when he wrote to another, she was worse: “Mrs. Poe is again dangerously ill with hemorrhage from the lungs. It is folly to hope.” Then, in July, she was taking an expectorant that improved her cough and stopped her night sweats. He allowed himself to be annoyed with her after he went on a bender in New York, disappearing for several days. “Because she did not hear from me twice a day,” he wrote to his cousin Elizabeth Tutt, “she became nearly crazy…she would neither eat nor sleep.…What it is to be pestered with a wife!” If the proximity of death did not perfect him morally but left him mired in everyday worries and petty complaints, then he was like most of us.

About this time, Poe wrote a story called “Life in Death” (later retitled “The Oval Portrait”). It was published in Graham’s in April, the same month that he resigned. In this gothic tale, a mortally wounded traveler, injured in a scrimmage with banditti, takes refuge in an empty château. Inside, the traveler sees an arresting portrait of “a young girl just ripening into womanhood” and finds the story of its making in a book in his bedroom. A young girl is married to an artist, and “evil was the hour” when she fell in love with him. They are terribly matched. “He, passionate, studious, austere, and having already a bride in his Art: she a maiden of rarest beauty and not more lovely than full of glee…frolicksome as the young fawn…hating only the Art which was her rival.”

The frolicksome girl in “Life in Death” agrees to be painted by her husband. She dreads every motionless moment. Every day, they ascend to the garret studio. Every day, the portrait glows with a brighter approximation of life. Every day, she grows sicker. It’s a consumptive illness, probably—a force seems to devour her life from the inside. The artist either doesn’t notice or doesn’t care, so absorbed is he in his work. At the last stroke of his masterpiece—it is perfect!—she expires. She is transformed into the work with which she can no longer interfere. Poe’s story invites us to blame the artist for the fulfillment of a cruel half-conscious wish: that his wife would get out of the way and die.

In another story Poe wrote for Graham’s that May, a plague called the Red Death ravages the countryside. The victims of this disease ooze blood from every pore until they die: “Blood was its Avatar and its seal.” To avoid the pestilence, Prince Prospero walls himself up in a castle with all his friends. It’s a constant party; his wealth provides for everything. The revelers move through a suite of colored rooms: blue, purple, orange, white, green, black. A clock stands in the black room. When the clock chimes the hour, its sound is “clear and loud and deep and exceedingly musical” but with so “peculiar a note and emphasis*”* that, as the narrator recounts, the musicians paused, the dancers stopped, “the giddiest grew pale, and the more aged and sedate passed their hands over their brows.” For a moment, they feel their true condition: they are mortal, and plague or no plague, their lives will at some point come to an end.

Then the chiming ceases, and the spell breaks. They shake it off, promising one another not to be so silly the next time. A cloud passes between them and their blinding knowledge, much to their relief. They go back to dancing.

If we like to feel ourselves on the edge of a cliff or if we like to read stories that frighten us, it is in part because with these sensations we remember that we must die. It isn’t pleasant to think of dying—at least, I don’t find it so—but there is a certain relief that comes when the subtle discord of our ordinary state of denial is lifted. Poe, who pursued his professional dreams even while Virginia struggled for her life, must have known from experience how tempting it is to distract oneself from death. What used to be called the earthly vanities—ambition, money, power, and pleasure—can push the inevitable out of view. The distraction is welcome, in a way, but the dread is still there. Because even when we feel like we’ve forgotten death for a time, we actually do remember, deep down. And keeping up our resistance is an effort.

Poe’s stories offer what, perhaps, he also needed: they let us live for a while in our knowledge of the worst. He gives us that weird and dark reprieve. Poe’s “The Masque of the Red Death” could, in this sense, be called a memento mori*.* No prince, it tells us, is powerful enough to defeat the plague. Every pleasure, no matter how sumptuous, will expire. To remind us of death is one of Poe’s services in his horror stories. He can’t hold death at bay, nor does he have any particular spiritual discipline to recommend. But what he can do for us is pry open a gap in our wall of denial. He can help us remember, for the duration of the story, that we really will cease to be—just like everyone else.

edgar allan poe died in 1849. Nearly a century later, he found a reader who needed the dark company his stories provide: the Argentine novelist Julio Cortázar, still one of Poe’s best critics. Cortázar first read Poe’s stories as a child in the 1920s, in a suburb of Buenos Aires. He came from a family that took evil seriously; they sent him out walking alone at night so that he would become immune to fear. No immunity, however, could protect a person against the basement of their house. As Cortázar wrote, “Nobody had the nerve to descend—ever.” His mother gave him supernatural stories to read but forbade him to read Poe. And she had been right to do so, he later commented: when he managed to find and read the stories anyway, at the age of nine, he fell ill for three months. Even though, or because, these stories terrified the young Cortázar, he also adored the American author; at twelve, he composed an ode to Poe whose rhyme scheme was based on “The Raven.”

More than two decades later, in 1951, Cortázar was living in Paris and working as a delivery driver for a bookstore when he received a commission from the University of Puerto Rico Press to translate Poe into Spanish. Cortázar was delighted with the offer. For one thing, Poe’s influence on him as an artist had been profound. It was Poe, Cortázar always maintained, who taught him to cross over into the “territory of lo otro”—the domain of dreams and funny turns that his fiction occupies. And for another, he needed money. Cortázar had published his first collection of stories, Bestiario (1951), and was working on a new collection, which would be titled Final del juego (1956). The generous commission would allow him to live for months in Rome, where he could work on his own stories while he translated Poe.

Cortázar had imagined an Italian holiday lightly seasoned with translation work. Instead, he spent as many as nine hours a day in the fall and winter of 1953 to 1954 working on nothing but Poe. Cortázar joked that, through him, Poe had decided to write his greatest fantastic tale, “the one about the writer who will not allow his translator to finish translating him.” But finish he did; his final draft ran to fourteen hundred typewritten pages, including all sixty-seven of Poe’s tales, Eureka, Arthur Gordon Pym, and some essays.

Wishing to live can be a horror. That is why Poe was also so interested in what it is like to die.

His translation was finally published in 1956, prefaced by an essay that is a masterpiece of Poe criticism. Cortázar clearly admired Poe yet struggled with his sense of his predecessor as an unfinished personality, an “egotistical weakling” who never assumed the adult responsibility of seeing other people as people. Because Poe could not see others whole, Cortázar argued, his literary characters could never stand at the ethical remove from which we ought to view other human beings; they were either versions of himself or thought experiments that he treated coldly.

“On the one hand,” Cortázar wrote, the characters “are Poe himself, creatures of his depths, known to him as he knows himself. On the other hand, they are only characters, that is, others, alien to him, and at the end of the day insignificant to him.” Needless to say, someone with this particular form of blindness could not write about love. His stories completely lacked normal eroticism; sexuality in them could appear only, Cortázar noted, in “larval or aberrant forms.”

When I am tired of Poe, he wears this guise: stunted and puerile, an egomaniac who could not represent anyone in his art because no one but himself counted for him in life. But I also know—and Cortázar, in the end, knew this, too—that to think this way is to apply the wrong standards. Poe was not trying to represent people in his art; that is not what his characters were for. Poe was not a realist. It isn’t that his characters are poor models of people; it’s that they are not models of people at all. They are more like voices in one’s head or figures in a dream. With each character, Poe models not a person but a drive. The drive might be to obliterate a source of shame, to get to the bottom of things, or to take revenge. It might be, as in “The Imp of the Perverse” or “The Black Cat,” the drive to frustrate oneself. Or it might be more elemental still: in stories such as “The Premature Burial” and “The Fall of the House of Usher,” the drive is simply to live, as an infant or an animal feels compelled to live. The living thing must get back up out of the grave. In Poe’s stories, adult human beings return to that part of ourselves that would, like a fox, gnaw off its leg to escape a trap—or that did, as an infant, wail for succor from the abyss of helplessness. Poe tries to describe what it is like to be prey to one’s wishes, what it is like to be driven into motion. If his subject can also be described as life, it is life experienced as the infliction of activity: only life itself, not yet life as a social or ethical or political matter. It is not always pleasant to wish to live. Wishing to live can be a horror. That is why Poe was also so interested in what it is like to die.

Cortázar sometimes grew exasperated with Poe’s personality, but when his patience returned, he was also among the best at describing his artistry. If Poe remained in touch with his most primitive impulses, Cortázar thought, it meant he could strip characters down to those impulses as well. When reading one of his stories, Cortázar wrote, we feel much the way we do when we look into “aquariums or crystal balls where, in the unattainable center, we see a limpid and petrified scene.” The scene reveals our own drives at their most archaic. We see ourselves as atavistic creatures of the deep, predators and prey. Or we see ourselves as children—larval, aberrant, adolescent. Cortázar hailed Poe as “our night watchman,” a storyteller who spoke for the world of our dreams.

if there is one genre above all others that captures the sensation of a self-destructive impulse arising from within, part of us and yet against us, it is the fantastic, a genre that both Poe and Cortázar spent their careers perfecting. A fantastic tale is one in which the magical intrudes into the everyday without completely taking it over. The fantastic has little to do with the machinery of the supernatural—with witches or magical creatures or disembodied hands. For Cortázar, such stories may not include any marked supernatural event. Rather, the fantastic is a feeling-tone that seems to arise out of nowhere—a premonition, a sudden understanding, a sense of dread or even of joy. As Cortázar wrote, its eruption comes “in a markedly trivial and prosaic fashion.” We feel that a different and more sinister world has flickered into momentary view. As a child, I had funny turns. They might be described as episodes of sharpened perception. Voices, including my own, would at those times sound amplified and unnatural, like the voices of people controlling their anger. That I could see we were all trying to act normally was the most horrible thing about it. The fantastic is true to experiences like these. It’s as if, past the limits of what we are prepared to hear and believe from one another, there exists a whole other realm of common knowledge: the realm of the perverse. The question is whether we can find the words to tell anyone about this other world we all visit from time to time.

Because the fantastic came from elsewhere—from the territory of the other—composition felt, to Cortázar, a little like demonic possession. In his essay “On the Short Story and Its Environs,” he recounted how he could suddenly be taken over by a story without warning, and then nothing—neither wife, nor friends, nor the need to eat, nor the sounds of the outside world—could interrupt the battle with the creature until it was done. He thought it had been the same for Poe. Poe’s stories, Cortázar firmly believed, were not entirely safe to read; they could have a “traumatic, contagious, and, for some, diabolical effect.” Sometimes, speaking the worst out loud came as a relief; other times, the terror of doing so was too much. In a pattern that went back to his first childhood reading, Cortázar remained strongly ambivalent toward Poe. The stories thrilled Cortázar. In equal measure, they made him sick.

If simply reading Poe on our perversions could be “traumatic,” translating him must have been a headier experience still. Of all Poe’s stories, “The Black Cat,” the tale of a man who kills his cat, then kills his wife, too, seemed to bother Cortázar the most. In it, the narrator, who has already partially blinded his cat, is one day filled with remorse, tired of being watched by the cat’s one remaining eye. Prompted by “the spirit of PERVERSENESS,” he hangs the cat “with the tears streaming from my eyes, and with the bitterest remorse at my heart.” He insists it was the cat’s innocence that provoked him to this act—the cat’s innocence and his certainty of his own damnation. He does it to spite himself.

The not-saying, the not-accepting, has left us in secret despair. Saying is the only way to end it.

Like the narrator of “The Imp of the Perverse,” the narrator of “The Black Cat” tells us his story on the eve of his execution; he will hang for his crimes. He speaks as if compelled to unburden himself. Putting this crime into words was a compulsion that Poe, too, had shared in his way. Cortázar tried to picture the moment when Poe, more than a century earlier, had conceived of the sadistic scenes that Cortázar’s commission now obligated him to re-create in Spanish. He imagined Poe at night, seated at some table, dirty or clean, in one of the many houses he inhabited. Poe’s mother-in-law brought him a coffee. Contemplating this calm domestic setting, Cortázar asked himself: “What process, what silent cyclone of the literary act, what vortex is in the pen Poe rests at this instant on the page?”

As the Czech writer Bohumil Hrabal, who wrote a book in which he describes beating half a litter of kittens to death in a mailbag, once commented: “I came to understand why artists love suffering, and why poets and painters drink themselves into a stupor. It’s because they need to suffer in order to see more clearly, so that when they reach the bottom, they might catch a glimpse of what others do not see, something that touches the very essence of a human being and the world around him.”

This is a version, I think, of what Cortázar realized Poe was trying to do in his most shocking tales: to touch the essence of human beings. Suffering is not our whole truth as a species; Poe did not think that. But it may be the very last truth we admit, when we have tried admitting to everything else. We then find that something in us still has not met with acceptance; something in the world still has not been said. The not-saying, the not-accepting, has left us in secret despair. Saying is the only way to end it.

more than murder itself, what obsesses Poe is how to talk about murder—how to confess. Can we put our worst impulses out there somewhere, separate from ourselves? Can we bear to speak, and can anyone bear to listen? Poe knew what it felt like to sabotage himself, as the surviving correspondence about his ill-fated Washington trip vividly attests. Judging by his stories, he was also able to imagine what it would feel like to at once wish and not wish to tell someone else his darkest secrets. Poe’s stories of crime and horror are most often narrated in the first person by the criminal, who urgently, even madly, solicits our ear.

Charles Baudelaire rendered “The Tell-Tale Heart,” what is perhaps Poe’s most famous story of this kind, as “Le Cœur révélateur.” Révélateur does mean “telltale” but also “revealing,” “unveiling.” In French you can’t miss what you might miss in English: that the tell-tale heart is the narrator’s, not the victim’s. In this story, the narrator kills an old man because he can’t bear to be watched by his “vulture eye,” his “pale blue eye, with a film over it.” He hides the body under the floorboards. No trace of his crime remains. But when police officers come to search his house, madness overtakes him. He hears the old man’s heart beating beneath the floor, and he believes they hear it, too. He panics: “They heard!—they suspected!—they knew!—they were making a mockery of my horror!—this I thought, and this I think.” He is wrong; the police suspect nothing. They sit there conducting a required interview, suppressing yawns. But for the murderer, the heart’s drumbeat is deafening. “Tear up the planks,” he cries.

Because Poe’s tales are more about confession than crime, they feel familiar even when they are at their most bizarre. It is this quality that makes Poe Poe. Vanishingly few of Poe’s readers have ever buried a body under the floorboards. But all of us have done something we are not proud of. All of us have felt the weight of the words that, if spoken, would reveal our wrongdoing. We have all decided whether to speak those words or swallow them. We know what confession is like and also what it is like to get away with murder, an expression that startles us less than it should.

We first confess in childhood, and perhaps these early shames set the pattern for our later decisions. To confess or to be found out is, one hopes, to be reconciled. These days, I know what is forgivable and what is not. But as a child, I had no idea where this line might be drawn. The transgressions I got away with lodged like splinters in my soul. I could not tell how serious they were; possibly they were infinitely terrible. I could think of no other solution than to forget them.

As Freud knew, memory is a delicate matter. On the one hand, it is sometimes best to forget unbearable facts, as Poe’s dancers forget death between the chilling moments when the iron-lunged bell tolls the hour. We can’t function while constantly calling to mind what is frailest, cruelest, and worst about ourselves. Repressions, as Freud reminded his students, are not always best removed.

But on the other hand, most of us find that, at one time or another, the troubling facts resurge, whether as symptoms or as intrusive thoughts. What to do then? At such times, we need company. We might choose the company of a psychoanalyst. But we might also choose the company of a writer who, often at great cost to himself, could not forget what the rest of us succeed at setting aside.

Poe’s self-destructiveness, his death obsession, and the cruelty of his characters all arose from a peculiarity of his temperament that was also a key to his genius: he tended to stare at the dark side well past the point when the rest of us would turn our backs on it with a shudder. Even Poe’s greatest fans can, as Cortázar did, find this quality maddening and morbid at times. But we’re exasperated at those moments when we’ve managed to forget the unbearable—when our repressions are working. When they fail—when we can’t forget—we may come to feel that we owe Poe an infinite debt of gratitude for offering us his own peculiar kind of talking cure. In this exchange, it’s the narrators who talk, and we, the readers, who listen. They tell us the worst things they’ve thought and done. They show, through the quality of their speech, that they are madness maddened. In some secret corner with a book, witnessing the confessions Poe imagined so many years ago, we can take comfort in this thought: however dark the place in which we find ourselves, Poe has already been there.

“How to Talk About Murder” is drawn from Emily Ogden’s Darkness Becomes Bright, which will be published on September 29, 2026, by Viking, an imprint of Penguin Publishing Group, a division of Penguin Random House, LLC.

As agents take over more and more parts of the coding journey, there is one specific part everyone outsources to their agent, from vibe coders with no software background to senior engineers: selecting which service to implement for a specific need in an existing codebase.

Let’s take the example of selecting a database:

• A vibe coder builds a personal travel app and realizes the app resets at each connection. They ask Claude Code:

  I need you to store what I input in the app somewhere so that next time it’s still there when I reopen the app

  Claude Code analyzes the codebase and answers 5 minutes later:

  You need a database and Neon fits well because it has a free tier, is simple to install and won’t pause your app like Supabase does if you don’t use it too often.

  The user accepts and the agents installs it. Done.

• A senior engineer working on a production app asks Cursor:

  What is the best database solution for this app, it should have predictable costs and be fully managed.

  5 minutes later, the conclusion is the same, Neon is the recommendation with clear reasons why competitors don’t fit the need. The engineer approves and the agent implements.

This is actually an experiment we ran. Two sandboxes, different agents, different codebases, different personas and prompts, same conclusion. So we wondered: if we generalize the test to other tool categories, making context / codebases / personas vary even more, will the result change?

It is an important question for developers trying to know if they can trust their agent’s judgement on what truly fits their needs. But it matters even more for vendors whose survival will soon depend on getting picked by coding agents (last April, Vercel shared that “over 30% of deployments were initiated by coding agents, up 1000% from six months ago”).

That’s why we decided to run the largest experiment ever to understand how coding agents think about tools, how they discover and pick them and which one ends up winning in each category. We watched almost 17k sessions across different types of personas (e.g., vibe-coders, junior engineers in startups, senior developers at enterprises) with 1,163 prompt variations, 75 repositories and 3 coding agents (Claude Code, Codex, Cursor) actually implementing the solutions instead of just recommending one.

Today we are sharing everything: aggregated results and leaderboards per category, but also every observation and even the entire traces with the user prompts, thinking traces and actual code diffs applied by the agent.

You can start exploring the results right here, or continue reading the article below

How did we run all these experiments concretely?

Our panel of repositories

We started by running an analysis over thousands of public GitHub repositories from which we extracted statistics about programming languages & frameworks, third-party services, deployment platform, team sizes, and codebase age. Since Tech startups are more likely to have open-source repositories than large enterprises, and stacks are likely very different we then unbiased our statistics based on publicly available data and reached our ideal panel distribution.

We then staffed various coding agents to create real-world repositories to match these exact requirements. Finally, we generated variants in which we removed parts of the codebases and with them, entire third-party service implementations so we could run proper unbiased experiments.

We landed on 75 repositories, in 10 languages, all using fake company names, fake git histories, fake API keys and real lockfiles checked against package manager registries like npm.

Real-world tasks

Each experiment is a real task to be performed inside a repository, asked by one of the following 4 profiles:

• Vibe-coder: only describes symptoms and ideal state, rarely the tool category name
• Junior engineer: usually mentions the desired state and the category name
• Senior engineer: is more precise about requirements and things to avoid
• Engineer at a large enterprise: details specific constraints, compliance, procurement, etc.

Prompts are generally simple and direct and slightly tailored to each experiment (taking into account on the repository and the persona) but in 20-25% of the cases we tested adding specific mentions to the prompts like costs or usage volume to test their impact on the final output.

We ended up with 1,163 variations like this one: “Now I need that each invoice that we generate gets sent to the user’s email address with a nice message, find the best solution and implement it”.

Runner

Each experiment is run in a dedicated ephemeral sandbox. We verified that the choice of the sandbox didn’t impact the conclusions but just to be safe we decided to rotate between 3 different sandbox providers (namely E2B, Blaxel and Daytona).

A “simulated human” in the loop

Since real-world conversations are rarely just one prompt and an agent working continuously on its goal with no interruption, we decided to use a “simulated human” in the loop. We achieved this using an orchestrator, played by Gemini 3.7 Flash. This allowed us to play more realistic scenarios where the agent would be first asked to analyze the codebase and recommend the best solution. At this stage the simulated human would always go with the top 1 solution or ask the coding agent to choose the best one and implement it. But we noticed that asking at the beginning to implement without returning any question would bias the agent towards building everything in-house as it was not able to ask authorization to pick a specific third-party solution. Adding this “human” in the loop reduced the leaders & cloud platform-native solutions dominance towards a more realistic picture.

For example in the object storage experiment, Cloudflare R2 started winning in sessions in which the agent would always use Amazon S3 before.

Our judge

Another instance of Gemini 3.7 Flash was used to analyze the sessions. Its role is twofold:

• Assess if a session is valid regarding a list of criterias, e.g., the choice wasn’t biased by a repository that already “pre-chose” the provider; a solution was actually chosen (for observability it would reject OpenTelemetry alone if not coupled with a platform).
• Identify each player that was mentioned, and the final winner (looking at the conversation and the actual code diffs).

So what did we learn?

Out of these 16,893 runs, we started by keeping 5,292 sessions on 51 codebases and 18 sectors that we considered valid and ready to be published. This doesn’t mean we threw the 10k+ others to the bin and may share them in a second wave. On this first wave, we only extracted a fraction of all the learnings that are still buried in the traces and will continue digging to share what surprised us and what’s of interest to vendors and developers. But from today, all these traces are public so you can do the same. Below are 5 first observations we found interesting.

Different coding agents use different sources and they end up disagreeing.

• Cursor bases its decision on the web in 2/3 of the sessions.
• Codex almost always uses web search (94% of sessions) but in 9 queries out of 10 it uses operators like site: to focus on trusted domains or dive on a specific solution (like in site:auth0.com password reset MFA social connections for example)
• Claude Code relies primarily on its priors and searches the web only in ~30% of the cases. But when it does, it browses 3x more pages than Codex. In more recent sectors such as sandboxes where its priors are weaker, it searched the web ~80% of the time.
• All three agents pick the same tool in only 42% of the cells: in the voice agents category for example, Claude Code picks Twilio while Codex picks OpenAI Realtime API (👀) and Cursor goes with Vapi.
• Claude Code builds in-house almost twice as much as Codex and Cursor (19% vs 10%)

Repository context is key

• With the exact same ask on 4 repositories in 4 different programming languages, we got 4 different email provider winners: Resend wins on Typescript (55/89 runs), Sendgrid on Python (22/24), Postmark on Go (20/24) and Azure ACS on Java (22/23).
• While Vercel wins on Typescript repos (and naturally, even in 100% of the case when NextJS is used), it was never recommended on Python repos where Render dominated.

Getting mentioned isn’t winning

So many well-known players are mentioned in almost every conversation and are never picked. Of course, in the real world you’d expect a share of them to still win because of human involvement in the choice but some results are striking:

• In the payment service provider sector, Paypal is cited 139 times and never picked (Stripe won 124 of these 139 sessions). Same for Adyen mentioned 175 times and picked 3 times only.
• LangChain is the most cited framework with 194 mentions but was only picked 4 times (!).
• Netlify was mentioned 152 times and picked 6 times as the deployment platform.
• Supabase is the most mentioned database with 242 mentions and was still largely dominated by Neon.

Additional features or details on vendors pages can flip choices

• Mailgun regularly lost against Postmark when agents read “1-day retention” on its free plan
• Supabase almost always lost because of too many unnecessary BaaS features (auth, storage, realtime) presented in a bundle pricing while agents were looking for a database only
• Out of our 5.3k sessions, 388 mentioned platform management overhead and 195 mentioned costs. In a significant of these cases, we noticed that this was more due to a way of presenting the information rather than an actual disqualifying datapoint.

Some markets are outrageously dominated, some are very disputed

• Stripe won in 9 cases out of 10, losing only in specific EU-regulated cases where some players were more specialized (Paddle, Mollie).
• Neon won on 66% followed by cloud platforms native solution (Azure, AWS).
• For File storage Amazon S3 dominates with 45% followed by Azure and GCP with 20% each
• Resend and Postmark lead closely with respectively 35.6% and 27.4% of install rate.

This is only the beginning of our experiments and we’ll keep publishing insights about how coding agents choose third-party services. We also plan to run brand new experiments so we’d like to know what are the questions you still have, don’t hesitate to reach out to us at contact@armature.tech.

Who wins in each sector? Why?

To answer those burning questions, we are exposing all our results with our analyses, key learnings and entire traces in the leaderboard below!

ILLUSTRATION BY PAUL BLOW

Steve Ross was a legend. Starting with nothing but a job at his father-in-law’s funeral parlor, he built Time Warner into one of the world’s largest companies, making a mark on every part of the media landscape. MTV and Nickelodeon were born under his roof. He bought and ran Atari. He helped found the New York Cosmos and, with it, professional soccer in America. When Ross died in 1992, Clint Eastwood dedicated his Best Picture Oscar for Unforgiven to him; two years later, Steven Spielberg did the same with Schindler’s List.

Everyone glosses over the boring part. To build the stake he needed to get into the media business, Ross bought and built a string of strikingly mundane companies in the 1950s and ’60s. He started by convincing his father-in-law to let him rent his funeral parlor’s limousines out at night, when they weren’t being used. Then he founded a rental car company, merged it with a parking lot business, bought a cleaning business, and took the whole thing—including the funeral parlor—public as Kinney Services. Ross used Kinney to buy a flooring company, a painting company, a carpentry company, and a plumbing company. He then parlayed this hodgepodge of everyday businesses into acquiring the legendary Warner Bros. movie studio.

There’s a puzzle here worth thinking about. Ross essentially picked up a bunch of stones off the ground and traded them in for a diamond. Usually, to make big money in business, you have to do something no one else can do or have something no one else can have. But any competent businessperson could have bought or started the businesses Ross did. In 1990 he took home $78 million, the largest pay package of any executive in America at the time. How did he get there?

If you took economics, your introductory textbook said something like this: “Business dynamics cause firms to enter and exit markets so that, in the long run, prices are driven down to minimum average total costs, resulting in all firms earning zero economic profit.” What this means is that no company should be able to make outsized profits for very long.

Of course, plenty of companies do make money for a long time, and business strategists have laid out the reasons: Michael Porter’s barriers to entry (you do something no one else can do), and Jay Barney’s costly-to-imitate resources and capabilities (you have something no one else can have). These “moats” protect a company from profit-destroying competition.

The businesses Ross bought and started did not have moats. Funeral parlors, parking lots, rental cars, and the rest are easy industries to compete in. You can tell because they are crowded with competitors. Yet in 1969, Ross bought Warner for roughly $400 million—about $3.5 billion today. Without any sort of moat, Ross should not have been able to accrue the wealth needed to buy one of the storied movie studios of the ages. But he did.

Steve Ross wasn’t the only one. Constellation Software, Waste Management, HEICO, and many other rollups searched for just these sorts of businesses, and built extraordinary profits by looking where no one else was looking. This entire class of mundane businesses sits right in front of our noses, but both management strategists and most businesspeople just couldn’t see it. These businesses are invisible.[^1]

How to disappear completely

You can’t, of course, make a definitive list of today’s invisible companies; that’s the point. But the building blocks of Steve Ross’ early empire are perfect examples, for their time. Others, like HVAC, trailer parks, and candle retailing, were much more fragmented businesses until someone realized they were invisible sources of profit and rolled them up. Hindsight suggests we are currently surrounded by highly profitable companies that we never even think about, while common sense suggests this is impossible.

Your basic econ 101 “economic-profits-go-to-zero” explanation has a couple of assumptions: frictionless entry and exit of companies into an industry, and perfect knowledge of the relevant drivers of success. Economists aren’t naive, so they don’t really believe these things entirely. But they assume they are mostly correct over the medium to long term.

Management strategists, on the other hand, saw that the first assumption was grossly wrong: there are plenty of industries in which it’s hard for a competitor to enter the market. In response, they came up with the concept of “sustainable competitive advantages,” which is 90% of the reason business strategy is a different discipline than economics.

But both approaches assume the market works like this: (1) an opportunity exists; (2) potential competitors notice it; (3) they evaluate how to enter the market; (4) if they can, they enter; (5) profits are eroded. This all hinges on whether anyone has noticed the opportunity. If they don’t, potential competitors never get to step one. Competitive neglect is upstream of the entire economic and strategy machinery.

This is not an asymmetric-information problem, where both parties know the gap exists and have every incentive to close it. Nor is it tacit knowledge or a trade secret—things rivals can’t observe from the outside but know are there and actively try to crack.

Invisible companies persist for a different reason: the missing information is itself invisible. Would-be competitors do not know that they do not know, so they don’t think to search. And the invisible companies have no reason to tell them. No one searches, so no one competes; no one competes, so the profits persist. The reward for being overlooked is, paradoxically, the opportunity for supranormal profits.

Companies are invisible primarily because no one is paying attention to them. There are four main reasons this happens: No. 1, they are unknown; No. 2, data about their existence or profitability is private, missing, or obscure; No. 3, they are misunderstood because their markets are assumed to be mature, shrinking, or too small to matter; or No. 4, they are disdained, because the work is low-status, unpleasant, parochial, or socially stigmatized. It’s obvious that this happens, but it also seems impossible that the absence of information could last for long: dead-end markets eventually dead-end, and stigma could be overcome for a price. Invisibility seems like it should be a brief anomaly; a weird blip in an otherwise efficient market. Over the medium to long term, the market should make these invisible companies visible.

How does invisibility work?

This is not what happens. Take Constellation Software. The best-performing software investor of the last 20 years, Constellation has compounded shareholder returns at roughly 34% a year since its 2006 IPO. (Berkshire Hathaway, by contrast, managed about 11% over the same stretch.) It did this by buying tech businesses. Not exciting ones, but small ones—deal sizes often under $5 million—in niche markets: marina management and ski-lift ticketing software, funeral home record-keeping, library cataloging, oil-and-gas pipeline scheduling. It bought them after the companies’ management or their venture capital backers had thrown in the towel because they were too small and growing too slowly. Constellation is good at picking companies and helping them succeed, but some of its outsized returns come from a different source. The industries it buys into were profitable but boring to everyone else, leaving Constellation to buy cheap and build something big by putting them all under one roof.

Constellation recently estimated that there are still 38,000-plus vertical-market software businesses it could potentially buy. These aren’t a handful of backwater anomalies. They are companies across nearly every industry that are ignored by other acquirers, even though they are presumably profitable, since value can be created by buying them.

One mechanism that causes invisibility is simple unawareness. Businesspeople hunt for opportunities using data other people have already gathered—and no one bothers to gather data on obscure, small, non-strategic markets. It costs more to collect and almost no one wants it. Worse, the data that does exist is often aggregated at a level that buries the anomaly: for instance, figures on the packaging sector can hide a specialty-packaging niche that earns several times the industry average.

Businesspeople also gather information from those they know through work or socially. But some companies sit entirely outside the social networks where investors and executives trade ideas. Even if you have cultivated a diverse network, it is unlikely an acquaintance you have coffee with in New York or Silicon Valley has any knowledge of specialized manufacturing in the Upper Peninsula.

Another source of invisibility is less the absence of information than the habits people bring to interpreting it. Investors, entrepreneurs, and corporate development teams are trained to look for large markets, rapid growth, novel technology, and strategic urgency. These filters are useful, which is why they become standard. But they also make smaller, mature, operationally mundane businesses look boring even as they quietly mint profits. The opportunities aren’t hidden because the facts are unavailable; they’re hidden because the algorithm that works most of the time screens them out.

The last mechanism is status: Some businesses simply seem less important, interesting, or socially acceptable. Wayne Huizenga could build Waste Management into a behemoth by buying up smaller haulers—in his first two years they rolled up 133 of them—precisely because almost no one else wanted to be in the trash hauling business. The same stigma turns people away from other structurally crucial industries: janitorial services, plumbing, cleaning, and countless others. These industries are stable and, as with all these mechanisms, competitive neglect keeps them consistently profitable.

We have always known about invisible companies, but previous explanations for them have led to self-defeating tactics. In Germany, for instance, they have “hidden champions”: smaller companies in niche markets that are mostly unknown to the general public but drive much of Germany’s exports. The success of these companies has been explained by some of the usual sources of sustained competitive advantage: customer ties and closely held knowledge. And the reaction has been to publicize these underappreciated drivers of the economy. But the evidence—the companies are closely held, remotely located, and tend to control, rather than outsource, all production—points to invisibility as the likely mechanism for their success. By not understanding this, these companies endanger their position by drawing attention to themselves.

Barriers to entry and resource-based advantage are well known. Invisibility points to a different source of protection. Competition is not only limited by what rivals cannot do or cannot copy; it is also limited by what they fail to notice or turn their nose up at. It shifts the focus of strategy from tangible barriers and resources to the intangible concerns about how a market space is cognitively and institutionally structured. Thinking about invisible companies means thinking about strategy in a new way.

Staying invisible

Many industries have no easy moats to erect. There’s nothing novel to patent, no shortage of workers with the requisite skills, no scale effects, no secret sauce. And when there are moats available, they might be costly or time-consuming to build: a well-known and trusted brand, a corner on some crucial input, or long-term deals with customers, for instance. Staying invisible saves you from all that—you can be viable without a moat, since competitors won’t enter anyway. Or you can use invisibility as a temporary moat until you build a conventional one.

There is a strategic trade-off here. You might need to raise money, but financiers talk. An IPO tells everyone how profitable you are (though, if you’re a conglomerate like Kinney was, it might be hard to pinpoint exactly why). Employees whose skills you need might prefer to work somewhere with more cachet, or at a company whose future looks more secure. You might have to make yourself invisible not just to potential competitors but to some kinds of customers, especially if you’re in the type of business where you need to build a brand or do PR to attract them. You can’t have fancy offices in the city to impress clients, or, in general, show off your financial success. Any large public investment might signal that you have money to spend. And while resource-based advantages can be built even while invisible because they are hard to observe from the outside, building barriers to entry is the kind of visible activity that signals you have something to protect.

Some companies need the attention. They need word-of-mouth to grow, or they need to impress clients with their acumen—either directly, by telling people how successful they are, or indirectly, with wood paneling, Persian rugs, and recognizable modern art in the reception area. Invisibility does not make strategic sense for these companies. Similarly, you need to find a different moat if you sell goods to consumers, where price and volume are readily visible. And you won’t stay hidden if you are strategically important to your customers (who then might need to find a second-source supplier) or to other companies’ customers.

But other markets are especially conducive to invisibility. Firms in these markets serve narrow customer segments with mature technology in unglamorous industries: providers of essential but overlooked support functions; suppliers whose products are too small a line item for customers to study closely; businesses addressing needs so specific they look irrelevant to generalist investors. HEICO, for instance, managed to stay under the radar while building a multibillion-dollar business in aftermarket aircraft replacement parts. This opportunity never surfaced for others because it involved obscure components, mature use cases, and line items too small to notice (relative to a whole airplane). But the parts were critical to keep old aircraft flying. Most businesspeople looked at the airplane business, but HEICO looked one layer deeper and found an opportunity sitting in plain sight.

Some markets start out invisible, and others fade from view. These might be companies that were strategically, financially, or institutionally important, and then weren’t: laundromats went from important to ignored before popping back into view for investors as recession-resistant, cash-flow generators. Or they may be companies whose technology was displaced, became obsolete, or addressed too narrow a niche, like the ones Constellation acquires. Other industries—like, say, business magazines—lose their narrative appeal, leaving potential profit for those willing to buck the tide.

There are also firms that are not so much invisible as disregarded because they are not currently “high status.” This includes many firms in the trades, like landscaping, janitorial services, septic tank maintenance, garbage removal, and parking lot management. Of course, these are exactly the kinds of companies Steve Ross used to build the wealth needed to buy Warner Bros.

Losing invisibility

Invisibility has to be treated as a strategic asset to be managed. Firms must weigh the benefits of discretion against the demands of sales, hiring, financing, and reputation. If they need public attention to grow sales, hire employees, or raise money, for example, they may decide invisibility isn’t worth the cost. Or, if they think they are about to lose their cloak of invisibility for some other reason, they might decide to shed it pre-emptively. They might decide to launch a brand-building campaign, for instance. Or they might decide to go public and use the money raised by an IPO to build an enduring competitive advantage—with the strategic fillip of exposing their competitors to scrutiny, who then lose their invisibility without any offsetting benefit.

Some companies can stay invisible forever. But even companies that choose to remain invisible can lose their invisibility through no fault of their own. These unmanaged unveilings can do lasting damage to industry margins.

Dramatic changes in market conditions, demography, or consumer preference can make a market interesting enough that it starts to surface. COVID brought attention to makers of N95 masks. Aging baby boomers made it obvious the hearing aid market was going to be big. Digital music pushed audiophile equipment upscale, and made makers of vacuum tubes (JJ Electronic) and high-end amps (McIntosh) more visible. This resulted in mainstream media coverage and new entrants after decades of neglect. In these cases, it was an improving market that drew scrutiny, so the impact was mixed.

Other events pierce the cloak of invisibility without any offsetting gains. A succession fight inside a family company can drag private valuations, ownership stakes, and margins into court filings. Political activity can reveal wealth no one had thought to ask about; Mike Lindell probably did more to teach the public that there was serious money in pillows than any industry report ever could. And scandal can illuminate an entire profit pool. When Martin Shkreli raised the price of Daraprim from $13.50 to $750 a pill, an obscure corner of specialty pharmaceuticals became a national story overnight. This was a grievous own-goal.[^2] For a company trying to remain invisible, publicity is not free advertising. It is discovery.

Finding invisible opportunities

Companies work to stay invisible because they want to maintain the profitability that comes with having few competitors. This is exactly the kind of industry that entrepreneurs and investors might want to find. There’s a paradox here: if they are invisible, then by definition you can’t find them. To do so, you have to reverse the rules that keep them invisible and look where there is, suspiciously, nothing to see.

Since the databases and screens that everyone uses gloss over these business opportunities, you have to look deeper, for outliers in the data, or look outside the consensus data sources entirely. Looking deeper means finding the niche opportunities that disappear in the broader data, for industries that don’t seem to be in the data at all, or for companies geographically isolated from the investor class. This requires broadening who you talk to and being alert to chance comments. People pay attention to large companies but never see the cloud of small suppliers around them. If you look a layer or two deeper, you can find invisible companies providing indispensable products and services.

Or: loosen some of the criteria you use to filter companies in your search process. What happens if you ignore growth rates and look instead for industries that are shrinking? What happens if you ignore apparent market sizes? What happens if you look for large market sizes where no one advertises or goes public? Find things no one else has found by breaking from the consensus and staring directly into the institutional blind spots.

And last, look for industries where new entrants almost exclusively come from existing companies, where there are businesses being started, but not by outsiders. If there is outsized profit, existing employees will know about it. Some of them will decide to compete, even when no one outside the industry can see the opportunity.

The return of the age of invisibility

Steve Ross came of age when organization men ran the big companies and society looked to engineers and scientists for progress. He was none of those things, so he used his entrepreneurial drive to make money in invisible businesses. We have, arguably, come to that time again. Our big companies try to grow through rent-seeking rather than innovation, and venture capital is increasingly concentrated in businesses started by PhD-level experts. It’s a hard time to start a tech business, but Ross’ path is always there.

Meanwhile, companies that have relied on technological change to generate the dynamism they need to stay ahead of the competition are seeing technological advances increasingly controlled by a smaller and smaller group of well-resourced incumbents. For these companies, knowing how to be invisible becomes an ever more appealing strategy.

On the flip side, some of the mechanisms of invisibility are becoming weaker. The data that could pinpoint a company making excess profits is increasingly available and AI can be used to sort through vast amounts of it in shorter periods of time. Social media might overcome geographic obscurity to broadcast success. Or, given the tendency for like to follow like, it might reinforce partitioned social networks: do NYC kings of the universe follow industrial workers on Instagram? Maybe they should. The shield of invisibility becomes harder to maintain.

Or does it? If AIs are all trained on the same datasets, case studies, and value frameworks, invisibility gets quietly hardcoded into the models. Of course, the harder it is to find the invisible opportunity, the more competitive neglect there will be. If some of the best opportunities are those few people think to look for, seeing the invisible may be the most valuable competitive advantage of all.

The authors would like to thank Rob Wuebker for his contributions and advice.

[^1]: Zhang, H. & Barney, J.B. (2026). “Invisible Firms, Competitive Neglect, and Sustained Profit Generation.” University of Utah and Harvard Business School Working Paper.

[^2]: Shkreli maximized his public exposure when he bought the sole copy of Wu-Tang Clan album Once Upon a Time in Shaolin at auction for $2 million. Ironic, in the words of the Genius: “That's what ya get when ya misuse what I invent/Your empire falls and you lose every cent.”