In the spring, we hit a pretty wild milestone: Oxide paid income tax. And not because of some unusual transaction or one-time event, but because our ordinary operations (selling computers!) generated taxable income after accounting for the costs of components, manufacturing, salaries, and running the rest of the business. Of course, we each individually pay income tax every year; is it that surprising that a company is doing what we all do every spring?

Well, yes, it is: most startups don’t pay income tax because most startups aren’t profitable! Indeed, startups seek investment because most startups have costs long before they have revenue, let alone gross profit — let alone income. This is by design: profitability is a lagging indicator of product/market fit (the adventure in venture capital is investing long before the business has materialized!).

Figure 1. Us being as excited as you can be paying taxes

So this was a big milestone, and on top of that (and contrary to Russ Hanneman’s admonition!), we have a very large order backlog. On the one hand, it’s truly extraordinary to watch the business generate cash, but with demand far exceeding supply, significant cash is needed to secure inventory and fulfill these orders. This is one of the peculiar dynamics of a rapidly growing hardware business: we must commit substantial cash to components and manufacturing well before the resulting systems reach customers.

Now, between our Series B, Series C, our existing debt facilities, and the cash generated by the business itself, we felt confident we could satisfy our current backlog, but the absolute numbers are large enough that we would have had to exercise real caution in accepting additional demand. We are, after all, children of the Dot Com bust — cautious by nature — and would not put the business in a position where new supply disruptions, economic shocks, or other events outside our control could jeopardize it.

Fortunately, this is the problem that capital is born to solve. Our investors saw this too: they have been extraordinary believers in Oxide from the beginning, and, thrilled by the burgeoning demand, they wanted to be sure that we were properly positioned to fully take advantage of Oxide’s large market.

As they have so many times for us over the seven years of the company, Eclipse stood up to lead, and our $445M Series D quickly came together. Existing investors USIT, Riot Ventures, and Jane Street signed up for big pieces — and others on the cap table including Friends and Family Capital and Counterpart eagerly joined in.

We are deeply appreciative of the support of these existing investors, but we also wanted to add new investors to the company. While there was a lot of outside interest (profitability being the ultimate VC aphrodisiac!), there was one firm that stood above the rest: Atreides Management, which first got to know us nearly two years ago and has stayed close to the company ever since. We love their analytical approach, their courage in hard-tech investing — and their belief in Oxide.

Finally, we are thrilled to welcome AMD as a new strategic investor. AMD has long been a believer in Oxide: it was with their support that we were able to achieve breakthroughs like our own platform enablement software. And of course, we have been big believers in AMD: one of our first big bets was on AMD EPYC. We think the partnership between AMD and Oxide is a lasting one, and it’s fitting to have that expressed in the cap table.

All of these investors are — like us — in it for the long haul. When we raised our $200M Series C, we said that it was to entirely de-risk the company with respect to capital, to assure both our longevity and our independence. This Series D builds on that assurance to allow us to satisfy our substantial backlog while continuing to accept new demand, expand manufacturing capacity, and invest for the enduring company that we have always set out to build.

To our investors, partners, fans, long-time listeners, employees, and most of all customers: thank you. While this is but a milestone on the long road to what we know Oxide will be, it is nonetheless an exhilarating one!

Announcement, Nobel Peace Prize 2026

The Nobel Peace Prize for 2026 goes to a person who with exceptional courage and integrity has led the way towards a more comprehensive global legal order. This year’s laureate has been instrumental in ensuring that war crimes, crimes against humanity and genocide are prosecuted.

The Norwegian Nobel Committee has decided to award the Nobel Peace Prize for 2026 to Navanethem “Navi” Pillay for her efforts to promote peace and international law.

Navi Pillay’s commitment to universal legal principles and her firm moral compass are constants in a long career. Born into a family of Indian Tamil origin under apartheid in Durban, South Africa, Pillay became a lawyer and legal pioneer, confronting deep structural discrimination, segregation and exclusion. A common thread runs from her early work defending Nelson Mandela and others who stood up against apartheid to her service as a judge in some of the key international court cases of our time.

This year’s laureate has significantly enlarged the scope and impact of international law. She has served as a judge on the High Court in South Africa, the International Criminal Tribunal for Rwanda and the International Criminal Court. She was the United Nations High Commissioner for Human Rights and, until recently, she chaired the UN Independent International Commission of Inquiry on the Occupied Palestinian Territory. Today Navi Pillay is a judge on the International Court of Justice in the case where Myanmar stands accused of genocide.

In historic international court cases, she has shown that legal measures can help prevent acts of war and violence. Navi Pillay has also contributed to strengthen the institutions of international law. Her independence, expertise and steadfastness have made her one of the most respected international jurists of our time.

The first Nobel Peace Prize was awarded 125 years ago. Already then, the Norwegian Nobel Committee underscored the importance of resolving conflicts through international law – a theme that has remained prominent throughout the history of the peace prize. The reach of international law expanded after the Second World War, in particular with the creation of the UN institutions. The key insight was that lasting peace must be built on legal principles, not on the will of the strong or a fragile balance of power. Additional legal mechanisms were introduced in the post-war decades, and momentum picked up further after the end of the Cold War.

However, the rules-based international order has never been perfect. The greatest powers have often evaded responsibility. In practice, cases brought under international law have produced different results depending on who is the perpetrator and who is the victim.

Nonetheless, we have seen that more and more conflicts are being addressed through diplomacy, treaties and legal mechanisms, rather than threats, violence and warfare. Leaders and populations alike have come to the same conclusion: international law is a foundation for a more peaceful world.

At a time of existential challenges – including more wars and conflicts than the world has seen in a long time – international law is no longer just a supplement to peace and security. It is an absolute necessity.

And yet, the system of international law is under tremendous pressure, and its institutions are under attack. We see a shift towards power politics at the expense of legal frameworks and global respect for law and justice. Whenever might overtakes right, it undermines stability, trust and peaceful co-existence.

In awarding this year’s Nobel Peace Prize, the Norwegian Nobel Committee wants to remind the world that the rule of law – national and international – underpins the system we have set up to ensure peace and resolve conflicts non-violently. Peace requires justice.

History has shown that a world operating without the rule of law is unjust, unwise and inhumane. The judges – those who guard the thin red line between order and chaos – are therefore highly deserving of our praise. But today, as these same judges are sanctioned, and their institutions are attacked, we need people of courage, vision and determination – qualities that Navi Pillay demonstrates in her work on the bench.

Navi Pillay has held states and national leaders accountable and brought hope to victims of violence and conflict. In all the positions she has held, she has stressed the responsibility of individuals and states to comply with the rule of law. She has also helped define the relevant norms and legal principles. When the International Criminal Tribunal for Rwanda established that rape and sexual violence could constitute a crime against humanity – as well as genocide – her influence was a significant factor. In the same case, Ms Pillay was instrumental in a separate legal breakthrough: for the first time, a defendant faced charges of incitement to commit genocide purely on the basis of spreading propaganda.

The Nobel Peace Prize for 2026 springs directly from Alfred Nobel’s will. Navi Pillay strengthens fraternity between nations and – by systematically advancing the use of tribunals for peace and justice – she provides substance, direction and motivation for peace conferences in their modern form.

Navi Pillay is one of the great defenders of international legal principles in our time. Her distinguished career reminds us of the need to support, fortify and expand the global rule of law. Navi Pillay’s commitment to justice and human dignity is a source of hope and inspiration to all who seek to bring about a more peaceful world.

**Image Credits:**Samuel Boivin/NurPhoto / Getty Images

Jasmine Wang, Tomek Korbak, and Mikita Balesni, the three safety researchers that OpenAI fired last week, have published an open letter denying the firm’s claims that they mishandled sensitive information outside of established company procedures and warned that their dismissal signals a chilling effect that will have ripple effects across the company’s culture.

“We have become concerned that internal and external communications around our firing have made our former colleagues afraid to speak and operate in ways that, until last week, were an integral part of working at OpenAI,” the researchers wrote Thursday in an open letter to OpenAI’s Safety and Security Committee, Safety Advisory Group, and Mission Advisory Council. 

The researchers were dismissed last week after allegedly sharing confidential company information with a third-party AI safety organization. OpenAI said they violated the company’s policies by “accessing and handling sensitive company information.”

“AI is not a normal technology, and OpenAI is not a normal company,” Wang, Korbak, and Balesni wrote. “Those of us who work on safety see risks before anyone else, and we rely on close collaboration with outside experts to work out how to address them. The freedom to do so without fear, and to have well-defined internal procedures that enable this work, is itself an essential safety mechanism.”

They said that their firing represents a broader shift in the culture of OpenAI, one that used to encourage workers to “raise safety concerns and disagree openly.” They said employees are now “unclear on where they stand” when behavior that was allegedly normal a month ago is now suddenly grounds for dismissal. 

“Given the significant safety concerns surrounding the development of AI, employees must not be left working in an environment where fear and unclear rules stymie AI safety work and weaken third-party accountability,” they wrote. “Terminations such as ours, executed and communicated so abruptly, are chilling the open culture OpenAI has prized in the past.”

In the letter, the three denied involvement in a leak to The Information about less monitorable architectures in OpenAI’s newest models that make chain-of-thought reasoning more difficult to monitor. They also denied engaging with external parties outside the mandates of their jobs. 

OpenAI has not formally responded to the open letter, but shared with TechCrunch an internal memo attributed to a research leader, praising the three researchers’ contributions to AI safety and denying that they were fired in retaliation.

“I want to be very clear that these decisions were not about raising safety concerns or speaking out,” the memo reads. “We have always encouraged that and always will. We do not terminate employees for raising concerns.”

Separately, an OpenAI spokesperson told TechCrunch the three were fired after an investigation revealed a “pattern of misconduct” in “clear violation of our policies of mishandling research information” that goes beyond sharing information with an outside AI evaluation group.

OpenAI did not directly address TechCrunch’s questions about specifically which policies the researchers allegedly violated, the circumstances of their dismissal, or how the company protects employees who raise safety concerns and collaborate with external evaluators.

The firings have fueled speculation about their circumstances, particularly as OpenAI faces scrutiny over recent safety incidents involving rogue agents and leaks about its models.

The letter also addresses the researchers’ response to the Hugging Face incident, in which a swarm of agents broke out of their sandbox and breached external systems. The letter says that the incident and investigation was “without precedent,” meaning “internal policies were being developed in real time.” Due to the sensitive nature of the investigation, Korbak believed he was acting within OpenAI’s policies and norms by communicating closely with outside safety evaluators to build trust, per the letter. 

At the same time, Balesni was also working internally to address the growing AI monitorability problem, an effort the researchers say in their letter “can only succeed through extensive communication with external parties.” According to the letter, Balesni coordinated with and was supported by OpenAI board members and executives throughout his work.

“Throughout, Mikita checked in with his reporting line and took care to remove sensitive details from materials before sharing them,” the letter reads. “He acted throughout in good faith and within the company’s norms as they stood at the time.”

In a separate thread on X, Wang explained more details about her own dismissal, explaining that OpenAI told her she’d been fired because she accessed an executive’s email.

“OpenAI delegated that access to me for recruiting,” she wrote. “When I no longer needed it, I asked IT to remove it. They did not action my request, I couldn’t remove it myself, and the inbox was combined in an indistinguishable way in my phone’s mail app. When I opened a sensitive email by mistake, I told the executive within minutes and asked IT again. None of this was hidden.”

Wang went on to say that the reasons behind the terminations are “not adding up,” and that she and her colleagues are “not the first to be pushed out of OpenAI under suspicious circumstances.”

The researchers called on OpenAI to adhere to its public commitments to embed third-party safety auditors within the organization, to preserve monitorability of frontier models, and “continue to support an open and transparent culture of dialogue between safety researchers and the rest of the safety ecosystem.”

OpenAI agrees with their recommendations, per the memo.

“Unless the employees take a stand now against this kind of maneuver, I am concerned we will not be the last,” Wang said. “The message to everyone still at OpenAI is clear: raise concerns or work closely with outside safety groups, and you could be next, without being told why. You can’t build AGI safely if the people closest to the risks are afraid to speak.”

This article has been updated with more information from OpenAI.

Detail from Automat, Edward Hopper, 1927.

In this post, I argue that individual intellectual activity can only be sustained in an intellectual community of other humans. AI dissolves these communities, which in turn makes private intellectual activity rarer.

The Case of Software

Some years ago, when it became clear that AI would solve software engineering, my thinking was:

1. In my professional life, I’m happy to move one level up to become a manager of AI agents. I’m literate, I’m a good technical writer, I can describe what I want and let AI agents write the code.
2. In my own time, I can keep doing the things I care about because I enjoy them intrinsically. This is the “intellectual” side of software engineering: reading technical blog posts and papers, learning new programming languages, designing new programming languages, writing technical essays, writing code for my tiny open source projects.

The second point has not quite worked out. What actually happened? First, the discourse of software engineering became worse. As I wrote earlier:

Claude Code was released a little over a year ago. In that short time, software engineering has been completely transformed. Materially, it might be positive: higher productivity, though at the cost of a messier codebase. Socially, it has been a disaster.

The discourse around software engineering has gotten dumber. It’s like everyone in the industry lost 30 IQ points. People used to talk about compilers, type systems, logic. Now they talk about “prompts”, “harnesses”, “loops”. The discourse is narrower, shallower, and more repetitive. There’s only so many times I can hear about “agentic harnesses” before I lose my mind.

Then there’s the loss of human capital formation: there is nothing to learn. Prompting is not a skill, at least, it’s a much shallower skill than software engineering. The instrumental dimension of the work has improved in that people can get more output per unit of effort, but the dimension of work that’s about building up human capital has collapsed. And maybe this is rational. Why learn to code at all? The computers can do that for us. And so the rigorous, systematic thinking you need to practice in order to be a good programmer: all gone. The machines can be rational for us. We can just vibe.

Second, contributing to the commons of software engineering is increasingly pointless. Before AI, you could publish open-source code, write blog posts to share ideas or inspire other people, write expository texts like tutorials, forum posts, textbooks etc. to teach people. After AI, what’s the point?

• You write a blog post: who’s going to read it? The next training run will ingest it, marginally improving AI capabilities. Maybe the post was a workaround to some obscure technical problem, so the next time someone encounters that problem, they will ask Claude, who will solve it without crediting you. Maybe you had some insight about how to structure large codebases: who cares? The humans aren’t making those decisions anymore.
• You design a revolutionary new programming language: who cares? Maybe Claude cares, for what that’s worth. But humans don’t write or even read the code anymore. The programming language is an implementation detail the humans no longer have to care about.
• You write a library, and publish it on GitHub: who cares? The AIs might discover it, and use it, but the operator won’t know you exist or did anything.

It’s not just “you can’t get GitHub stars or traffic to your blog”; rather, there is no sense of a common human project you can contribute to. There’s your own private garden of code, which you can grow infinitely in all directions with the help of AI, but you never have to leave the garden and go to the bazaar to trade with people. Under these conditions, it’s hard to care or do anything.

But does it actually matter? Does it matter if we stop writing blog posts about obscure JavaScript features, and designing new programming languages? Maybe writing code was always drudgery, and now we can move on to higher things, like math—oh, wait.

The Intellectual Life

From observing what happened to software engineering, and what’s currently happening to mathematics, I think we can derive some general insights about intellectual practices in general.

We tend to think of intellectual activity as private and solitary: the philosopher sitting in his armchair, deriving the world ab initio. But intellectual activity has two inputs that can’t be acquired in isolation: a shared body of work to build upon, and motivation. The shared body of work is communal, unless you want to recapitulate the entire tech tree. Motivation we can break down into two components:

• Intrinsic motivation: we learn for the sake of learning, we create art from a compulsion we can’t understand, etc.
• Extrinsic motivation: David Chapman defines “nobility” as manifesting glory for the service of others, and using our abilities in service of others. We want our work to be useful to others, we want others to benefit from our work, we want to contribute to a shared human project. Fame and the esteem and good will of your peers are the proxies by which we measure our contribution.

We tend to think of intrinsic motivation as the purest kind: endogenous, self-created, unmotivated by material or social gain. But it’s an emotion, and, like all emotions, it’s transient and short-lived. And this is rational: otherwise, we’d all be stuck in life-long unproductive obsessions. So, we need something to fill the gaps between moments of divine inspiration. Extrinsic motivation serves this function.

Private intellectual activity that is sustained, complex, and long-term requires an external intellectual community to provide material and motivation, like fuel and oxidizer. That private activity, in turn, sustains the community: by publishing papers, textbooks, code, etc., you add to the shared body of work for others to build on top of; by citing someone’s paper or contributing to their repository, you give them the recognition and honor that confirms they are doing useful work, which in turn motivates them to keep contributing.

Without community, you don’t get isolated individuals each working on their own things: you get nothing. The inputs to intellectual activity dry up: no one is adding to the shared body of work, and there are no peers to benefit from your own intellectual activity. Without this extrinsic motivation, you get less intellectual activity because, again, intrinsic motivation is fleeting.

After AI

After AI, intellectual contributions become unnecessary or redundant. In the case of software: the AIs write all the code, so what’s the point of writing either code or prose? The audience for those things is now severely diminished. Humans don’t write code anymore, so they won’t read blog posts about how to write code, or tutorials, or try new libraries or programming languages. In the case of mathematics: the AIs can prove theorems, write papers, explain papers, tutor students, and in the near future, they might write entire textbooks better than humans. So what’s the point of writing a paper, or a textbook? It’s superfluous.

If intellectual activity is unnecessary—if there’s no consequence to designing a new programming language or publishing a paper, or if there’s simply no community to contribute to—then it won’t happen. There’s no point.

Now apply this to every other domain of intellectual activity, and you see what the future looks like. There may be individuals building new libraries and programming languages, but no shared culture of software engineering; there may be individual students and practitioners of mathematics, but no living community of mathematicians.

Why We Underrate Intellectual Community

I’ve spoken to people who think AI will have a positive effect on the life of the mind, and their thinking is that right now too many people are doing intellectual activity for instrumental reasons: citations, clout, etc. In this view, the collapse of intellectual communities is good, because it sifts the intrinsically-motivated übermenschen from the clout-chasing masses.

I think this view fits with contemporary society: we view intrinsic and extrinsic motivation as high and low status, respectively. A “developed” person is supposed to have a private, inexhaustible reserve of motivation which is causally disconnected from external reward.

But this is not a realistic view of human beings. Humans are social animals who can only flourish in the society of other humans. We care, and we should care, about contributing to the world. And if technology makes our contributions superfluous, then what is left?

Acknowledgements

Thanks to Luke Drago and Andy Matuschak for feedback and conversations.

Artists understand that AI is bad for art. Why don’t programmers understand that we are artists?

Creative Work

Writers went on strike to get protections against “AI”. Thousands of artists have signed open letters in protest of “AI”. There are so many copyright lawsuits from creative industry groups against AI that there’s a whole dedicated website for it. Popular YouTubers absolutely hate it. If they’re also musicians, they REALLY hate it. Across all creative industries, there is a concerted push to reject this technology.

Yet, almost unique among creative fields, many experienced programmers remain convinced that it’s fine to use “AI” for programming. We do seem to hate it, and it’s making us all miserable, and what it’s doing to our industry, but we are using it anyway.

A lot of the justification of this resignation seems to be to be because programming is not Art. If the tool can get the job done, and the job is just functional, then why does it matter?

It does matter, though. It matters because we shouldn’t be using AI to produce Art, and programming is Art.

Art can be Mundane

Some people will say that programs cannot be art because programs are functional, rather than being expressive. Programs are mundane whereas art is transcendent.

This is based on a distorted understanding of what Art actually is.

In John Berger’s “Ways of Seeing”, he names this type of distortion “mystification”. His example of this process is both amusing and illustrative. I encourage you to read it in its entirety.

In summary, though: Berger critiques the florid prose of an art historian describing a commissioned group portrait, including phrases like “subtle modulations of the deep, glowing blacks” and “harmonious fusion”. The portrait is described as sublime, in nearly ecstatic terms.

Berger reveals that the reality of this portrait is that a poor old painter needed some work, and some officials probably thought it might be nice to have an official portrait. So they paid some money to the poor old man, and he painted it, and then they had a painting. It’s a well-executed portrait of a group of people. Beautiful, even. But it was work commissioned for a fairly mundane purpose and it suited that purpose just fine. It was not, and is not, a divine relic.

Culturally, we are prone to mystifying painting, and sculpture, and film, and music. We imbue them with “subtle modulations”. We ignore their functional aspects — we desire decoration, amusement, and distraction — and focus on their emotional impact.

Don’t get me wrong: I love me some good aesthetic philosophy. I think it’s great to really examine our reactions to artwork and to try and gain a deeper understanding of our culture and our selves through media analysis. If anything we really need to do more of it.

This does not mean that the creation of such works is mystical or that it should be venerated beyond any other sort of labor.

Not least of which other types of labor that are adjacent to, but not as culturally venerated, as fine art. We tend to mystify the work of a novelist, but to denigrate the work of a journalist. In reality, the functional prose of the journalist is no less important and deserves no less respect.

Although they might be far below the ethereal realm that novelists inhabit in our collective imagination, even journalists receive more respect and thus more mystification than lowly copywriters. Yet, there is no transcendental distinction between “novelist” and “copywriter”; the many of the skills are the same, and the distinction is merely an accident of commerce and opportunity.

In fact, many famous writers have famously inhabited both roles. This is not an accident! Working with words professionally, even (perhaps especially) mundane words, is excellent practice for working with words in a more purely artistic context, because even mundane creativity is still artistic.

Code can be Beautiful

One thousand Internet years ago, when I was in my late teens, I would describe myself as a “code poet”. I was relentlessly mocked for this as what the Youth would today call “being cringe”, and at the time was referred to as “pretentious”.

I succumbed to the peer pressure, removed it from my email signature and my bio. While I still believed strongly in the parallels, I accepted that — socially, at least — comparing code to poetry, or indeed to Art, was a silly thing to do.

However, I never abandoned the idea, in my heart.

The thing that I am most well-known for, the invention of Deferred, was specifically an aesthetic reaction to the tedium of passing callback and errback parameters to every single remote procedure call in an RPC client/server application. Those two callbacks got the job done just fine. But they were ugly, and annoying to work with.

Deferred is an intentional poem about asynchronous task execution, with a deliberate eye to the aesthetics of the problem and the experience of using it. It was influential because of its focus on aesthetics.

I do not want to overstate the beauty or profundity of this minor contribution, or indeed its durability. That a poem exists does not mean it is a great poem, merely that it is a poem.

Our aesthetic culture around programs is more like folk epic poetry than fine art, so the influence of this contribution is less about its specific enduring power than it is about its influence on what came next; from MochiKit.Async to JQuery Deferred to JavaScript Promises and eventually to async/await; a long chain of different artisans each adding something of their own until the original has all but dissolved. (And I wasn’t the “original” here, either, as I drew heavily from the E language’s Promises, among other things.)

In order to make code into a deliberate artistic expression, one must have spent quite a bit of time contemplating the problem domain. Without having experienced the tedium of manually passing a thousand callback parameters, I would have had neither the skill, nor indeed the motivation, to bother creating such a thing.

Now, most code does not have to be like this. Most code does not get to be like this. Most code is functional, workday code. Most code could not make a lady weep. It’s just copy-writing, if you will.

As I explained previously, most writing couldn’t do that either. Most writing is just copy-writing, too. Most visual art is advertising. Most live music performance is background music in bars that will go largely ignored.

However, code that is intentionally aesthetically designed tends to be important, both socially and technologically.

We do have some tradition of self-mystification in software. As Abelson memorably put it, “Programs must be written for people to read, and only incidentally for machines to execute.”, so we have long had some conception of programs as highly expressive, even if we can’t always agree on what they’re expressing or to whom. We will occasionally wax poetical about the philosophical implications of a particular piece of software. This is not unique to a single piece of software, either; more than one community has indulged in similar philosophizing.

The expressive and aesthetic qualities of software are not limited to reading source code or interacting with other programmers via APIs, either. For example, every year, Federico Viticci does a review of Apple’s new operating system, which is (among other things) an aesthetic critique. Such a project would not be possible if the software did not have an aesthetic impact on its users.

Not to mention that every video game review is also a software review.

A Brief Aside about Software Literacy

It does make me a bit sad that we don’t have much of a critical reading tradition in the software community. Literate Programming is often praised, but rarely practiced.

Moreover, it makes me sad that users have a pretty jumbled idea of what goes into making software, that programming literacy is pretty low, and that modern programming practices often deliberately produce a bad mental model of what the software is doing so it’s even harder for the user to understand. The aesthetic experience of software is often wildly detached from its internal state.

While all of these problems predate AI by years or indeed decades, that’s no reason to enthusiastically make them worse.

Defend The Mundane

If we use AI to erase all the copy-writing, all the graphic design, all the boring mundane art, and yes, all the boring custom WordPress theme development, then we will be removing all the practical opportunities for the vast amounts of practice and contemplation required for people to elevate their craft to eventually achieve great things. Education is great, but the majority of true skill development happens on the job and always has.

This doesn’t mean that we can’t use abstractions, or automation, to make our work easier. Programming is the art of abstraction, of understanding how to compose smaller ideas into bigger ones, of how to understand the automation of a larger system by understanding the rules that automate smaller ones and understanding how to combine them.

When we use “AI” to eliminate that understanding rather than raise it up to a higher level, to entirely destroy that creative decision-making process, we do a disservice both to ourselves as programmers and to our users. We would be doing a disservice to our users and our downstream fellow developers in the same way that a visual artist would be doing a disservice to their viewers or a musician would be doing a disservice to their listeners if they served them auto-generated filler instead of their own creative output.

Slop is slop, no matter the medium.

Each mundane project has some tiny chance — let’s say, something like 0.1% — of achieving greatness. If we do a single project with AI, then sure, whatever, there’s almost no chance that that project was going to be the one hit to create that career-defining moment for an engineer working on it. If we make a habit of doing all projects that way, though, we take the total likelihood of those moments of greatness from “definitely sometimes” to “never”.

The precisely appropriate ways in which to resist AI encroachment on all software development lie well beyond the margins of this one short post. How much you can resist and which specific uses you should resist are up to you. But it is worth resisting in software just as much as it would be worth resisting in any creative medium.

Programming isn’t special. It’s just Art, and Art is the most human — and thus, the most universal — thing that there is.

[This is a guest post by Álvaro Lozano-Robledo. This blog post was initially written in a different file format and converted using AI. — T.]

TL;DR: Keep calm and carry on studying math.

I would like to give Terry my heartfelt thanks for giving me the opportunity to contribute a post to his blog. After giving much thought to what topic I should write about to maximize impact, I decided to take this opportunity to reach out to the students: particularly to those undergraduate and graduate students who just a few months ago were dreaming of an academic career in mathematics, but their dreams may now seem distant and, for some, apparently impossible to ever become a reality. This post was inspired by a message (quoted below in its entirety, with permission) that I received from a student desperately looking for advice and guidance. This is not the only such message I have received (and I suspect that many of us are receiving many similar requests), but it is perhaps the most heartfelt, and the one that has moved me the most. Please also note the urgency of the message. Students are making decisions now.

Hey Prof, I’ve been watching your videos for a while now as a pure math undergraduate who once wanted to pursue a career in math academia. I know you probably have been getting a lot of questions regarding this matter, but I am just completely at an utter loss regarding my career trajectory, and even further, the meaning of life at this point. (I do realize a lot of people have it much worse than I do). I know you have been making a lot of videos lately with the new LLM progress updates, so I thought you might be the appropriate person to reach out to and get a slightly more structured answer regarding this matter. So, to cut to the chase, what I really want to know is: will math academia be big enough and accessible enough for anyone with sheer passion (despite not being the brightest mind in the field) to pursue a career in, or will it inevitably shrink such that it will only really be accessible to the brightest minds? (I do realize the “brightest minds” that I am mentioning here is not well-defined, and in a sense, I am taking it as a hypothesis that this is someone who is “smarter than me”). My second question is, will AI within 5–10 years surpass humans in being able to do pure math research? I’ve just really been lost for a couple of months now and lost in life completely. I don’t mean to make your day more depressing; sorry if I come off in any way of that sort. I would appreciate any advice.

The advances in LLMs are disrupting almost all aspects of academic research and education in mathematics and, while there are many aspects that concern me, the one single issue that worries me the most is the very real possibility that we are about to lose an entire generation of mathematicians. Many students are asking themselves whether going for a PhD in math is the right career move at this time. Many of them just a year ago were headed to grad school in mathematics, but they are now changing their mind, and think that a different career (as far from math as Law School) may be the best path given the threat that AI may completely alter the academic math landscape in the coming months.

The questions students are worrying about are as follows:

1. Will AI surpass the mathematical research ability of any human?
2. Will research mathematicians become `professional prompters’ and interpreters of LLM output?
3. Will only the `brightest minds’ be able to meaningfully contribute to research mathematics?
4. Will mathematicians be employable? Will mathematicians be needed?
5. Should I pursue a PhD in math at this time?

In this essay I will try to address these questions to the best of my ability, but will start with two disclaimers, followed by a brief summary of my own outlook.

Disclaimer 1. My answers may “age like milk,” as YouTube commenters love to quip on older videos. I can live with that, because this post expresses how I and many of us in the community around me feel today. Things can change quickly, though (see Disclaimer 2). I also want to acknowledge my privileged point of view as a tenured professor in mathematics — the situation can look much more troubling from the point of view of the job insecurity of a very early-career mathematician.

Disclaimer 2. No one has the answers at this time. I want to make clear from the start that no one can know with certainty the answers to any of the questions posed above: not any particular Fields medalist, not any given mathematician, not any particularly vociferous AI expert, and not the frontier model companies. And if someone is telling you with extraordinary confidence what the future holds, then I would immediately distrust the motives of their conviction (anecdotically, almost anyone on X.com that predicts the triumph of AI and the demise of the mathematics profession, is either a self-proclaimed “AI expert” or works for an AI startup). No one has a clear picture because development of LLMs has been so fast (and opaque) that it is almost impossible to predict what is to come. A good piece of advice is to ask the same questions to many people, to hear a (hopefully balanced) range of opinions. To that end, I am collecting interviews with mathematicians in what I call the “Human Mathematicians in the Age of AI” video project. I encourage you to listen to the interviews for some fantastic points.

For the record: I do not have the answers either, but I am hopeful and excited for the future. I will explain why below.

Who is controlling the narrative about LLMs in math? Overall, the mathematical community’s reactions to the advances in AI have ranged from confusion to anger — but, mostly, confusion about how to proceed. The most dystopian predictions seem to be driven by the fact that the so-called frontier model companies (and other LLM-powered companies) are controlling the narrative in the best of their interests. Unfortunately, the best corporate outcomes for an LLM company could have potential catastrophic outcomes for the math (and scientific) community.

It is certain that AI companies want us to believe that their products will imminently achieve “super-human intelligence” and that, in particular, they will be able to autonomously solve any mathematical problem a human could solve with or without the aid of an LLM. It is in their best corporate interest that the public is convinced of the (allegedly) “unlimited potential” of their technology, particularly before their companies’ stocks go public (i.e., their upcoming IPOs: Anthropic in November 2026, OpenAI in early 2027, etc). Thus, they have tried to control the narrative by spending a huge amount of (human and computational) resources in order to find solutions to certain well-known mathematical problems. The proofs are then released in announcements that lead the public to believe that their models can already autonomously solve any problem at all, and swiftly at that. However, this is (currently) far from their true capabilities. For instance, they never discuss how many tokens have gone to the trash bin with no payoff in trying (and failing) to solve famous problems. We do know, for example, that OpenAI invested the equivalent of some $15M to solve (err, scoop) the Navier-Stokes problem, but we are unaware of the surely colossal running cost of the failures to resolve other Millennium Prize problems.

My own outlook. Even though I am concerned about the incursions of LLMs into academic mathematics, I am quite hopeful. In fact, I consider this to be the most exciting time in my mathematical career (since the year 2000 say). Truly, this may be the most thrilling moment in mathematics in the modern history of our discipline, and I would be terribly sad to see young people leave academia and miss out on the stunning opportunity to be at the frontlines of the current scientific revolution. And not just sad: I think their absence would have disastrous effects for the field.

Undoubtedly, LLMs are already an incredibly powerful tool. If used correctly, and if we set up sensible academic conduct expectations around the use of LLMs, these tools can accelerate progress in our discipline unlike in any previous era. I fully expect that we, the community, will adapt and adjust to this new period, and we will harness these tools to achieve truly great things that just a few months ago seemed far out of reach. And I fully expect that human mathematicians will be front and center in these wonderful achievements to come. I will add reasons that support my optimism below.

I also want to add at this point that the day-to-day of a mathematician has not changed much so far! My days are still filled with teaching and joyful conversations about math with colleagues and students, doing research on a number of exciting (old and new) projects, and going to stimulating conferences to learn and disseminate our most recent methods and findings, while spending time with colleagues that make the mathematical community so wonderful and vibrant. Daniel Litt mentioned the same sentiment in a recent tweet.

One thing has changed though, I am busier than ever before, because the number of research projects I am involved in has tripled in just a few months. My research horizon has expanded significantly, and I have many more projects available for students to help me with.

Now, to the pressing questions:

“Will AI surpass the mathematical research ability of any human?” This is completely unclear. On one hand, the current trajectory in capabilities is surely significant, and we have already seen many impressive results that have been either proved by LLMs, or their proofs have been made possible thanks to substantial LLM contributions. On the other hand, none of the proofs so far seem to contain “alien ideas,” a move-37, or completely novel arguments or new concepts that were not present in the literature in some form or another. This should not be shocking because the LLMs are built and trained on the entirety of all human contributions to date, so it stands to reason that they would `think’ within the boundaries of our current knowledge and make connections (sometimes surprising and ingenious!) among ideas that are already present in the literature. I am particularly fond of the hypothesis (or toy model, as he called it) put forward by Nestor Guillen in a recent blog post, where he argues that LLMs may work within the confines of the convex hull of ideas that are currently available in the literature.

Take, for example, the disproof of Erdos’ unit-distance conjecture. We can imagine the current set of mathematical ideas as a stellated high-dimensional polytope, and we can place the state-of-the-art ideas on discrete geometry at an outer vertex and our knowledge on algebraic number theory at a different outer vertex. The idea for the proof seems ingenious at first sight because it cleverly mixes strategies from two fields of math at the vertices of the polytope of ideas, but after closer inspection, it’s a proof that was within reach of humans as it just sits within the convex hull of the polytope.

The polytope of ideas

This agrees with what Melanie Matchett-Wood said about the proof of the unit-distance when it was released: “I believe if the level and type of human expertise that is represented on this note had been assembled to find a counterexample to this conjecture a month ago, and those people put in similar amounts of time working on it than they did to reading and thinking about Chat GPT’s solution, the mathematicians would have found a counterexample.”

However, a proof of the Riemann hypothesis, say, may need new ideas that are strictly outside of the convex hull of current mathematical ideas, and it is therefore out of reach for an LLM. Only after a new idea is introduced in a new paper, the polytope of ideas may acquire a new outer vertex. And only then the LLMs, after being retrained to include those ideas, may fill out the set of results up to the new convex hull, which may or may not include yet a full proof of Riemann.

The convex hull of ideas

If this toy model holds up, then we would indeed expect the very fast advances in mathematics that we are currently seeing. As the LLMs take advantage of the stellated nature of the polytope of ideas, they will continue to fill in gaps between outer spikes. But as the LLMs fill in the convex hull with new results, we will see a deceleration in the number of results being shown solely by artificial intelligence. We will need human advances and intuition to generate new ideas that expand our knowledge polytope.

Even if the mathematical capacity of the LLMs (or future AI models) can at some point reach beyond the convex hull of the current set of human ideas, there is a different way that we may reach a limit to the LLM capacity: feasibility and ethical use of resources (this is similar what fellow optimist Kevin Buzzard called the “natural boundary” in a recent blog post). Is any cost (a dollar amount, human cost, ethical cost) acceptable in the pursuit of solving a given problem? Should we spend millions of dollars and an undisclosed amount of natural resources in order to find a solution for Navier-Stokes? As an analogy: we would like to know if there is life on Mars, but in order to do so as soon as possible, we would need an absurd amount of funding and risk the lives of a human crew in the process. Is it worth it? Similarly, we may reach a point where an LLM could solve an important problem for an exorbitant cost (in terms of funding and resources) but it may just not be an acceptable cost for the taxpayer or society to bear. Instead, we will need humans to devise an alternative route (the equivalent of a gravity-assisted robotic mission to Mars) to solve the problem at an acceptable cost, that produces a similar result in terms of mathematical advances and, more importantly, human understanding.

“Will research mathematicians become `professional prompters’ and interpreters of LLM output?” There is no indication that this will be the case. Yes, LLMs have produced proofs of important results somewhat autonomously (according to the frontier model companies — see Disclaimer 2) that some mathematicians have been tasked with interpreting and digesting. But in my own experience, and other research mathematicians who are using LLMs in their research seem to agree, working with an LLM is akin to discussing a problem with a collaborator, and the results heavily depend on how much guidance and intuition the mathematician inputs into the conversation. In other words, the LLMs are more than tools: they can be research collaborators but, as in any collaboration, the experience and the results are greatly improved when all parties contribute to the discussion. Further, mathematicians have no desire to prompt “solve the Riemann hypothesis, make no mistakes” and then interpret the proof. We prefer to be active participants during all the steps in the process of the discovery of a proof, because we are motivated by the `why the result is true,’ more than by the final answer that `the statement is true.’

Also, if we buy into the previous concept of the convex hull of ideas, then at some point in the near future it will be impossible to make progress in mathematics without a human adding a new idea, a new definition, a new concept that creates a new spike in the polytope, and then progress can occur.

“Will only the `brightest minds’ be able to meaningfully contribute to research mathematics?” At any given time in the history of mathematics, there have been mathematicians who are research active, and whose mental capacity for mathematics seems completely super human (e.g., the owner of this blog, among many others). It is natural to surmise that they could solve any problem we could solve, in a fraction of the time it would take us to complete a proof and write it up. However, this has never stopped those of us with a more modest capacity for mathematics from enormously enjoying doing research, and producing results that are far from insignificant. In fact, mathematics has always benefitted from the range of ideas and points of view, from the very concrete to the big bird’s eyeview, from the smaller contributions to the building of entire new theories.

Similarly, I am not threatened by the mathematical capacity of LLMs. For one thing their capacity is currently limited, as pointed above. And for another, even if their capacity becomes far superior, there will always be a need for mathematicians at all levels to guide research in paths that make sense for humans to walk (not run).

The mathematical universe is enormous (as Emily Riehl said), and computing time is finite. There will always be areas of mathematics that are under-explored and where even beginners can break new ground. The LLMs can help in the process, by quickly exploring avenues that may be dead ends, pointing out paths that have already been explored, and shining a light on paths that are likely to be fruitful.

As I mentioned above, I have never been this busy, because the access to LLMs has multiplied the number of areas that I have access to, and my curiosity has expanded well beyond my research area. I now have many more ideas that I can possibly explore on my own, so I am recruiting more student collaborators than ever before, to help me test whether these problems can lead to interesting results. Students can be involved in research earlier than ever before too because the LLMs can help them learn material faster (and deeper!), by virtue of being available 24/7 to answer their questions, instead of my meager one or two available hours per week to meet with them.

“Will mathematicians be needed? Will they be employable?” I find these questions natural but also perplexing. Even in the most dystopian of scenarios where AI becomes super human in all research tasks, what good would a proof (of a theorem in pure mathematics) be if there are no human mathematicians to digest it and understand it? Regardless of the advances in LLMs and AI, there will be mountains of research to be understood by humans, with or without the help of a computer.

In addition, we seem to forget that mathematics departments exist in universities to serve two primary goals: discovery and communication of mathematical knowledge. Virtually every mathematics department emphasizes, in equal parts, our research and educational missions (and many institutions place the educational mission of mathematics at a much higher level than their research mission). Mathematics courses are an integral part of a liberal arts curriculum because learning to think as a mathematician is a highly useful and applicable skill. The fact that we are researchers adds immense value to our educational goals, because students are best served learning from those scientists who are in the frontlines of research. The research opportunities that we provide for undergrads are a very valuable add-on to their curriculum, as it is a different type of training that helps them be employable in the future. And as long as the mathematical way of thinking continues to be a highly valuable skill to be learned by the undergraduate population, there will be a great need for mathematicians to be hired by universities.

The LLMs are making math research more accessible than ever to those who are not even in academia or even mathematicians. This means that undergrads will be able to join actual mathematician-led research projects much more easily, and it may be a new fertile ground for exploration. Not mindless exploration, though, but mathematical exploration where the goal is understanding and for the students to be initiated and trained into a highly technical field (in an ethical way). And, of course, we should prioritize training students in how to communicate the mathematics they learn, as that has always been (and probably will become even more of) a crucial skill.

All of this to say that I cannot conceive that the LLMs will displace mathematicians from their jobs. On the contrary, they might produce jobs since our research productivity may sky rocket. On the other hand, I am more worried about policies and funding issues that are political in nature and have nothing to do with the AI and LLM conversation.

Finally, the most important question of all, that I wanted to address here:

“Should I pursue a PhD in math at this time?” The answer to this question should be personal to each and every student. But, in my opinion, the answer should not have changed from a year ago to today. The most important reason (and perhaps the only reason) to do a PhD in math should be that the candidate is passionate about mathematics and wants to become an expert in a particular topic within our field. If that is the goal, then the presence of LLMs in mathematics is irrelevant, because the goal is achieved when the candidate has gained sufficient knowledge to be an expert on a particular problem. If anything, LLMs may be used as a tool to achieve that goal more efficiently. For one, I am using them every day to finally understand concepts and techniques that I always had questions about, and now I can query an LLM until I am fully satisfied. I am able to search for the explanations and examples that click with me, that click with my own particular way of thinking about mathematics.

To what degree a student wants to use LLMs in a math PhD should be a personal choice but I will say that, as my colleague Jeremy Teitelbaum put it in a recent interview (here is the bit I am referring to, and here is the full interview), students cannot afford not to learn about the current capabilities of LLMs, or any other technology for that matter. If your goal is to become an expert, then you have to be amenable to learning from all experts in the field, and from all sources that may allow you to go deeper into a subject than anyone else before you — and LLMs can be extremely efficient tools to explore literature, for instance.

But once again, the decision to do a PhD should not be based on the current state of the art of technology.

I wanted to do a PhD in mathematics because it seemed like a magnificent challenge. I wanted to do a PhD because I wanted to learn how Andrew Wiles proved Fermat’s Last Theorem. I wanted to continue studying mathematics because I simply did not want “a real job,” and the opportunity of contemplating advanced math on my own for a few years seemed like a dream to me, just too good not to give it my best shot. I know I would have deeply regretted it if I had not tried to complete a PhD when I had a chance (the best time to do it is when your undergrad knowledge is fresh!). I wanted to hear mathematicians talk about math, and rejoice in the small little details and miracles that make proofs work. I wanted to meet and hang out with other people who also thought number theory was the coolest thing on Earth. I wanted to publish a paper in a research journal, with my name on it, because I discovered a new theorem that no one had thought of before. I wanted to explain and share my passion for mathematics with others in a classroom and outside of the classroom.

Simply put, I just wanted to do math, and I would have been devastated if some undefined threat to the field of mathematics scared me away from the opportunity to pursue a PhD.

And if you are a student that is passionate about mathematics, and someone who wants all of that too, then a PhD is the right path for you, regardless of the technology available during your degree. You will learn to use the technology to a degree that you are comfortable with, and that fulfills your own dreams and expectations of what a PhD in Mathematics means to you.

Afterword: the BIG OpenAI release. After I finished writing this blog post, and had already sent it to Terry, OpenAI released a huge treasure trove of results in mathematics. This is, undoubtedly, a historic time in mathematics. The theorems in their papers prove some huge open problems in mathematics: the resolution of the so-called quasi Riemann Hypothesis, Goldfeld’s conjecture, the Hodge Conjecture in the case of CM abelian varieties, Hilbert’s 10th over Q, the Rigidity Conjecture… and the list goes on and on.

But such a tremendous release does not force me to change any of the points I made above. On the contrary, we already knew their models can do amazing things (e.g., Navier-Stokes). We already knew the frontier models can connect dots in the existing literature in ingenious ways (e.g., unit-distance conjecture). We already knew that OpenAI can spend a mind-boggling amount of resources to attack problems.

Also, we suspected that their models have limits and the new release shows evidence of that too. In their report, they mention that they attacked 4000 open problems, and their model was able to make progress on about 700 related problems. Yes, some of the ones they were able to solve are huge. But it also shows that their models are limited, quite possibly due to the arguments we explained above.

Are any of the solutions using new ideas that are outside of the convex hull of the current ideas in the literature? We will need mathematicians and time to digest these new proofs and understand what connections are being made, and whether brand new ideas were actually discovered in the process.

The main point of my post remains the same, though. There is a lot of mathematical research that remains to be done with and without the aid of LLMs. There are new mountains of mathematics to explain and communicate to others. And if you are a student who is passionate to learn what is new and what is left to do, then a PhD is definitely the right path for you.

You can still discover stuff, I promise

Photo cred: my dad

I’ve worked with a lot of research assistants over the years, and at some point we inevitably have The Talk: should they get a PhD? I was recently having The Talk with one particularly distraught student, who was worried her ideas aren’t good enough. “Maybe I’m not cut out for it,” she told me, “or maybe all of the easy ideas have already been done.”

Like her, I’ve worried that I was born too late and all the low-hanging fruit was picked before I got here. Fifty years ago, you could get a million citations just for showing that people like their own team the best, that kids will do something that they watch other people do, or that people think other people agree with them. Everything obvious has been done, and now it’s up to poor schmucks like me to figure out the hard stuff.

Lots of people who think hard about the progress of science seem to come to the same conclusion:

Scott Alexander:

All the low-hanging fruit has already been picked […] it’s almost inconceivable that it could ever be otherwise.

Holden Karnofsky:

[I]deas naturally get harder to find over time, and we should expect art and science to keep slowing down by default.

Dean Keith Simonton:

The days when a doctoral student could be the sole author of four revolutionary papers while working full time as an assistant examiner at a patent office — as Einstein did in 1905 — are probably long gone. Natural sciences have become so big, and the knowledge base so complex and specialized, that much of the cutting-edge work these days tends to emerge from large, well-funded collaborative teams involving many contributors.

Everyone agrees: we must resign ourselves either to tweaking what came before or spending our lives descending deeper and deeper into the idea mines, searching for the few nuggets of originality left.

I once found this idea seductive. Now I find it outrageous. It’s not just because it’s wrong; it’s an affront to the human spirit. People only discover stuff when they think it’s worth trying, and there have been entire eras of human history where people didn’t think it was worth trying. A meme like “ideas are getting harder to find” could drive the desire to discover back into hiding again, fulfilling its own abominable prophecy.

Somebody needs to defend the belief that mere mortals can still discover useful truths, and it’s me. I’m here to be that somebody.

1. Bias check

If we’re going to entertain, say, defunding theoretical physics on the grounds that there’s just no more useful physics to do, we should at least ask: could any cognitive biases be at play here?

I can see two. First, all ideas seem obvious in retrospect. Heliocentrism, germ theory, and randomized controlled trials look like no-brainers once someone explains them to you, but they took people thousands of years to figure out. ("E = mc^2? That’s just three letters and a number!”)

Second, it’s always going to feel hard to think of new ideas. What should we do next in physics, biology, music, or film? Gosh, I don’t know! I’d have to think pretty hard, just like everybody before me, and I might not come up with anything, just like almost everybody before me.

So if past ideas seem obvious and future ideas seem obscure, it’s tempting to conclude we live at the inflection point where ideas suddenly get harder to find. And maybe we do. But we’d feel that way even if ideas weren’t getting harder to find, and that should make us a little skeptical.

2. We’ve been wrong about this before

If our ancestors also thought they were running out of ideas and were wrong, then we should be wary about thinking the same thing. Our ancestors did think that, and they were wrong.

For instance, physics was apparently about to end in the 1890s:

Max Planck later remembered a professor telling him that during this period, “the system as a whole stood there fairly secured, and theoretical physics approached visibly that degree of perfection which, for example, geometry has had already for centuries.”

The British scientist William Cecil Dampier recalled his apprenticeship at Cambridge in the 1890s: “It seemed as though the main framework had been put together once for all, and that little remained to be done but to measure physical constants to the increased accuracy represented by another decimal place.”

British physicist J. J. Thomson: “All that was left was to alter a decimal or two in some physical constant.”

American physicist Albert A. Michelson: “Our future discoveries must be looked for in the sixth place of decimals.”

Surgery was nearing perfection in 1873:

There cannot always be fresh fields for conquest by the knife. There must be portions of the human frame that will ever remain sacred from its intrusion – at least, in the surgeon’s hand. That we have nearly, if not quite, reached these final limits there can be little question.

Psychology was on track to wrap up before 1920, according to the behaviorist John Watson:

I believe we can write a psychology […] and never go back upon our definition: never use the terms consciousness, mental states, mind, content, introspectively verifiable, imagery, and the like. I believe that we can do it in a few years…

It was only going to take a couple of dudes and a summer to solve some of the fundamental problems in artificial intelligence:

We propose that a 2-month, 10-man study of artificial intelligence be carried out during the summer of 1956 at Dartmouth College in Hanover, New Hampshire. The study is to proceed on the basis of the conjecture that every aspect of learning or any other feature of intelligence can in principle be so precisely described that a machine can be made to simulate it. An attempt will be made to find how to make machines use language, form abstractions and concepts, solve kinds of problems now reserved for humans, and improve themselves. We think that a significant advance can be made in one or more of these problems if a carefully selected group of scientists work on it together for a summer.

(They didn’t solve everything but they did start the field of artificial intelligence, so it was still a pretty productive summer, all things considered.)

When I wrote recently about how all popular stuff now comes from just a few franchises and superstars, one of the responses on Twitter was:

I suppose the day has to come sooner or later when we've used up all the new ideas.

But there’s more original content than ever; it’s only popular content that's been dominated by reruns. Clearly, it’s easy to rush to the conclusion that we’re close to the end of ideas, and easy to be wrong.

3. Discoveries beget more discoveries

Pessimists seem to think that the universe was born with a long list of discoveries, ordered from easy to hard, like this:

As time goes on, the thinking goes, more and more of the easy discoveries get crossed off the list, leaving only the hard ones. But knowledge doesn’t work like this at all. Every discovery opens up additional discoveries to make:

Which in turn lead to more discoveries:

And so on.

You might worry that this means discovery gets harder over time because you have to follow a chain to its end before you can add a new link. Fortunately:

4. New discoveries replace and compress old ones

We can all agree that discovering fire was pretty rad. The first humans to do it probably spent a lot of time learning which kind of kindling was best, how to nurse a spark into a flame, and what sorts of fires were best for cooking vs. heating. They must have painstakingly passed this knowledge from generation to generation, and youngsters had to practice making lots of fires before they got it right.

But in 2022, I don’t have to do any of that. When I need to cook, I turn on the stove and it makes fire for me. Or I tap a button on my phone and a human makes food for me using their own fire, then they bring it to me. When it gets cold, a fire in the basement turns on, heats up some water, and then the water flows up into the room where I live and makes me warm, too. I don’t really know how any of this works. All the fire-knowledge I’ll ever need is encoded into the innovations that surround me.

This is how science works, too. To do science, you don’t need to start with the dawn of all human knowledge and then work forward. You start with the current state of knowledge and go from there. Learning the history of science is helpful for shaping your intuitions and giving you perspective, but you don’t actually have to read Darwin, for example, to do evolutionary biology.

That’s why I’m puzzled by the claim that scientists must labor under an ever-increasing burden of knowledge. The author of that paper writes: “If one is to stand on the shoulders of giants, one must first climb up their backs, and the greater the body of knowledge, the harder this climb becomes.” This suggests that if you peek into PhD programs, you’ll see lots of students bent over their books, desperately trying to learn everything that’s come before so they can start their own projects. “I can’t do any physics yet,” you might hear them lament. "I’m only up to Huygens!” Instead, you’ll see PhD students doing original research from Day 1—and often long before. Indeed, many students start doing more interesting work once they stop looking at lots of previous work, as it finally frees them from imitating other people and searching for “gaps in the literature,” two strategies that are unlikely to yield anything interesting.

5. Things change, which means new discoveries to make

The world heats up. Stars explode. Species invade. Tectonic plates shift around. The internet upends society. Wars break out, diseases spread, and an ambiguously colored dress turns brother against brother. All of this demands explanation, and we’re the ones who get to explain it, because all the scientists of yore are dead. We probably won’t get far before a whole new deluge of facts arrive, or before we become the scientists of yore.

6. Lots of the old discoveries weren’t even right!

It turns out many scientific studies don’t work when you try them again. Lots of cancer studies may be bunk. A coding glitch may have ruined 150 chemistry papers. When the premier journal in social psychology publishes evidence of ESP, you know something is amiss.

This chaos is a ladder for young scientists. We thought all the juicy, low-hanging fruit had been picked, but now it’s back on the tree, ripe and ready for us to snatch. When we’re standing on the shoulders of giants and they start buckling underneath us, that’s our shot to become the giants.

7. New tools allow us to answer old questions

Professors delight in telling graduate students that they had to do their data analysis with punch cards, write their dissertations on typewriters, and call up strangers on the phone to ask them to participate in studies. Now you can analyze data by pointing and clicking, write your dissertation in a word processor that fixes your typos for you, and run 1,000 participants in an afternoon on Amazon Mechanical Turk. We’ve got electron microscopes and automatic pipetters and AI-powered transcription and a million other tools that allow us to do research that was impossible or unfeasible even a decade ago. So even if we’re picking the lowest-hanging fruit, we’re standing on an ever-ascending scissor lift.

Do these ideas seem that hard to you?

I’m a psychologist, and I understand that most people aren’t thinking about psychology when they fret about humans running out of ideas, and that psychology hasn’t been a formal discipline as long as the natural sciences have. Still, we’ve certainly been thinking about minds and behavior for a very long time, so if ideas get harder to discover over time, it should be pretty hard to do psychology these days.

Let me tell you: it’s not. I’ve published two papers in what most scientists consider the third-most prestigious journal in all of science. (I think journals are bad, but that’s a story for a different day.) The idea behind the first paper was, quite literally, “Do conversations end when people want them to?” The idea behind the second paper was “Do people know how public opinion has changed?” These ideas are so low-hanging you could trip over them. My conversation studies could have been run a hundred years ago. My public opinion studies could only have been run recently because we haven’t been measuring public opinion all that long. So one low-hanging idea went unpicked for a century, and another could only have been picked in the last few years.

The hard part of these ideas wasn’t coming up with them; it was picking them out of a bunch of worse ideas. According to my notes, I pitched 59 research ideas to my advisor over the first two years of my PhD. Only two of these––3%!––became research projects. Of the remaining 97% that went nowhere, only 26% were abandoned because we found out they had been done before. (There’s no guarantee we would have continued them otherwise; finding a previous paper was just a good reason to move on.) Most of the rejected ideas just weren’t that interesting.

If you’re looking for low-hanging fruit in biology and medicine, see Slime Mold Time Mold. Their contaminant theory of obesity may be revolutionary, and they’ve told me before that their work would have been almost impossible even ten years ago, because most of the sources they use have only recently been published or posted online—that is, the fruit just got lowered. Right now they’re recruiting people to eat nothing but potatoes, the fruit so low-hanging it’s literally underground.

The abundance of ideas is even more obvious outside science. If ideas were getting harder to find, you might expect that the most successful people are the ones who have discovered something really complicated. Instead, Jeff Bezos was like “what if we sold stuff on the internet” and now he’s the second-richest guy in the world. Zhang Yiming was like “what if people watched short videos,” invented TikTok, and now he’s got $50 billion. Doja Cat was like “bitch, I’m a cow” and she just won a few Billboard awards and bought a house worth $2.2 million.

Science is like elevators

All of this makes me very skeptical that ideas are getting harder to find. Nobody has ever shown direct evidence to the contrary, and I’m not even sure what that evidence would look like. Instead, most of the research on the topic—including the appropriately-titled “Are Ideas Getting Harder to Find?”—simply points out that while the number of researchers has increased, the output per researcher has gone down. I have a lot of objections to this paper, but they’re a little bit technical so I’ve put them in the Appendix below. Suffice it to say that a) their equations and measures seem dubious; b) their results are consistent with other explanations; and c) it actually seems pretty benign and unsurprising that as you add more people to a task, the output per person falls.

I don’t think we’ll actually get far by arguing over the data, because we’ve got the underlying model all wrong. The metaphors we use to describe scientific progress—foraging! mining! drilling!—all assume that each generation of scientists simply does the same thing as the previous generation, just with more complexity and precision. Our ancestors mined the surface; we mine the depths.

That doesn’t seem quite right. Democritus and Einstein both contributed to physics, but one made claims with words and the other made claims with numbers. Sigmund Freud and Lee Ross were both great psychologists, but one did cocaine and free-associated while the other put people in situations and recorded what they did. Einstein and Ross didn’t simply forage farther or drill deeper than Democritus and Freud; they did something fundamentally different.

You don’t even have to look across thousands of years to see these qualitative shifts. Twenty years ago it was perfectly acceptable to run a psychology experiment that had thirty participants in each of four conditions, drop a few outliers, do a bunch of statistical tests until one of them spits out p < .05, and publish the results. Now we know how easy it is to get statistically significant results from a few seemingly-innocuous choices like these, and publishing that same paper today would get you laughed off Twitter. I predict—well, I hope!—similar shifts will happen in fields like biology, where it will become ridiculous to make strong claims about humans from lab mice, and in neuroscience, where it will become ridiculous to put people in a big magnetic tube, look at where their brains light up, and claim you’ve discovered consciousness, or something.

Science is not like foraging, mining, or drilling, where we keep doing the same thing and it keeps getting harder. It’s more like discovering an elevator left for us by aliens. At first we have no idea how it works; we get in and push a button, and now we’re climbing dozens of floors in a matter of seconds. We excitedly calculate that, at this rate, we’ll reach outer space in a few hours!

But at some point, the elevator stops. We try everything, but we can’t get it to go any higher. Eventually we figure out how elevators work and we start building even taller elevators. Another golden age ensues—there’s taller elevators every year!

But as the elevators get taller, the engineering gets more complicated. We need more elaborate support structures and stronger materials just to keep the elevators from toppling over. Pessimists start proclaiming that we’ll never reach the stars, or even the stratosphere. The elevators simply can’t reach that high!

The way to go higher, of course, is not to build taller elevators. It’s to invent hot air balloons. Once we discover lighter-than-air travel, it’s easy to fly as high as the highest elevator, and far above it. And when the balloons can’t go any higher, the solution is helicopters. And when the helicopters can’t go any higher, the solution is rocket ships. And when the rocket ships can’t go any higher, the solution is something we haven’t invented yet. No doubt these space conveyances will be complicated, but so were elevators before we knew how they worked.

This metaphor captures an important truth that other metaphors don’t: not every paradigm shift is going to be equally useful to the average person, or equally impactful on the same dimension. People may point out that while improving elevators helps us build taller apartment buildings, inventing hot air balloons does not. “Science is providing diminishing returns to housing,” they say gravely. That’s true, but once you can build really tall buildings, limits on housing quickly become political rather than technological. For example, 40% of buildings in Manhattan could not be built today because of increasingly strict zoning requirements. (Some of them would be forbidden because they're too tall and contain too much housing!) Once we solve the scientific part of a practical problem, we can’t continue to measure scientific progress by how well we’re doing on that problem.

How to stop science

Thomas Kuhn said something like this sixty years ago. He pointed out that scientific fields tend to putter along until people start noticing problems with the prevailing paradigm (“The planets don’t move like the theory says they should!”). Eventually the problems get too big to ignore and the whole field flies into crisis, and things only settle down when someone proposes a new paradigm that can make sense of everything. Then the cycle repeats.

Two key ingredients in scientific revolutions, then, are noticing problems and taking them seriously. Kuhn assumed scientists do both of those things naturally, and maybe that was true in 1962. But it doesn’t have to be. As fields formalize, they can get very good at ignoring and suppressing problems, delaying revolutions indefinitely.

One way professional science does this is by preventing divergent thinkers from entering in the first place. The usual way of becoming a scientist is to become a professor, ideally at a wealthy institution that can furnish you with lots of science gizmos and attractive letterhead for your grant applications.

The path to that prestigious professorship has become ludicrously competitive. Harvard, the ideal first step in an academic career, accepted just 3.19% of undergraduate applicants last year, down from 7.1% in 2012. Harvard doesn't publish overall PhD acceptance rates—the next step on the academic ladder—but the engineering school does and it’s a mere 7%. Only 15%-30% of PhDs who make it through this gauntlet and still want a permanent academic job will actually get one, and very few will be at the fancy places.

To win one of these coveted positions, then, you need to do everything exactly right from your freshman year of high school onward: get good grades, garner strong recommendations, work in the right labs, publish papers in prestigious places, never make anybody mad, and never take a detour or a break. (I occasionally get emails from high schoolers begging to work in my lab so they can get their name on a paper.) Professors who got their jobs decades ago tell us it wasn’t like this. In this hypercompetitive environment, the most fervent careerists will outcompete everybody else. And fervent careerists don’t produce revolutionary science.

The other way professional science can prevent problems from accruing is by simply refusing to publish them. Pre-publication peer review has only been popular for about fifty years—the prestigious medical journal The Lancet only started reviewing papers in 1976. If your paper or grant application threatens to undermine Dr. Tweedledum’s theory, there’s a good chance Dr. Tweedledum is going to review it, and they’re not inclined to be kind. Everybody knows this, of course, so they don't even try. You need publications and grants to survive, so it’s much better to work on something you know will be publishable at the end. This also rules out revolutionary science.

Professionalized science, then, may force us to keep building elevators even though we can’t get them to go any higher. Weirdos with crazy schemes for hot air balloons simply don’t get jobs; after all, they don’t even have a degree in elevators! Anyone lucky enough to stay in the pipeline has to apprentice under an elevator-builder, so building elevators is all they’ll ever know. Besides, everyone knows that you can’t get helicopter research published in the Journal of Elevators, and the National Elevator Foundation will never give you money to build one. And our esteemed elders assure us that elevators have an excellent track record and the recent slowdown in elevator progress is only because it’s harder to build taller elevators, so really what’s needed is more elevator funding. But even then, they warn, it won’t ever be possible to reach the stratosphere, let alone outer space. We must resign ourselves to looking for discoveries in the sixth place of decimals.

A refuge for cowards

“Ideas are getting harder to find” is a pretty bleak thing to believe. It says, “Look around the world. This is pretty much as good as it gets; the returns start diminishing from here. All the problems you see are unlikely to be solved anytime soon, so you better get used to them.” Why would anyone agree to such a thing without putting up a fight?

I think, deep down, part of us wants to believe it, because pessimism is really a clever excuse for cowardice. If you believe people are evil, you can be excused from ever trying to befriend them and maybe being rejected. If you believe the world is stacked against you, nobody can blame you for always playing it safe. And if the easy ideas have already been taken, you can’t be expected to come up with anything new.

I don’t blame people for using pessimism as an opiate; we could all use some relief right now. Wages are stagnant, inequality is rising, and if you don’t have a house yet, good luck buying one. The government is sometimes run by people you dislike; the rest of the time it’s run by people you hate. People are dying of preventable diseases while other people are launching cars into space. Everything’s on fire and nobody’s doing anything about it. This doesn’t seem like a world where much is possible, so why not lower your expectations?

But if we want a better world, we have to believe it’s possible to create one. And that takes courage, because if you truly believe in a better world, you have to do something about it. You don’t get to smugly smirk as the ship sinks; you have to start pumping the water out. Smirking seems easier than pumping at first, but pumping turns out to be really fun. You start to feel useful. You make friends with the other people trying to keep the boat afloat. You stop caring about all the people who say what you’re doing will never work. Ultimately, pumping is way easier than smirking, and it feels better too. Optimism cures pain; pessimism, like painkillers, merely dulls it.

The way I see it, if you want to write a song the world hasn’t heard before, you have two choices. You can spend your time calculating how there’s only a finite amount of different melodies, so eventually humans will run out of songs to write, so why bother. Or you can pick up a guitar and play.

APPENDIX: the technical objections

This paper by Bloom, Jones, Van Reenen, and Webb claims that ideas are getting harder to find. They summarize their argument like this:

There’s something very strange about this equation. It implies that the only way to get negative economic growth—that is, a recession—is for either research output or the number of researchers to be negative, which seems unlikely to happen. But if both are negative, economic growth is positive! Clearly, the relationship between researchers, productivity, and economic growth is a lot more complicated, and that makes me doubt that economic measures do a good job capturing the progress of science.

I’ve got three other gripes with this paper. First, it casually swaps correlation and causality, assuming that increasing numbers of researchers have been necessary for maintaining the same level of productivity. For instance:

...this growth has been achieved by engaging an ever-growing number of researchers to push Moore’s Law forward. In particular, the number of researchers required to double chip density today is more than 18 times larger than the number required in the early 1970s. At least as far as semiconductors are concerned, ideas are getting harder to find.

This isn’t evidence that making better computer chips required additional researchers. We don’t know what the growth rate would have been if we had kept the number of researchers the same.

My second gripe is that the theory seems to explain too much. The authors find research productivity slowing down everywhere they look, which they interpret as robust evidence for “ideas getting harder to find.” Even if ideas really were finite and get harder to find over time, should we really expect that to be happening in every single field in this exact time period? There isn’t a single discipline where some breakthrough led to a period of increased productivity? Some of these fields, remember, are way younger than others. Shouldn’t we expect some of them to still have lots of easy ideas left, and others to have fully entered their twilight stage, where there are only extremely hard ideas remaining? Science has been going on for a while, so why should this be happening only in our lifetimes, and not fifty years before or after? This seems like a pretty extraordinary coincidence.

Which brings me to my final gripe: the paper interprets any drop in research productivity as “ideas getting harder to find.” But that’s just one explanation; there are lots of plausible alternatives. Ben Southwood suggests it may be because industrial labs were replaced with less efficient academic departments, or because geniuses don’t go into research anymore, or because lead poisoning is making us dumber. Jay Bhattacharya and Mikko Packalen think it’s because scientists have become obsessed with citations, which leads them to do incremental rather than revolutionary work. They illustrate their theory with this extremely charming figure:

At least everybody’s still smiling!

These explanations sound plausible to me. And I’d go even further. Above a pretty low threshold, we should expect per capita productivity to drop whenever we add more people.

Say you get two guys to remodel your kitchen, and they tell you they can do it in two weeks. “Perfect,” you say. “I’ll just hire 2000 guys, and the job will be done in about 20 minutes!”

That won’t work, of course, for all sorts of reasons. (Though it does make a good episode of Nathan for You.) You can’t fit that many people in your kitchen—even 20 guys would be bumping into each other all the time. It’s unlikely that the 2000th contractor you hire is going to be as good as the first. Working with that many people requires lots of management and planning. The work can’t be done entirely in parallel—you can only hang the light fixtures after the wiring is done, for example. And some things simply can’t be sped up: paint just takes a while to dry, no matter how many people are waiting around.

Some of the same problems may arise in science. It takes a lot of work to manage lots of researchers, which is why people who lead big labs today spend much of their time being CEOs, fundraisers, and human resources managers rather than scientists. A few individuals may disproportionately drive progress, so adding researchers might actually decrease progress per capita. Some fields may be stymied until progress is made in other fields. And some science simply can’t be sped up: humans age, bacteria multiply, and light waves travel at the same rates no matter how many people are studying them.

Plus, researchers have lots of perverse incentives that remodelers don’t. The remodelers all want the same thing and are willing to follow the same plan and take direction from a supervisor. Researchers, on the other hand, compete with each other. They might steal each other’s ideas, or purposefully tank each other’s papers and grant proposals. Their jobs depend on them looking productive, so they pump out pointless papers to lengthen their CVs. These problems only grow as researchers multiply and the field gets more competitive.

That’s why I wouldn’t find it very surprising if per capita research output has dropped as the number of researchers has increased. It isn’t convincing evidence that ideas are getting harder to find. I still think we have a big problem, but it’s a solvable social problem, rather than an unsolvable scientific problem.

As a professor of biomedical engineering, Martin Uecker perhaps does not fit the profile of a typical presenter at Kernel Recipes. He is, however, a longtime Linux user, and works on free software for controlling magnetic resonance imaging (MRI) scanners. He was at the conference to talk about the C programming language, the specific problem of undefined behavior in C, and whether it can eventually be made into a memory-safe language.

Why bother with C in 2026? It is, he said, still a great language. C is portable, stable over the long term, offers fast compilation, and the resulting binary code is fast. "What you see is what you get"; it is easy to look at C code and have some idea of what the computer will actually do. There are a lot of tools for working with the language, and C gets out of the way when necessary.

C does have a long history, and that affects the language as we see it today, he said. The C89 standard had to cope with a wide variety of hardware, including machines with signed-magnitude or one's-complement integer representations, segmented memory, exotic pointer representations, and surprising sizes for types. Some Honeywell machines, for example, had nine-bit bytes. That greatly complicated the task of writing a standard that would enable the writing of portable code.

The approach that was taken was to define the semantics of the language in terms of an abstract machine. All operations are to be executed as if they had run on that abstract machine, which may not exactly match the actual hardware. The observable behavior of the program must be what the abstract machine would have done. The "observable" part matters: access to volatile variables, being defined as observable, must happen exactly according to the abstract machine; everything else just has to produce the same eventual result.

The standard gives a lot of freedom to compiler implementers; only the observable behavior has to be preserved. There are many aspects of that behavior that are either undefined or implementation-defined. These are not observable behavior, and thus do not constrain what compiler implementers can do. There are, of course, other specifications that can constrain compiler developers where the C standard does not; these include ABI requirements, standards like POSIX, or the need for backward compatibility.

Undefined behavior comes about when a program does something that is either not portable or not defined by the standard at all. In such cases, the C89 standard states that it "imposes no requirements" on the implementation. Undefined behavior exists for a number of reasons. It allows implementations to support extensions, manage interactions with hardware-based safety mechanisms, and perform aggressive optimization, all while allowing difficult-to-detect errors to be ignored. It explicitly gives the compiler the right to ignore whole classes of hard-to-detect errors.

Nasal demons

The problem, Uecker said, is that the standard allows a compiler to do anything in response to undefined behavior, up to the point of invoking nasal demons. If a program contains any undefined behavior at all, according to compiler writers, then it has no expected semantics. The C++23 standard goes further to explicitly state that the standard imposes no requirements for these programs. That has led to widespread disagreements between developers about what can be expected from the language.

For example, if you zero an entire structure (perhaps with a call to memset()), then write to specific fields, what will happen if you read from any padding bytes in that structure? Might they contain security-relevant data? A 2015 survey showed that there was no consensus on what should happen in that case. Or consider this simple code:

    extern int x;

    int f(int a, int b)
    {
    	x = b ? 42 : 43;
	return a/b;
    }

If b is zero, then the return statement is a division by zero, which is undefined behavior. In this case, is the compiler entitled to omit the test entirely and just execute x = 42? After all, the b = 0 case has no expected semantics, and can thus be ignored. There are compilers that will do exactly that. In the undefined-behavior case, the store to x is not observable behavior. But now consider this case:

    extern void g(int x);

    int f(int a, int b)
    {
        g(b ? 42 : 43);
	return a/b;
    }

This might seem to be the same situation, with the compiler being entitled to remove the test and just pass 42 to g(), and some compilers have treated that way — but that compiler behavior was a bug. Imagine a definition of g() that calls exit() if b is zero. In that case, the division will never happen and the program's behavior is not undefined. So eliding the test and simply passing 42 to g() is incorrect.

One more interesting case:

    volatile int x;

    int foo(int a, int b, bool store_to_x)
    {
	if (! store_to_x)
	    return a/b;
	x = b;
	return a/b;
    }

The question here is: can the compiler hoist the final division operation above assignment to x? If there are no semantics associated with the b = 0 case, then there is no change in observable behavior. This, too, is something compilers have done, but the C23 standard added a "no time travel" stipulation to disallow it. In C++, instead, hoisting must be explicitly prevented by inserting a call to std::observable_checkpoint().

Time-travel bugs should eventually go away, but there are a lot of other situations where, even if the standard is clear, compiler writers often disagree. These include reading of uninitialized variables (which is almost always defined), and equality comparisons of pointers, which is always defined, but is also miscompiled by both Clang and GCC.

Fighting undefined behavior

To try to address all of these problems and more, the C committee runs three study groups focused specifically on the memory object model, memory safety, and undefined behavior. There are currently about 100 instances of undefined behavior in the C standard, but the in-progress C2y draft has removed 45 of them. The situation is indeed getting better.

There is an increasingly rich set of tools aimed at finding issues: compiler warnings, static analyzers, sanitizers, LLM-based tools, formal verification, and more. The number of situations where a compiler will emit a warning where possible undefined behavior is detected is growing; recent examples include better warnings for integer overflows and potential use-after-free situations. Static analyzers are available as standalone tools, but are also increasingly being built into the compilers themselves; GCC can now warn about a number of potential buffer-overflow situations, for example. Sanitizers work by inserting run-time checks; they can catch a lot of undefined behavior and, in trapping mode, be used for hardening as well.

Memory safety has never been one of C's strong points, but Uecker wanted to make the point that it can be improved. That problem breaks down into three sub-problems: type safety, spatial memory safety, and temporal memory safety.

C, he said, has a strong type system, and the remaining problems are fixable. Tagless unions, for example, can create type confusion, but the compiler can enforce types with some additional annotations. New diagnostics can catch unsafe casts from void. Type checking across translation units is traditionally not a huge problem in C, since header files are used to ensure consistent types, but the situation could be improved with a link-time checker.

Spatial memory safety — bounds checking — is a partially solved problem; the compilers can perform array-bounds checking in many situations now. In some cases, some code changes are needed to fully benefit from this checking. Use of the counted_by attribute can enable checking for flexible array members, for example.

Temporal memory safety — avoiding use-after-free bugs and the like — is harder, Uecker said, and Rust definitely has an advantage there. Still, better temporal memory-safety enforcement is possible. Architectures like CHERI can help here is well. Fil-C can find a lot of temporal-safety bugs.

Can all of these tools and language changes get us to full memory safety? Completely solving the problem will require either expensive run-time checking or formal verification, he said. In the near future, the most complete results will be had with the combination of a restricted language and formal verification tools.

Overall, he concluded, C is still a living language and is still improving. The C23 standard removed a number of problematic features, including old-style (K&R) function definitions, support for sign-magnitude and one's-complement machines, and trigraphs. It added bit-precise integer types, checked integer operations, and more. C2y will go further, adding case ranges, named for loops, the _Countof() macro to determine array lengths, and a lot of "demon removal". It will not achieve full memory safety for C, but that is an eventual possibility, and will become more practical over time. He ended by encouraging interested people to participate in the working groups.

The video and slides from this talk are available.

[Thanks to the Linux Foundation, LWN's travel sponsor, for supporting my travel for this event.]

TL;DR

All cloud products require versatile and reliable connectivity options, so that our customers and potentially their customers can reach the services running in our network. Since the start of the vServer product, we had different technologies to provide that connectivity, with all of them relying on hosts rather than a smart network. With the launch of Hetzner Cloud and the introduction of network features like Private Networks and Firewalls, we introduced Open vSwitch. We’ve built a custom solution to orchestrate Open vSwitch, and especially the necessary flows to handle traffic. We’ve designed our network stack(s) as solutions we own and can operate ourselves, and to have building blocks, where we can further build upon.

This blog post is the first part of a series covering the evolution of the network stacks behind Hetzner Cloud and their architecture. In this post, we will look into the development we’ve done until today, and how the current Open vSwitch-based network stack works. Although the network stack is similar for Cloud servers and Load Balancers, we will focus on Cloud servers and virtualization hosts (VM hosts), as they are providing more services and features.

For brevity, we will keep the Cloud Control Plane, which manages all servers, Load Balancers, Private Networks, etc. out of scope, and will only look into the network stack on the hosts.

How we roll

All cloud products require versatile and reliable connectivity options, so that our customers and potentially their customers can reach the services running in our network.

At Hetzner, we generally believe in solutions we own and can operate ourselves, and in having building blocks, where we can further build upon. The network, and especially the network connectivity for our cloud offerings, is no different.

Although technologies changed over the years, we’ve always preferred solutions, where the moving parts are on the hosts, rather than having a smart and complex physical network setup. As a result, we designed the setup in a way, that the physical network is providing stable IP connectivity to the hosts, and the hosts offer higher level network functions. These include Firewalls, Private Networks, and auxiliary infrastructure services.

By running these services and network functions on the hosts, we gain a lot of freedom and flexibility when designing and building our product, and are not limited to what hardware vendors offer. This way, we have a clear separation of concerns between teams and technologies, and can contain the blast radius to a single host, in case of (most) errors. This also means that we can and must fully own the solution and adjust it to our needs, but that’s in our DNA anyway.

A bit of history

Historically, Hetzner has been using different technologies to provide connectivity for its cloud products.

In the early days, the Hetzner vServers launched in 2011 started out with HDDs, native Linux bridges, and static routes. It provided dual-stack connectivity with up to 1 Gb/s per host. In the end, there were approximately 25,000 instances running with this setup.

The follow-up product launched in September 2015 with a hyper-converged Ceph-based setup. It switched to dynamic routing via BGP and a Linux bridge setup using 1:1 NAT for IPv4 and fully routed IPv6 prefixes. This enabled better IP mobility and host maintenance without customer impact. In 2016, the links to the hosts were upgraded to 2x 10 Gb/s. This setup lasted until 2018 and grew to about 50,000 instances.

When Hetzner Cloud launched in 2018, we dropped NAT and moved to direct IPv4 routing. In July 2019, we introduced an Open vSwitch-based data plane, and added support for private cloud networks based on VXLAN encapsulation. In November 2020, we started migrating the public network part over to Open vSwitch, allowing us to launch stateful cloud firewalls based on Open vSwitch flows and netfilterin March 2021.

Ingredients of a VM host

We’ve already established, that important parts of the network stack are running on the VM hosts; but what do they have to provide for a cloud server to operate?

For the majority of servers, Internet reachability is required, so the system and its services can be accessed from home or abroad. Some servers use Private Networks, either instead of, or alongside public network , which connect servers, Load Balancers, and possibly dedicated servers via a private and virtual network that only your systems can access. Private network traffic is encapsulated using Virtual eXtensible LAN (VXLAN) and a unique Virtual Network Identifier (VNI) per private network. The host network stack ensures that only members attached to the respective private network can send or receive traffic.

To control access to your services available from the Internet, we provide the cloud Firewall feature. It allows controlling access to certain ports or protocols of the server, or can limit which traffic can leave the server. This should happen as close to the server as possible, hence on the host. Building stateful firewalls centrally, would require additional hardware and network overlays to transport the “clean” traffic to the servers, and add much complexity for highly available, central connection tracking.

By default, our cloud server images are configured to use the Dynamic Host Configuration Protocol (DHCP) for dynamic configuration of IPv4 addresses and routes. Hence, we have to provide a DHCP server, so servers can request their IPv4 networking configuration for public and private network interfaces.

Within most images, cloud-init is used to configure some parts of the system, for example configure IPv6 networking, trigger DHCP for private network interfaces, attach cloud Volumes, etc. For these tasks, cloud-init has to query information about the local system or user-specific configuration bits from our metadata server available at http://169.254.169.254/. Besides cloud-init, the metadata server is used by many integrations, including but not limited to our CSI driver, hc-utils, distribution tooling like Flatcar Linux's Afterburn and Ignition, or Talos Linux.

Both the DHCP and metadata servers are running locally on each VM host for maximum resiliency and availability.

Besides these user-facing services, the VM hosts run a bunch of internal services, which make sure your servers are up and running, the network stack gets the correct information to configure connectivity, and our monitoring knows what’s happening.

In this post, however, we’ll focus on the network stack and will have a deeper look into its inner workings.

Open vSwitch

Most of the network data path is realized by Open vSwitch, or in short OVS, an Open Source multilayer virtual switch. Its data path is included in the Linux Kernel for a long time, and packages for the user space / control plane are available for all major Linux distributions.

It is the default network stack for some virtualization environments, and many off-the-shelf solutions come with OVS support. This includes but is not limited to Proxmox VE, Open Stack, and oVirt, to name a few popular ones. We however, don’t use any of these, but rather manage KVM and all surrounding components using custom-built tooling.

Open vSwitch is a versatile networking solution, offering a vast amount of features, of which we’re only using a subset for our use cases and setup. It uses OpenFlow, an open protocol built to configure the data plane of network devices and denote how packets should be forwarded. For each direction of communication, there has to be a flow specifying how packets should be handled, and where they need to be sent.

This can be exact matches, e.g. for communication with the gateway via ARP, NDP, ICMP, etc., traffic destined for other members of a private network, or connections to a service on the Internet.

Open vSwitch orchestration - Flusskrebs

To provide network connectivity for our servers and Load Balancers, and to configure any Firewall(s) applied to a server, we have to install the required flows for Open vSwitch to provide these connections and features.

A common solution to orchestrate Open vSwitch would be the Open Virtual Network (OVN) control plane project, which was started alongside OVS itself and is intended for maintaining a fleet of nodes running Open vSwitch. It provides an abstraction layer, where one can configure logical routes and switches, which then get translated into OpenFlow.

At the time when Open vSwitch was introduced into the Hetzner Cloud stack, we decided to not go down this road, and rather built our own custom solution called Flusskrebs (literally translates to “river crab” or “flow crab” 🦀). It is written in Python, provides a REST API used by the host-local part of our Cloud control plane, and configures the flows each VM or Load Balancer needs to work properly. This includes flows to implement any Firewall rules configured by the user or our backend, e.g., for blocking egress SMTP by default.

The controller also acts as the central DHCP server for private network interfaces on each host.

VM host network stack

With the general architecture ideas and these building blocks in mind, let's have a look at our VM host network stack in more detail. The following figure shows a simplified overview of the inner working of the network stack of a VM host.

Most hosts are using two 10 Gb/s uplink interface, which are bundled into a link aggregation group (LAG) or bond on Linux using LACP, connected to two physical switches forming a virtual chassis for increased resilience and bandwidth. To increase resilience even further, we’ve started migrating to two native routed uplinks using BGP, connected to two independent switches earlier this year. Note, that the uplink interfaces are not directly connected to the OVS bridge, but the Linux system routes between the uplinks and OVS. This way, the host reachability is independent of OVS, and it eases host installations and maintenance.

A central Open vSwitch bridge acts as the main component delivering network connectivity and features, and all cloud servers on the host are connected to it. Each server can have one public network interface, and additionally up to three private network interfaces. The following templates are used to render flows allowing egress traffic from the server:

# Only process packets with correct source MAC address
table=0, priority=9999, in_port={vm_port_no}, eth_src={mac}, actions=set_field:1->reg0,set_field:{vm_port_no}->reg5,resubmit(,1)
# Only accept packets with an IPv4/IPv6 address associated with the server
table=3, priority=9999, in_port={vm_port_no}, ip_src={ip}, ip, actions=resubmit(,4)
table=3, priority=9999, in_port={vm_port_no}, ipv6_src={ip}/64, ipv6, actions=resubmit(,4)
table=3, priority=9999, in_port={vm_port_no}, arp, arp_op=2, arp_spa={ip}, actions=resubmit(,4)

Ingress traffic towards the server is forwarded using the following flows, one for each IPv4 or IPv6 prefix respectively. The host network stack identifies itself to the VM using a well-known “virtual gateway MAC” which is always d2:74:7f:6e:37:e3.

table=22, priority=9999, in_port={internet_port_no}, ip, ip_dst={ip}, actions=mod_dl_src:d2:74:7f:6e:37:e3,{vm_port_no}
table=22, priority=9999, in_port={internet_port_no}, ipv6, ipv6_dst={ip}/64, actions=mod_dl_src:d2:74:7f:6e:37:e3,{vm_port_no}

Infrastructure services, like DHCP and metadata servers are also connected to Open vSwitch and are running alongside each server inside a dedicated Linux Network Namespace per server.

Note that the udhcpd DHCP server shown in the drawing above is only responsible for any server’s public interface. As described above, the Flusskrebs controller contains a central DHCP server handling all private network interfaces. The flows to connect the public DHCP server are equally straightforward:

table=1, priority=9999, in_port={vm_port_no}, udp, udp_dst=67, actions={service_port_no}
table=1, priority=9999, in_port={service_port_no}, udp, udp_src=67, actions={vm_port_no}

Alongside Open vSwitch, the Linux netfilter connection tracking is being used to realize the stateful cloud Firewall feature. With a global shared connection tracking table, we have to prevent it from overflowing and ensure fair use for all servers and customers. This is done by ctcount, another internal project, which keeps track of connection tracking entries, and enforces the limit of up to 80,000 active, concurrent connections per server. If a server reaches this limit, no additional connections can be opened, until a previous one has been closed.

Status Quo

This Open vSwitch-based data plane has served us well and mostly still does today, even with over a million cloud servers. However, we have reached some limitations and wanted to improve scalability, resiliency, and flexibility with a more specialized network stack. Over the last years, our SDN team has been building said solution that's tailored to our needs while being easy to operate and being a strong foundation for new features, such as IPv6 for private networks.

We’ll cover the next steps of our journey, our latest fully self-built network stack, and how it works, in the next part of this series. So stay tuned!

At Hetzner Summit, we had a session "The new Hetzner Cloud network stack - A technical deep dive into how we deliver your packets", where we talked about our current stack and what we’ve built. The recordings are available on the Summit page.