Despite what I said in April, I'm still using coding assistants on my personal projects.

Using them to one-shot entire features leaves me unsatisfied and disoriented, but I do enjoy using them to fast-forward through the boring parts of my projects.

However, allowing my coding assistant to roam free in my projects leaves me with a colossal amount of cognitive debt. I might hate the idea of poring over the Django documentation to figure out how to add tagging to my website, but I still fundamentally want to understand how it works. Just because a problem is boring doesn't mean I want to fully offload my understanding of the solution to a machine.

Of course, I could review every single line of code the LLM produces. That's what most developers are expected to do in this cursed year of 2026. Robots raise PRs, humans review them. It's a brave new world.

But I don't enjoy reviewing AI-generated PRs. Poring over hundreds of lines of overly-defensive, badly-commented, subtly incorrect code is not fun. I might grudgingly do it for an employer—while making sure said employer becomes an ex-employer as soon as possible—but I'm sure as hell not doing it for my personal projects. Personal projects must be fun above all else. The joy of working on personal projects comes from the process, not from the outcome.

So what's a boy to do? How do I offload the boring work to LLMs without ceding control of my own work and cognition to the slop machine?

I've come up with a solution that's grossly inefficient and perhaps slightly comical: I ask my coding assistant to generate code in the chat, then manually make all the edits myself.

I have these instructions in all the agents files in my personal projects:

I want to understand every line of code that goes into this project. Never create, edit, move, rename, or delete project files unless I explicitly ask you to do so. Instead, show me every proposed edit in the chat so I can type it in manually.

Do not run commands that modify project files, install dependencies, or change repository state unless I explicitly request that action. Instead, show me those commands in the chat so I can run them manually.

I'm an experienced developer. Do not explain syntax, APIs, programming concepts, or implementation details unless explicitly asked.

Using LLMs this way allows me to work faster than not using LLMs at all, but I'm still slower than those who are willing to allow the machine to think for them. Instead of being 10x faster, I'm probably only 2x faster. But what I lose out on in terms of speed, I gain in terms of a deeper understanding of my code.

As I manually type every single line of LLM generated code into my editor, I build up a mental model of how it works and fits into my existing codebase. If I don't understand an API or algorithm, I can stop to look it up, or just ask the LLM to explain it.

Typing the code myself forces me to slow down, which means I'm more likely to detect hallucinations or bad design choices the LLM might have made. I can clean up the code as I go, reorganizing it, refactoring it, adding comments, and generally adapting it to my own taste.

Most importantly, this workflow allows me to build a spatial map of my codebase. I know where every bit of functionality lives in the codebase. When I need to make a change, I know exactly where I need to make it. It not only helps me work faster within my projects, it also makes it easier for me to better prompt and instruct the LLM in the future.

When I was learning to code as a teenager, experienced programmers would often tell me to never copy and paste code into my projects. If I was learning from a book, I was advised to copy all the examples into my computer and make sure I could run them. If I was learning from a blog post or forum answer, I was advised to type it out and adapt it to my codebase so I understood it completely.

Manually typing LLM-generated into my codebase feels like the exact same learning process. It might not be the most efficient way to work with an LLM, but I value comprehension over productivity. I've been doing this for a few months now, and it's been working well for me. I plan to continue using this workflow for as long as I can.

I fear the software industry is taking on a large amount of cognitive debt that we'll have to pay back very soon. There will come a time when we no longer understand how large parts of our digital infrastructure are put together. I might not personally be able to change the course of the entire industry, but I can at least make sure I completely understand the software I put out into the world. Anything else would be professional malpractice.

In the 2010s, if you had technical gaps (say, you couldn’t write CSS), you had to either rely on a skilled colleague or just hope that the answer to your exact problem was out there on the internet. Today, everyone can write sort-of-okay CSS by delegating the task to an LLM. LLMs make everybody into a generalist.

Because of this, lots of people don’t think there’s any skill involved in working with LLMs. If you want the product that LLMs can deliver — PhD-level mathematics, pretty good but sometimes tasteless computer code, or awkward LinkedIn-style writing — you can simply ask for it. Since everyone is talking to the same models, “skilled prompters” are getting the same results as people touching LLMs for the first time.

This is wrong. The most important skill in prompting is expertise in the domain you’re prompting for.

A good illustration of this is Terence Tao’s conversation with ChatGPT about the recently-discovered counterexample to the Jacobian Conjecture. This is not the same ChatGPT I talk to! I couldn’t get to where Tao gets, even with unlimited tokens to burn.

There’s a lot to learn about good prompting from Tao’s conversation. Here are a few observations:

• Tao’s messages are very short and to-the-point. He doesn’t respond point-by-point to the model, just to the gist
• The model outputs are much more concise than when I try and talk to GPT-5.6 Sol about mathematics. By signalling expertise, Tao shunts the model into “talking-to-mathematicians” mode, not “explaining-to-amateurs” mode
• Tao pushes back when the model’s responses look wrong, but he doesn’t directly contradict; instead, he says things like “this looks more complex than I was hoping for”
• Tao makes several leaps and suggestions himself. He almost never takes the model’s advice about where to go next

However, you can’t prompt like Tao on mathematical questions just by following these tips. The key to his technique is actually understanding the mathematics: pulling the relevant idea out of ChatGPT’s multi-paragraph response, suggesting alternate approaches or formulations, and identifying what “looks weird”.

Terence Tao is a better mathematician than I am a programmer. But the idea here — that domain knowledge makes you better at using LLMs — is something I’ve also experienced in my own work. If you have a good theory of your codebase, you can push the LLM much harder than if you have no familiarity. Because you have your own sense of what a good solution might look like, you can say “no, I think it could be simpler here”, or “but don’t we already do X?”, or “can we express this problem in these familiar terms?“.

This touches on an idea I’ve written about before: that system design problems are dominated by concrete specifics, not generic principles. Of course both are useful, but I’d rather have familiarity with the codebase than a deep general understanding of software systems. In his conversation, Terence Tao asks a lot of specific questions like “does X work here?”, or “given Y and Z, why A?“. I can’t ask those questions about the Jacobian Conjecture, but I can ask them about the systems I own at GitHub.

If you have no domain knowledge, you can cling onto the LLM to at least get something. That’s not bad! But if you have domain knowledge, you can wring far more value out of the same LLM by steering it hard in the direction you want. Most of us will have to do a mix of both these approaches, since we have domain knowledge in some areas but not others.

The usefulness of domain knowledge suggests that human expertise will continue to be useful even as models get stronger. For many tasks, the human is the bottleneck, not the model, because the difficult part is in communicating to the model exactly what kind of solution the human wants. The information is “in the model” already, but it takes a very smart human to pull it out.

edit: this post got many comments on Hacker News. Some commenters share their anecdotes about how expertise has helped and lack of expertise has hurt. Other commenters say it’s plausible, but they have a sensible suspicion of a view that’s reassuring them about how they’re still valuable. I agree with that, though I suspect by the time we get around to studying this, the landscape will have changed under our feet again. Some commenters point out that OpenAI’s math prompts were inexpert, and so expertise isn’t required. Here I’d respond that OpenAI do have a team of expert mathematicians that checked and filtered the model’s suggested discoveries, and that you cannot currently skip that step.

We want to empower scientists and mathematicians with tools that accelerate discovery. That is why we recently announced ChatGPT for Academic Researchers⁠, an initiative providing 100,000 scientists and mathematicians with free access to our best ChatGPT models. We also continue to evaluate our models on open research problems during development.

In May, we shared an AI-generated disproof⁠ of the Erdős unit-distance conjecture, discovered while evaluating an unreleased model. This work has already inspired further developments in mathematics and theoretical computer science1. Today, we are sharing a selection of ten results, each of which resolves or makes substantial progress on a long-standing open problem. These problems span high-dimensional geometry, coding theory, arithmetic circuit complexity, group theory, operator algebras, quantum complexity, lattice cryptography and extremal combinatorics. All of these problems are of substantial interest to their respective mathematical communities, and several are of broad interest across mathematics as a whole.

The results

We provide new results for the following problems. The results were achieved by an internal version of Astra, our next major model. The total number of tokens needed to find solutions to these problems would cost roughly $2,000 at Sol API rates. These arguments were then prepared into manuscripts by humans with the same model. Afterward, the model formalized each argument in a Lean certificate⁠(opens in a new window). We are also releasing for each solution a model’s narration of its thinking process.

1. High-dimensional sphere packing. New upper bounds on sphere-packing density down to the Cohn–Elkies threshold.
2. Binary and spherical codes. Exponentially improved bounds on the maximum size of binary codes at any prescribed minimum distance, with analogous results for high-dimensional spherical codes.
3. Non-sofic groups. A construction establishing the existence of non-sofic groups, addressing a central open question in group theory.
4. Connes’s rigidity conjecture. Disproof of a longstanding conjecture that certain groups are uniquely determined by their von Neumann algebras.
5. Arithmetic circuit complexity. New lower bounds for computing the permanent using arithmetic circuits and formulas, including an arithmetic-formula lower bound of order n4/log n.
6. Quantum parallel repetition. An exponential parallel repetition theorem for general two-player quantum games, extending a foundational principle from classical complexity theory.
7. Closest vector problem. Polynomial-factor hardness of approximation for the closest vector problem, a foundational lattice question related to post-quantum cryptography.
8. Ehrhart’s volume conjecture. Determining, in every dimension, the maximum possible volume of a convex body whose centroid is its only interior lattice point.
9. Multicolor Ramsey numbers. A superexponential lower bound for multicolor triangle Ramsey numbers, resolving Erdős problem 183.
10. Extremal number conjectures. Results on the compactness and degeneracy conjectures in extremal graph theory, resolving Erdős problems 146 and 180.

Responsibility to the mathematical community

The emergence of systems capable of contributing to mathematical research raises questions that cannot be answered by a technology company alone. There are many views as to the role of AI in mathematics, and we have deep respect and understanding for those concerned with its impact, including the signers of the Leiden declaration on AI and Mathematics⁠(opens in a new window). We believe attribution should honestly reflect how a result was produced: claiming human authorship for a proof generated entirely by an AI system would misrepresent both the system’s contribution and the nature of genuine human intellectual work. We helped prepare the manuscripts and formalize the proofs in Lean, and we take responsibility for their correctness, while the mathematical arguments themselves were generated by our system. We hope the mathematical community will engage deeply with these results, place them in context, and bring the ideas behind them to life through new research and discovery.

As AI systems evolve into more sophisticated research collaborators, ensuring widespread access is fundamental to supporting scientists and mathematicians as they navigate and define the future of their disciplines during this transformative era.

Footnote

1. Subsequent research includes Bloom, Sawin, Schildkraut, and Zhelezov, “The sum-product conjecture is false for real numbers⁠(opens in a new window)”; Pohoata, “Split primes and the Elekes-Rónyai problem⁠(opens in a new window)”; Saha, Xu, and Ye, “Furthest Pair Requires Quadratic Time in Superconstant Dimension under SETH⁠(opens in a new window)”; Goh and Hatami, “Communication complexity of point-line incidences over the reals⁠(opens in a new window)”; and Lee, Pohoata, and Zhu, “The Minkowski grid has robustly many repeated distances⁠(opens in a new window).”

The economics behind where movies are filmed, why Hollywood left Los Angeles, and whether audiences can tell the difference.

La La Land (2016). Credit: Warner Bros.

Intro: #StayInLA

The past decade has not been kind to Los Angeles’ entertainment industry. The combined shock of the 2023 labor strikes and the 2025 wildfires intensified fears that film and television production is migrating overseas, threatening a fragile ecosystem of below-the-line workers based in Hollywood.

In the days after the wildfires, filmmaker Sarah Smith and writer Alexandra Pechman organized a WhatsApp group of showrunners and industry workers, drafted a petition, and launched the grassroots #StayInLA campaign. What began as an emergency effort to help Los Angeles rebuild grew into a broader political movement, drawing more than 20,000 petition signatures, endorsements from prominent actors and filmmakers, and even the fleeting attention of Donald Trump—an actor himself, best known for his performance in Home Alone 2: Lost in New York.

Having worked in the entertainment industry and lived in Los Angeles, I have long wanted to investigate this often-vague anxiety: to understand exactly where these productions are migrating and quantify the extent of Hollywood’s flight from, well, Hollywood itself.

So today, we’ll examine Hollywood’s great production migration, explore how these logistical shifts have reshaped the filmmaking process, and investigate whether outsourcing production has any measurable impact on the quality of the movies themselves.

How Hollywood Stopped Making Movies in Hollywood

The American film industry began concentrating in Los Angeles in the 1910s, drawn to the city’s year-round sunshine, varied landscapes, inexpensive land, and distance from Thomas Edison’s patent enforcement on the East Coast. By the 1920s, Hollywood had become the global center of filmmaking, with studios, skilled workers, and supporting businesses clustered together in a self-renewing ecosystem.

For decades, most American films were produced on Los Angeles backlots, where studios provided filmmakers access to turnkey sets and skilled crews. This mechanized approach kept costs low and enabled directors to crank out four to six movies a year. During this era, Hollywood operated much like Henry Ford’s Model T assembly line; studios were maniacally focused on speed and frugality.

LA’s stranglehold over movie production would ultimately be tested by two filmmaking trends:

1. The rise of the blockbuster: In the 1970s, box-office smashes like The Godfather and Jaws demonstrated the commercial potential of event cinema. By the 1980s, corporatized studios like Paramount-Gulf-Western were actively manufacturing blockbusters, pouring ever-growing budgets into spectacle-driven stories. Films like Predator, First Blood, and Raiders of the Lost Ark were shot on location in jungles, deserts, and other far-flung settings, as exotic locales amplified their sense of scale.
2. The rise of IP and tax credits: The 2000s and 2010s brought a surge in big-budget filmmaking built around familiar IP. Marvel’s vaunted Cinematic Universe and other franchises required considerable financial investment, leading studios to offset costs by relocating production to incentive-rich cities such as Vancouver and Atlanta.

The result of these production strategies was a slow-building migration out of Los Angeles and California altogether.

So where did all the movies go? Hollywood’s production exodus has benefited a small group of filmmaking hubs offering some combination of established infrastructure and financial incentives. London, for example, boasts world-class facilities such as Pinewood Studios, while cities like Atlanta and Vancouver provide generous tax credits and rebates that reimburse studios for their local spending.

The savings these destinations offer are substantial, making them an obvious choice for a superhero movie whose story could unfold almost anywhere—be it a generic cityscape or somewhere in outer space. For a major blockbuster, tax incentives equate to tens of millions of dollars, freeing up money for name-brand actors, elaborate set pieces, and unremarkable CGI effects.

As such, the larger a film’s production budget, the more likely it is to be filmed entirely outside the United States.

In some cases, this globe-trotting appears directly on screen, like when Mission: Impossible stages an elaborate car chase through Rome. In other cases, location choice is driven entirely by financial considerations.

The latter trend has created a widening gap between where stories are set and where they are filmed, as productions increasingly follow tax incentives and lower costs rather than narrative necessity.

Look closely, and you’ll spot Toronto’s skyline standing in for New York, or Prague’s streets masquerading as nearly every other European city. With each passing year, filming locations have become untethered from the places depicted on screen.

This disconnect has become a persistent grievance among film critics, particularly when it comes to streaming movies shot far from their narrative setting. Nearly every week, the hosts of my favorite movie podcast complain about some disposable Netflix rom-com ostensibly set in New York but unmistakably filmed in New Orleans. After hearing this criticism ad nauseam, I began to wonder whether shooting on location actually has a discernible impact on a film’s quality—at least in the eyes of viewers.

As a cinema purist, I believe movies should be filmed where they are set, even when that setting is outer space. Most audiences, however, either cannot tell the difference or simply do not care.

Perhaps I have gone to great lengths to prove something studio executives already know

Perhaps I have gone to great lengths to prove something studio executives already know: the average viewer is not scrutinizing every frame for telltale glimpses of the Toronto skyline. More likely, they are glancing at Instagram while the film’s protagonists laboriously explain the plot for the narrative comprehension of multitasking viewership—at which point the authenticity of the location becomes irrelevant.

Final Thoughts: Cost Disease

Guardians of the Galaxy Vol. 3

Guardians of the Galaxy Vol. 3. Credit: Marvel.

My junior year of college, I scored an internship at Conan O’Brien’s late-night talk show. So, like Miley Cyrus circa 2009, I hopped off the plane at LAX with a dream and an unpaid internship.

Within a week, I’d discovered three things:

1. Entertainment internships are often glorified daycare for the children of industry executives: I could be classified as many things, but “nepo baby” is not one of them—which made me an exception among my intern class.
2. There was no work to do: The show had far more interns than meaningful tasks. I never even earned the privilege of fetching coffee.
3. The Warner Bros. backlot was eerily empty: I had grown up on movies like The Player and Singin’ in the Rain, where Hollywood appeared as a bustling factory of overlapping productions squeezed onto compact studio lots. Instead, I found emptiness.

This combination of things led to a bizarre summer where me and a posse of underutilized unpaid interns wandered a near-vacant Warner Bros. studio as if it were the Backrooms. This meant hanging out in the Central Perk set from Friends or cart-racing other interns from The Ellen DeGeneres Show on the street where they filmed Gilmore Girls.

My lasting impression was that Hollywood had become a collection of offices where white-collar workers decided how the sausage got made, while remarkably little sausage was actually being made there.

It was also during this summer that I learned of the hazards of cost disease.

Cost disease is an economic phenomenon in which labor-intensive industries with limited productivity gains—healthcare, education, live theater, filmmaking—become steadily more expensive over time. The quintessential example of this is Broadway, where the cost of living in New York City increases, leading wages to rise, but there remains a fixed set of theaters for Broadway shows to be staged, thereby limiting revenue growth. The end result is economic unsustainability, as these productions either curtail spending by employing fewer people or pass higher costs along to consumers through ticket pricing, often both.

My summer in LA was cost disease in miniature. I was paid exactly $0 to relocate to Los Angeles, where I then paid thousands of dollars to live in America’s third most expensive city. And this was the very bottom rung of entertainment’s corporate ladder.

Higher up the corporate hierarchy, cost disease takes different forms. For entry-level assistants, it means accepting low wages and paying high rent for the possibility—however remote—of eventually joining the executive class. For camera crews and VFX artists, it means becoming part of a traveling circus that chases tax incentives to Atlanta, Prague, and Vancouver. For studio executives, it means making do with smaller crews, outsourcing production overseas, and, eventually, using AI to replace some human labor.

If this has all drifted into bummertown, it’s because many of Hollywood’s problems can be traced back to a single economic theory of everything: the rent is too damn high. From there, everything else starts to unravel.

The end result is a peculiar version of Hollywood: a place increasingly devoid of physical production, populated instead by executives, AI art, and unpaid interns drag-racing golf carts through empty studio streets.

I am excited to announce that I am joining ClickHouse to establish and lead a new research team called ClickHouse Labs. I want to share how it came about and what we plan to do.

How It Started #

I started as a professor in the Computer Science Department at Carnegie Mellon University in 2013. I have spent my career seeking to understand the science of modern database management system (DBMS) internals. I make it a priority to track every new system that comes along, both in industry and academia, to understand their implementations.

I have known about the ClickHouse DBMS since it was first announced as open-source software in June 2016. My initial reaction to this news was that it had to be vaporware because it seemed too good to be true. ClickHouse had features that at the time were only found in a handful of closed-source, commercial analytical DBMSs. For example, ClickHouse was written in C++ and supported vectorized query execution using SIMD in 2016. Most prominent open-source analytical DBMSs in 2016 were JVM-based and did not support SIMD optimizations until years later.

Since then, I have followed ClickHouse's development closely. It has always been a leading system that was highly relevant to our academic research projects. You can even see me wearing my original ClickHouse shirt in my first remote lectures in 2020, when the pandemic forced us to move our database courses online.

Given this history, I was honored when the ClickHouse co-founders invited me to establish this new research group at ClickHouse. The chance to work with one of the strongest engineering teams on the next generation of database technology was an opportunity that I could not pass up. This will be a next-level collaboration like when Killer Mike hooked up with El-P to create a hip-hop supergroup.

What Is ClickHouse Labs? #

The goal of ClickHouse Labs is to establish a best-in-class industry research organization focused on databases. It will not operate as an isolated research organization that throws ideas over the wall to engineering. Instead, we will work closely with ClickHouse engineers, customers, collaborators, and industry partners to develop and disseminate new ideas that keep ClickHouse at the bleeding edge.

We will also work with ClickHouse's PostgreSQL team to help establish its burgeoning managed service as a market leader in performance and reliability. PostgreSQL and ClickHouse serve different workload requirements, but the combination gives us a broad foundation for investigating both transactional and analytical database problems.

Our objective is straightforward but ambitious: conduct research with scientific value and then help transform the best ideas into technology that matters to users. I want to achieve the same level of impact associated with pioneering industry research organizations, such as IBM Research and Microsoft Research. Those groups demonstrated that industry laboratories can simultaneously advance fundamental computer science, influence commercial products, and train generations of database researchers. That is the tradition we want to continue.

What is Next? #

The ClickHouse team already has an exceptional record of publishing deep technical material about its work. Since the establishment of the company in 2021, its engineers have produced detailed articles that explain the DBMS's implementation. There is also the 2024 VLDB paper that describes ClickHouse's core architecture. These works are so thorough that I assign them as readings to my students at Carnegie Mellon. At the same time, there is a backlog of interesting ideas and optimizations that the ClickHouse engineering team has explored but has not yet had the time to validate fully and push into production. One of my immediate priorities is to help accelerate this process. We will then use that as a springboard to explore new ideas that push ClickHouse even further.

One larger question we will investigate is how DBMSs like ClickHouse and PostgreSQL fit into emerging AI and agentic technologies. There are two sides to this problem. The first is determining what a DBMS should look like to better support agents. The second side is determining how agents can improve and automate the development of DBMSs themselves. Everything is on the table: new hardware, new algorithms, new data structures, new execution strategies, and new ways of building and operating DBMS software. Although I do not have answers to these problems yet (this is why it is research), the one thing I am certain about is that ClickHouse's solid relational model foundation positions it well to evolve alongside these data-intensive workloads.

I have spent my career studying how database systems are built and helping train the people who build them. With ClickHouse Labs, we now have the opportunity to create an organization devoted to advancing both.

Image via CBC

Take capelin, a small silvery fish in the smelt family. Now add salt. Then throw those two ingredients in 110 vats, each of them about 10 feet tall, and let them sit in a defunct factory for over two decades. The resulting stench might haunt an entire community, as it has the people of St. Mary's, a coastal town of about 300 people in the Canadian province of Newfoundland and Labrador. In 2019, one resident described the ritual of abandoning her home every summer, boarding up the windows and doors so the smell doesn't permeate her belongings. The people of St. Mary's have long fought for the fish sludge to be cleaned up; after an aborted attempt in 2016, that work finally began in earnest this week.

There are 900,000 liters of fish sauce to remove—some of it just liquid, some of it with fish solids. The province of Newfoundland and Labrador budgeted $2 million for the job, which was taken up by a local engineering firm and is expected to be completed in October. As workers cut open the vats, viscous brown fish goo shot out onto the factory floor. It will be mixed with peat moss, hauled out in about 200 truckloads, and thrown in a landfill lined with polyethylene, after which it will be sealed up "like a Ziploc," according to the lead engineer on the project. St. Mary's Mayor Steve Ryan reportedly teared up while describing new hope. "To be known as the 'sauce plant mayor' … it's time for that to be over," he said.

Even in ideal circumstances, the creation of good fish sauce is a stinky process, but the resulting umami bomb is well worth it. The Atlantic Seafood Sauce Company began in 1990 with the (sound!) idea to make use of someone else's waste product. A processing plant in St. Mary's was sending female capelin to Japan so their eggs could be used for masago; the male capelin were of no use to them. Vietnamese immigrant Sahn Ngo saw in these castoff fish an opportunity to brew one of the signature sauces of his cuisine. But after his company was blocked by regulators—check out this Eater feature for nuance—it closed up shop in 2001, leaving behind the vats of sauce in progress. I can't be the only person out there staring at these torrents of fish mud and wondering about the tasting notes. In fact, I know I'm not, having spoken to food scientist Bryan Quoc Le earlier this week. Given his background in chemistry and Vietnamese cuisine, he was the ideal expert to unpack this horrifying and possibly delicious situation.

Our conversation has been edited and condensed for clarity.

I wanted to start off and take a more normal scenario and understand the chemistry in a more traditional fish sauce, where we start with the fish and the salt. My understanding is that enzymes from the gut of the fish do the work of digesting the proteins, and breaking them down into amino acids, which have tasty properties, and that's why we like the fish sauce. And the high salt content helps keep this process safe and makes sure that the right kinds of bacteria are able to outcompete the harmful kinds.

That's exactly right. Basically what's happening is in the intestines of the fish, the enzymes are being released, and the enzymes start to break down the proteins. First, they chop them up into fairly large chunks, but then over time they start to liquefy as you get down to the amino acid content. And most proteins are about 30 percent or 40 percent glutamic acid, and so a lot of the initial flavor profile is from that glutamic acid content that gives you the umami flavor.

The salt helps protect the fish. It protects it from the presence of pathogens. That's the really important part of that process. But it also protects it from bacteria and yeast that might cause some kind of rancidification that is undesirable, right? Then it serves as a sort of amplifier of the flavor, too. In order to have glutamic acid be experienced as umami, you also need sodium ions. Together, it's necessary—pure glutamic acid actually tastes like soap.

Fish sauce is typically made on something like a yearlong time frame. This is a very, very different situation, and I doubt there's any literature on long-term fish fermentation.

I would want to fly there and just see what happens.

That's how I feel.

There are studies to be done.

Definitely. This is an incredible resource. Based on your knowledge of the underlying chemistry, what else do you think is happening once the process extends for this long?

Within that one- to two-year time frame, generally speaking, a lot of the process is the enzymatic breakdown [that produces amino acids].

Now, there are certain yeasts and bacteria that are super slow, but they tend to work their magic on the same timescales as something like a wine or a whiskey. There are these 100-year-old soy sauces that are made in Korea, these monasteries that just have this continuous soy-sauce broth going on and on. Apparently it's supposed to be really good. I haven't tried it myself, but I would imagine something similar is happening, where you have these 20-year-long timeframes, and the initial strong fermentation action, because there is going to be some bacteria or yeast that lives on the skin of the fish, these are not necessarily sterile conditions.

This bacteria or yeast that exists can tolerate really high levels of salt. But because what salt really does, and why it prevents the growth of pathogens, is it inhibits the breakdown of a lot of enzymes. With these long timescales, the yeast is able to not only grow, but they'll die and then release the contents of their enzymes. So there's this ongoing forever process that's happening, as well as the more conventional chemistry that would occur.

An example: There's the reaction between amino acids and very minute amounts of sugars. That's why soy sauce is black, right? It's a chemical reaction that occurs called the Maillard reaction, and is accelerated when you heat or roast something. The baking of bread or roasting of coffee gives you those dark compounds. But if you give something a year or two, or 10, the darkening occurs so long as there's some kind of sugar in the vat. So that would be an interesting thing to see: What are the flavor profiles that form because of this Maillard reaction? Again, you can also see that in wine. The chemical reaction that changes the profile of the really rich, strong flavors that you see in 10-year-old or 20-year-old wines is because of this Maillard reaction that takes forever to occur at room temperature.

That's a good detail to point out, because this abandoned factory isn't climate-controlled in any way. It's just been exposed to the elements, and I imagine it's basically refrigerated for a huge chunk of the year, based on where it is. So this is probably very unusual fermentation playing out: every few months, going back in the fridge, and then getting warmed up again.

Exactly. You have that seasonality to it.

The locals in the town have reported scents so strong that they board up their house in the summer, they can't hang their laundry out. Do you know what compounds might be produced by fermentation of this time length?

It's primarily amines, a class of compounds that basically come from the breakdown of amino acids. So the amino acids—there's some kind of biological reaction with the bacteria. They sort of split off a carbon dioxide, that breaks off that acid portion of the amino acid. The amines themselves, generally speaking, humans are extremely sensitive to them because they're indications of death, right? A corpse, a dead animal or a dead fish, it's giving off amines, and a lot of them tend to be toxic to humans. Even if it's minuscule concentrations that are going into the air, people are going to sense it a lot more, especially if you're not used to it. The people who live in this island Phu Quoc in Vietnam, the fish sauce capital, I'm sure they can tolerate quite a high heavy dose versus, you know, if your cuisine doesn't really have it.

People are describing rancid odors, too. Is there something about the fat content of the fish?

Yeah, definitely. That's the same sort of process with the enzymes, but with the added bonus of oxygen. [The fat is] oxidizing, and then as it's breaking down, which is still a slow process, those molecules are really, really volatile. Aldehydes and ketones that form from the breakdown of fatty acids, that's gonna give you that smell. Again, humans are very sensitive to that. We don't tolerate it very well.

I've seen footage of the workers cutting open these vats, and it's not a pretty sight. It's this gray-brownish sludge hitting the factory floor, but obviously real fish sauce is processed and strained. Would you be a little bit tempted to taste this forbidden fish sauce if it was processed?

I would 100 percent taste the forbidden fish sauce. I would do it for science. My wife would kill me if I'm not already dead. I will. If there was a way, if I would be able to fly out there, I would do it, and write a whole report about it because I think that would be a pinnacle achievement for me.

For the record, I agree, and I don't even have the scientific pedigree. I just want to taste it. With the prospect of dying, what kind of pathogens would you be worried about specifically?

Honestly, I don't think there would be any pathogens, because the salt content is so high. There's really nothing that lives in that. There are a lot of products that are shelf stable as long as they have a high concentration of salt, decently acidified, or high concentration of sugar. Basically, the only thing that can live is a handful of bacteria and yeast. It's one of the principles of food safety.

So this is speculative, but given your knowledge of fish sauce, and just extending some of these chemical processes out more and more years, what would be your tasting notes, if you had to guess, for this fish sauce?

Oh my gosh, that rancidity. Unfortunately I think the rancidity and amines are going to be forward, in a way that is probably going to be overwhelming. That said, there's so many complexities with that. You have not only the reactions forming those molecules, but also the molecules are interacting with each other. So amines and aldehydes really like to form bonds, and in doing so, they might create a level of complexity that we've never seen before, right?

There could be an undercurrent of sweetness, even. That would be very bizarre, right? But if you think about the chemistry of sweeteners, artificial sweeteners, some of them have that kind of structure. So that would be interesting. It could be like where you're hit with a horrible bitterness, a rancidity, a horrible fishy overwhelming—but maybe there's something underneath, right? Or maybe we just need to acidify it a little bit. Add a little lemon juice. This is common in Vietnamese cuisine, when you have the fish sauce dressing that's used for a lot of noodle dishes. Maybe we could cut it.

[Both laughing, with hunger]

Maybe we can cut those amines and just not even taste them. That could be really cool. Then you would know: What level of savoriness could we actually achieve out of it? Because this is the fun part. Worcestershire sauce is fish sauce. Basically, what happened was a bunch of chemists accidentally abandoned it for a couple years, they put a barrel of anchovies or mackerel or something like that, and then just threw in some vinegar.

What could happen is, oh, actually this is kind of flavorful in a way that we wouldn't expect—something that's 20 years old, especially the umami flavor, the intensity of the glutamic acid. Maybe the glutamic acid from the amino acid breakdown is reacting to other components like these amines and these aldehydes, and now we have all sorts of new molecules. It's actually a black box. We don't really know.

Oh man, that's so interesting. We've got to get a hold of this somehow. They've got a million liters of it.

Right? If it turns out to be some kind of goldmine ...

I reached out to you because I love the piece you wrote about fish sauce, and I was wondering what you like most about it in its ordinary form.

Being Vietnamese, it's such a quintessential part of my cuisine. My mom would always cook with it. It was always incorporated in some way, but it was also kind of the situation where my friends would come over and be like, "What? What is that horrible smell?" You know you're in a Vietnamese household because you can smell the fish sauce, because it's always in the air. It has a childhood element of warmth; it helps me think about Mom a lot.

Then, as an adult, I use it, and now that I understand the chemistry, it makes a lot of sense. It's a very versatile ingredient because what you're doing is just dumping a lot of glutamic acid, but it's from a very specific flavor profile. It's not soy sauce, which tends to be have a caramelized element to it because of that long-term processing, and has more sugar because it requires rice to jump-start that fermentation process. Whereas fish sauce is pure, right? It's just fish, guts and everything. There's an element of authenticity to it. Not that, of course, soy sauce isn't really authentic, but I just like the idea of that.

There's that piece, and then one of the reasons why my wife and I started dating is because she's Israeli and Romanian, totally removed from my culture, and one time we went out on a date, and she took me to a Vietnamese restaurant, and I'm like, "How does this person even know anything about Vietnamese food?" I kind of wrote it off, like, She's not gonna understand. Of course, they bring out the food, and she's like, "Could I get more fish sauce?" My friends would stay away from my house. But this person who ended up becoming my wife, she's like, "I love fish sauce. I need more of it in my life."

I just wanted to do an internship in Europe so it would be easier to find a job after graduation. That internship completely changed my life.

My favourite view in Hamburg

In 2017, after I finished the third year of my Computer Science studies, I decided to do an internship in Europe. So while I was applying for the Erasmus Scholarship, I also started looking for internships. In the end I got the scholarship, which helped a lot with the visa process, and I also found an internship at a company in Hamburg.

I had never been outside of Turkey before. So I had also never really talked to people from other countries. What I knew about Germans came mostly from the Turkish people who moved to Germany in the 60s for work. Those people came back to Turkey almost every summer and told us about their life there. Most of what they said was the same as what you can read on the internet, the stereotypical German.

People who don't laugh, who are cold and unfriendly, who always follow the rules, who speak a language that sounds too harsh, and so on. And of course, someone always brings up the Nazis.

Apart from the people I talked to in the interviews, my first real contact was my German flatmate. Before I arrived he showed me the room on Skype and we agreed on it. When I landed in Hamburg, he picked me up from the airport. That was a great start, and during my three months there we chatted in the kitchen almost every evening for half an hour.

I had a lot of questions for him and he always answered them properly. On the weekends he picked up his girlfriend and they either stayed home or went rowing. Their relationship looked very healthy to me. She was studying medicine, so sometimes she was working at her desk on a Saturday while my flatmate played Witcher 3 right next to her. I never heard them argue about anything.

Near the end of my internship he told me that he and his girlfriend were going to Africa, so I would be alone in the flat for a week. When I asked him what I should do with the keys, he just said to drop them in the mailbox. That much trust after such a short time really surprised me.

We had a last dinner together and I told him things were going well at the company and there was a chance I could come back to Hamburg.

I was lucky with my flatmate and I was also lucky at work. I had a great team. They taught me new things, gave me responsibilities the other summer interns didn't get, and included me in all the team events. The offsite we had outside the city is something I still remember today. And I was getting all of this while earning minimum wage.

My first offsite event

I was really surprised by how friendly everyone was. We weren't just working, we were having fun. Everything I had heard about Germans until that point was gone. It was the best summer of my life.

My lead was happy with my work and offered me a job right away. I could work as a freelancer from Turkey while finishing my studies, and if they were happy with my performance I would get a full-time contract. That is what happened.

Coming back

In April 2018 I got my contract, even before I finished my studies. The next month something nice happened. My German flatmate emailed me and asked when I was coming back, because he already had a room for me for the summer. The company was going to give me a place to stay, but I couldn't say no to him. Right after my last exam, before I even had my diploma, I moved to Hamburg at the end of June and started working full time with the same team, living with the same flatmate.

On my first day there were candies and balloons on my desk. Everybody was happy to see me again. One colleague who knew my interest in Apple gave me a German flag pin he got at WWDC18. He said he tried to swap it for a Turkish flag but couldn't manage it. I was already happy with the gift, because it meant he was thinking about me while he was in California.

My desk

When you start living in Germany you have to register your address. When I went to the public office, the clerk told me I had already lived in Hamburg before. I said I had done a summer internship a year earlier. He just said: Welcome back!

That was the moment I felt like I was coming home, not moving to a new country.

The first years

My first months were mostly about finding a flat and figuring out how things work here. Since my German wasn't good enough, my colleagues came with me to flat viewings after work. When I had a problem with my electricity provider, a colleague came along and did all the talking. When I moved to a different place a year later, a German colleague rented a van and helped me carry everything.

At the time, the company was one of the biggest internet companies here, with around 2.000 people. Sometimes I saw the C-level in the kitchen getting their own coffee or even washing their own glasses. They had assistants, but the assistants helped with work instead of acting like servants. We also had a CFO who rode a very old bike to the office.

Whenever we had a team event or an offsite, the organisers made sure they knew everyone's dietary needs. Vegan, vegetarian, halal, no alcohol, allergies. So everyone would have something to eat and drink.

When I was doing a good job, my lead kept telling me to take vacation instead of working more. When I was sick, he told me to stay home until I felt better. I didn't even need a doctor's note. It took me a while to notice this was not only a work thing. You get on a bus or a train here and nobody checks your ticket. I wasn't used to this level of trust.

My second offsite

Across from our office there was an Italian restaurant where we went for pizza sometimes. The pizzas were very tasty and huge, and it was the only place that sold half a pizza. At lunch I saw people in suits eating there, and right at the next table people from the construction site nearby in their work clothes eating the same pizza. Nobody looked at the other table and nobody thought it was worth noticing except me. That is the part that stayed with me. It was completely normal, and it was normal because a man who spent his morning on a building site can afford an Italian lunch and eat it like everyone else.

I noticed the same thing in the rich districts of Hamburg. You can walk into one and not feel like you don't belong, and the prices in the shops are not that different from anywhere else in the city. Sometimes they are exactly the same. The more I saw of that, the more I wanted to stay. I believe in social democracy, and this is a country that calls itself one and mostly behaves like it.

It didn't take long for me to move up at work. I didn't feel any discrimination, even though people say Germans prefer Germans when it comes to promotions. There were already a lot of people with a migration background at the top. When I ran as a candidate in the works council election, I got the most votes of anyone, even though there were many Germans running and only around 25% of the company were immigrants like me.

You see the same thing outside of companies. Cem Özdemir, whose parents came from Turkey, has been the prime minister of Baden-Württemberg since May. Sinan Selen, who was born in Istanbul, runs the office for the protection of the constitution.

Being in the works council gave me more contact with the C-level. Sometimes I found myself arguing with the CTO or the CEO, and later at a company party we would have a beer together and talk about something else. That is when I understood how flat the management is here. I saw the difference later when I worked for an American company, where they made sure you knew you were lower on the ladder.

I met hundreds of Germans at that company and they were always super friendly. I say this because years later I still see many of my old colleagues at different events, and most of my friends today are people I used to work with. A lot of immigrants complain that it is hard to become friends with Germans, but that wasn't my experience.

My landing was soft

I should be honest about one thing here. I landed softly. I came into a big international company where everyone spoke English, in a city like Hamburg, with a salary that let me pick where I live. I have worked with other German companies since then, mostly smaller ones, so what I write here is not only about that one place. But the start was much easier for me than it is for most people. A lot of them arrive with none of that, and the country they meet is not the same one I met. Everything I write here is my experience, not a report on how it works for everyone.

The language is part of that. Every story I told above happened in English. The team, the flatmate, the works council, the arguments with the CEO, all of it. Once I decided I was going to stay here for a long time, I started taking German classes, and after the citizenship law changed I put much more effort into it. So the German came from the decision to stay. It was not the thing that made me stay.

I never really tried to integrate. It just happened that the way I think about life turned out to be similar to how people here think.

Why I like the rules

Most people complain about the rules in Germany, but I like them. It means someone, or a group of people, sat down and thought about that specific thing and decided on a standard for everyone. When you have to do something, you don't feel lost, because you can just look it up. You don't have to find someone who knows.

Rules also make normal days easier. Something simple like waiting at a red light even when there are no cars takes away the work of deciding whether to cross. And if everyone does it, you don't have to worry about someone suddenly stepping into the street. Or the fire safety rule that makes exit doors in public buildings open outwards. You never have to think about pull or push.

There are also unwritten rules, like being on time. For me the logical thing has always been to aim for 15 minutes before the agreed time, so if something happens on the way I'm still fine. After going on dates with a lot of Germans, I realised they do the same thing. So instead of me arriving early and waiting 15 minutes, we just meet 15 minutes earlier.

My favourite one is Ruhezeit, the quiet time between 10PM and 6AM, plus Sundays and public holidays. My sleep is very important to me and I'm a light sleeper, so this rule really helps me. Sometimes I had neighbours who had parties during those hours, and I didn't hesitate to complain (with Lärmprotokoll) to my landlord. Many Germans have told me I'm more German than many Germans. It is probably because of examples like this one.

The one time a rule was pointed at me instead of working for me was my first application for permanent residence. I had just left my first company and I was between jobs, and that was enough to get it rejected. That was when I learned what a sustainable livelihood means to the German authorities. It doesn't mean the money you have in your account. It means a permanent contract with the probation time already behind you. I had savings and it didn't count for anything.

It was an emotional day. But after I calmed down I could see their logic. A bank balance can be gone in a year and a signed contract that already survived six months says something a number can't. It was the same system I like for every other reason, just this time I was on the wrong side of it.

Understanding the country

The longer I lived here, the more I saw myself spending the rest of my life here. That made me want to read about the country. I'm a heavy reader and I track every book, so here are the ones I read about Germany:

• German Men Sit Down To Pee & Other Insights Into German Culture
• Germany: Memories of a Nation
• The German Genius
• Kaput: The End of the German Miracle
• Man's Search for Meaning
• Blood and Iron: The Rise and Fall of the German Empire 1871 to 1918
• Why the Germans Do it Better: Notes from a Grown-Up Country

Apart from the books, I also watched a lot of documentaries, especially about the Nazi era.

Like most people, I didn't know that Germany wasn't one country until the 19th century. It was made of small kingdoms, city states and so on. That is why there are so many kinds of bread, beer and sausage, why some parts have different public holidays, different accents and even a different way of living.

The German Genius surprised me the most. It is about 1.000 pages about Germans who contributed to humanity in different fields, and I didn't know how many things in our daily life exist thanks to them. Some of them had to escape the country because of the Nazis and reached their full potential somewhere else, in countries like the US and the UK. The books also helped me understand how Germany built an educated middle class, how workers' rights and social reforms happened, and where the rule about no shopping on Sundays comes from.

What stayed with me is that this is probably the only country in the western world that faced its own history properly, admitted the worst things it did, and built a lot of its identity around not repeating them. You feel that here in small ways, not only in museums. In January 2024, after it came out that far right politicians had talked about deporting people who already hold a German passport, 50.000 to 80.000 people filled the streets of my city in one afternoon.

"Hamburg steht auf!" demonstration

In eight years I have not had a single case of racism. I know that isn't true for everyone, and I know the far right keeps growing.

I don't think most of those votes are about hating foreigners. Some people believe the story they get every day, that the country is broken and that immigrants are the problem. You find that story almost everywhere. The rest are angry that certain problems here have not been solved for years by the existing parties, so they look for an alternative.

For me, Germans are people who give trust before it is earned. People who are friendly without being fake about it. People who make sure everyone is included, down to what is on the table. People who care about someone they will never meet. People who know exactly what happened here and don't look away from it.

Making it official

That is why I decided to become one of them.

The chance came when the traffic light coalition, which I think was the most progressive government this country has had in decades, changed the citizenship rules and brought the waiting time down to five years. I was eligible, so I applied.

People seem to think a German passport is handed out easily now. It isn't. You have to prove you paid into the social system for at least five years. You have to pass the citizenship test. You need German at B1 level. You need a sustainable livelihood while your application is running. And most importantly, you have to accept the democratic values of this country.

For me the process itself was smooth. I applied online by uploading a pile of PDFs. A week later I got a letter saying my application had arrived, with a few papers to sign, scan and upload back. Then the waiting started. In total it took around 14 months to get the email saying my application was successful and that I could come and collect my Einbürgerungsurkunde.

I picked it up this week, at an appointment that only took 15 minutes. I am German now.

I believe I have contributed something to this society in these eight years and I will keep doing it. And I will keep complaining about everything that needs to be better. Because that is what real Germans do.

Why Book Corners won't sync contributions back to OpenStreetMap

I wanted Book Corners to contribute newly submitted public bookcases back to OpenStreetMap. After researching the operational and community requirements, I decided not to implement it.

A feature that sounded obviously good

When I introduced Book Corners, I explained that much of its initial data came from OpenStreetMap. OSM gave the project a useful starting point, with thousands of public bookcases already mapped around the world.

Book Corners also accepts new libraries directly from its users. People can submit a location and a photo, and the contribution becomes public after moderation. It seemed only fair that, when one of those submissions was missing from OSM, Book Corners should be able to contribute it back.

The idea was not to create an uncontrolled background synchronisation process. The workflow I had in mind was deliberately cautious:

• The person submitting the library would explicitly allow the contribution.
• An administrator would review the library first.
• Book Corners would search OSM for possible duplicates.
• The administrator would preview the exact data being sent.
• Nothing would be written until the administrator confirmed it.

From a software-development perspective, this looked like a manageable integration: add consent, track contribution state, build a preview, authenticate with OSM, and create the new feature through its API.

The code was not the difficult part.

Contributing data is not just an API call

Once I started researching the implementation properly, I discovered that writing to the API was only a small part of the work.

Because the information would come from the Book Corners database, OSM could consider it an external-data import. Because software would prepare and submit the changes, it could also fall under the rules for script-assisted or automated edits, even though an administrator would review each individual library.

Following the conservative interpretation of the OSM Import Guidelines and the Automated Edits code of conduct would require more than a dedicated account and an OAuth token.

Before the first production contribution, I would need to:

• Create and maintain a dedicated OSM import account.
• Publish a detailed import plan on the OSM Wiki.
• Document the data source, licensing, field mapping, duplicate detection, software, quality checks, changeset policy, and rollback procedure.
• Open a proposal on the OSM Community Forum.
• Contact the relevant local communities affected by the contributions.
• Wait through the review period and resolve any concerns.
• Keep permanent links between the import account, plan, discussions, and changesets.
• Maintain a contact and opt-out route for future questions or complaints.

There are also important licensing questions. A user’s permission to send a library to OSM is not automatically the same as having a sufficiently clear right to release that factual information under terms compatible with OSM. The user-facing explanation and consent would need to cover that distinction, including confirming that the information was not copied from an incompatible source.

These requirements are not a one-time form to complete and forget. They create an ongoing responsibility around the account, the documented process, community feedback, failures, and potential reversions.

I understand why these rules exist

OpenStreetMap is a shared global database. A bad import can create thousands of duplicates, overwrite better local knowledge, or introduce errors that are difficult to remove once other people have edited the same objects.

From that perspective, requiring documentation, licensing clarity, duplicate handling, accountable operators, and community discussion is reasonable. The OSM community has to protect the quality of the map, and good intentions do not guarantee good data.

Book Corners itself benefited from that data quality. It would be hypocritical to expect OSM to accept changes from an external service without safeguards.

At the same time, the process has a real cost. It asks a small project to become not only an API client, but also the operator of a documented import programme. That may be appropriate for organisations importing large datasets, but it is a considerable commitment for a low-volume feature whose only purpose was to contribute a few carefully reviewed public bookcases back to the commons.

The operational work outweighs the value

Book Corners has a simple purpose: help people discover little free libraries and share new ones with others.

Operating an OSM contribution pipeline is not central to that purpose. It would add credentials, production safeguards, audit and reconciliation code, community processes, licensing work, and a long-term support obligation. Every part is individually defensible, but together they make this a much larger feature than I initially intended.

There is also an opportunity cost. Time spent operating this integration is time not spent improving library discovery, moderation, photos, accessibility, translations, or the mobile experience. Those improvements directly help Book Corners users and are much easier for a small project to sustain.

I started from the feeling that contributing back was simply the nice and fair thing to do. After investigating it, I no longer think goodwill alone is enough reason to accept an open-ended operational responsibility.

The decision

I have decided to suspend the OSM write-back implementation indefinitely.

Book Corners will continue to acknowledge OpenStreetMap as the source of records imported from it, and OSM-imported libraries will never be submitted back as if they were new. But libraries contributed directly to Book Corners will remain in Book Corners; the service will not automatically or manually create matching OSM objects.

Nothing currently writes from Book Corners to production OSM, so suspending the work does not require disabling or migrating an existing integration.

I may reconsider the decision if a genuinely lightweight workflow becomes available, or if the scale and value of Book Corners contributions eventually justify the process. Another possibility would be a user-driven workflow that opens a proposed feature in an existing OSM editor, although that would still need to be discussed with the community rather than treated as a way around the rules.

For now, the responsible choice is not to build and operate a feature I am not confident I can support properly.

Sometimes the right feature is no feature

It is easy to think of implementation decisions as purely technical: can the API do it, can the application authenticate, and can the code avoid duplicates?

This experience was a reminder that external integrations also come with organisational and social contracts. Sometimes those contracts are more expensive than the code. Discovering that before deployment is useful, even when the result is disappointing.

I still believe contributing data back to shared open projects is valuable. I also understand why OSM protects its database so carefully. But for Book Corners, at its current scale, the balance between benefit and responsibility does not work.

So this is one feature I am choosing not to ship.

Every 12 hours, a backup system must turn the entire state of a busy database into a consistent, encrypted snapshot, with no impact to production queries.

Such backups are crucially important and simultaneously something that most engineers would rather never have to think about.

Just make the backup work.

Our goal at PlanetScale is to make taking, scheduling, managing, and restoring Postgres and MySQL backups effortless.

Though this is what our customers experience from the outside, achieving this internally requires careful orchestration of cloud infrastructure and DBMS tooling.

It's especially interesting to look at backups for sharded databases, which requires spinning up backup-specific nodes, pulling data from object storage, and WAL replay, all with massive parallelism. These techniques allow for petabyte-scale databases to be backed up in hours, at rates over 50 GB/s.

Here we take a behind-the-curtain look at how to effectively back up a sharded database with massive parallelism.

The backup lifecycle

Here is an example of a Neki (sharded Postgres) database with 8 shards, happily handling hundreds of thousands of queries per second of production traffic.

If sharding is a new concept to you, check out our recent post Making 768 servers look like 1 on how it all works. The first step in taking a backup depends on whether this is the very first backup or if we've taken one previously. We'll start with the steady-state case, which assumes a prior healthy backup already captured and stored in Amazon S3 (or similar object storage in other clouds).

Since a sharded Postgres database is many individual primary Postgres servers working together, we use regular Postgres backups as the building block for large-scale backups with Neki.

There are three ways to take backups in Postgres, which we've written about in detail previously. The best of the three, and the one that Neki uses, is combining filesystem backups with the replay of archived Write-Ahead Log (WAL). To summarize, the steps for this are:

1. Begin a full backup of the on-disk Postgres database files at time T1
2. The backup completes at time T2; between T1 and T2, rows on disk may have been mutated
3. Replay the write-ahead log modifications between T1 and T2 to correct mutated data
4. Store the final result in a distinct storage location like Amazon S3

We could complete these steps directly on the primary, or perhaps one of the traffic-serving replicas. The problem is that this task utilizes a significant amount of IOPS and compute, especially for a large database. Our goal should be to minimize the impact a backup will have on production query serving.

Because of this, we take a different approach. We spin up a brand new set of EC2 instances, one per shard, to manage the backups on.

These new instances will be responsible for the majority of the work in the backup. Because we operate these sharded databases in clouds like AWS and GCP, dynamically spinning up tens or hundreds of instances for short time slices to complete backups is achievable. It adds a small amount of cost, but is worth it to minimize negative production impact.

Reusing old backups

The next step is restoring the most recent backup to each shard. Prior backups are stored in object storage. Here we will use Amazon S3 as the example, but the same applies to other clouds (like GCS in Google Cloud). These are streamed directly from here.

This approach requires temporary compute and transfers each shard's data out of and back into object storage. We accept that cost for two important reasons:

• Only recent WAL comes from the primary, minimizing production impact
• Every cycle proves the previous backup can be restored and replayed

Once all the copying completes, these 8 servers have the exact state of each of the 8 shards from the previous backup, 12 hours ago.

Replaying the WAL

We now must catch up each shard's old backup to match the present state of the database. This requires replaying all changes from the Postgres Write-Ahead Log between 12 hours ago and now.

A naïve approach would be to pull the WAL directly from the primary. This is problematic for several reasons:

1. This would have nontrivial production impact. Replaying 12 hours of WAL on a high-churn database could take tens of minutes, or even an hour+.
2. Because we don't want too much server storage consumed by WAL, we continuously archive it to S3. Thus, we likely don't even have the full past 12 hours of WAL resident on the primary node.

On PlanetScale, all Postgres databases use wal-g to continuously archive their write-ahead logs. A sharded Neki database is similar, except each shard has a distinct WAL archive stream.

What if we streamed from there instead?

This almost solves the problem. The remaining problem is that Postgres archives WAL only after a segment is complete. If write traffic doesn't fill a segment sooner, our five-minute archive_timeout setting forces a segment switch so it can be archived. This means the newest changes may not have reached S3 yet. Thus, we take a hybrid approach. The system uses S3 for the majority of the replay, then streams the last ~few minutes of changes directly from the primary. Ideally, this last step takes on the order of seconds, not minutes or hours.

Putting this step together, back in our sharded database:

Completing the cycle

When every node has caught up replication to time T, where T is the timestamp we will log for the backup time, we stop WAL replication, freezing the point-in-time of the backup. Time T is saved to ensure we know the precise time, down to the second, included in this backup. The full backup is now consistent, with no smeared data. The final step is to encrypt and send these backups off to a new S3 bucket for safe keeping.

When complete, the backup nodes have served their purpose, and are decommissioned.

The initial backup

The cycle just described assumed we had a good backup from 12 hours prior to start with. In the steady-state this is true, but not for a brand new database.

Within 12 hours of a new database creation, pg_basebackup is used to seed a backup node for each shard.

pg_basebackup is a built-in PostgreSQL client utility that takes a physical backup of the entire database cluster: the data directory, tablespaces, and configuration needed for recovery.

Why not upload the pg_basebackup output directly? We use it only to seed the temporary backup nodes. The durable backup is created with wal-g, keeping the format and restore process identical for initial and steady-state backups.

The steps for this first one are similar, but slightly different than the steady-state:

1. Spin up one new EC2 instance per shard.
2. Run pg_basebackup on each instance to copy data from its primary.
3. Configure each instance to replicate from its primary.
4. Replay WAL until every instance catches up.
5. Stop replication at time T.
6. Use wal-g to format, encrypt, and upload each backup to S3.

Once this exists, all future backups happen with the restore -> catch up -> save flow.

Need for speed

Part of the reason we do so much parallelism is due to the nature of sharding. Whether it's 4 shards or 400, since each shard contains its own Postgres primary, we may as well take a rock-solid system (regular Postgres backups) and do it over and over on each shard.

A side-effect of this is that backups are VERY fast, even for large databases.

For every backup, the data transfer steps are:

1. Restore old backup from S3
2. Catch up the backup using a hybrid of S3 + the primary
3. Send the new file back to S3

Consider what it would take to complete this backup cycle on an unsharded 32 terabyte database. Backups on PlanetScale are compressed and stored with encryption at rest in S3. We'll assume that backups for this 32 terabyte database compress down to 20 terabytes. Therefore, we have

1. 20,000 GB transfer from S3
2. Catch up (Ex: 20 GB, compressed to 10 GB)
3. 20,010 GB back to S3

This equates to a total transfer volume of ~40,030 GB. If our various nodes and interconnects can sustain a 500 MBps transfer rate, this means the backup would take ~22 hours. Crucially, this means that if we want twice-daily backups, multiple backups would overlap. This would prevent us from meeting our recovery point objective (RPO).

This same database in Neki, spread over 8 shards storing ~4 TB each, performs quite differently. In total, we need to transfer, catch up, and store the same 40,030 GB. But if we can do so across 8 distinct backup nodes in parallel, each capable of 500 MBps, we reduce the total time to ~2.8 hours.

This gets better the more shards you add. The same data spread across 32 shards would back up in 42 minutes. Backup speed scales well. 100 terabytes on 100 shards backs up at ~the same speed as 1 terabyte on a single shard.

What are backups used for?

One reason to take backups is data safety. In the case of accidental data deletion or a one-in-a-million database disaster, backups (and the WAL) are an essential fallback.

But at PlanetScale, backups are also a core part of everyday database operations. For Metal databases specifically (ones with Local NVMe), backups are also used every time the database is resized.

Resizing a Metal database requires:

1. For every existing node in the sharded database, spin up a brand-new EC2 instance at the new size. In a database with 8 shards each with a primary and 2 replicas, we may currently be running it on 8 x 3 = 24 i8g.xlarge nodes. While running, we initialize 24 additional i8g.2xlarges to double the compute capacity.
2. Each of the 24 new nodes pulls down a copy of the most recent backup from S3, and begins catching up to the present state of the database via the archived WAL.
3. All nodes are initialized as standbys of the original primary, and complete final replication catch-up.
4. When all nodes are synchronized with the primary, a switchover is made from the old i8g.xlarge primary to a selected new i8g.2xlarge primary.
5. The smaller nodes are decommissioned, leaving only the larger primary and replicas.

We've now completed a full resize across all shards. Backups are used to facilitate the creation / catch-up of the new nodes. This process can take anywhere from minutes to hours, depending on how large the backups are and how much WAL there is to replay.

Backups are also used when a node needs to be replaced due to unexpected failure. If your average server lifetime is 5 years, you'll rarely notice a failure on a small database with a single primary. For a database with hundreds of shards, each with multiple replicas, the statistical likelihood for node failure in a given week or month increases significantly. When we see an individual node fail (say, a single replica in a shard), the process for replacement looks like:

1. Initialize a new cloud instance of the same size as the previously failed one.
2. Use the process described earlier to restore a recent back-up to it, catch up the WAL.
3. Initialize as a follower of the original primary, synchronize data.
4. This new node can now actively serve read queries and/or serve as a new primary during a switchover or failover operation.

What about MySQL?

Everything here has been about how we complete sharded backups and resizes on Postgres databases, powered by Neki. PlanetScale also operates large-scale sharded MySQL databases, powered by Vitess as our query routing / sharding layer.

The process for backing up and resizing these databases is quite similar! We have a whole separate blog on how this works, but the main difference is that we use VTBackup instead of the Postgres backup builtins, we use the MySQL binary log replication instead of WAL, and replication catchup is done from the primary instead of a hybrid between S3 and primary.

Make it boring

Ultimately, we want all this to be transparent to you. Taking a backup should be as easy as an automated schedule, or clicking a button. Resizing a database should be a single click or API call.

However, we also have a deep appreciation for the finer details of database operations. By taking a journey through the lifecycle of backups, we hope you have a new-found appreciation for how incredible database orchestration is.

If you find this fascinating, we're always looking for talented database engineers.

Ask five people what DMARC does and you will get five answers: it stops phishing, it kills spam, it proves an email is safe. None of that is quite right. DMARC (the current spec is RFC 9989) checks one narrow thing: did the owner of the domain shown in the From line actually authorise this message, provable through SPF or DKIM?

It is a good question to answer. It is also a lot smaller than the reputation DMARC has built up. Reach p=reject thinking you are now phishing-proof and you will quietly drop the controls that handle everything DMARC never touched in the first place.

How email proves who sent it

Two building blocks sit underneath DMARC. SPF is a list a domain publishes of the servers allowed to send mail on its behalf; the receiver checks whether the mail actually arrived from one of them. DKIM adds a cryptographic signature to the message, which lets the receiver confirm it came from the signing domain and was not altered along the way. DMARC then pins both of those to the address you see in the From line.

Here is the part that trips people up. An email actually has two "from" addresses. There is the envelope address, which works like the address on a posted parcel: mail servers use it to route the message and then throw it away, so you never see it. And there is the visible From your mail app displays, the "Your Bank <alerts@your-bank.com>" you read at the top of the message. Nothing forces those two to match. That gap is the whole game: an attacker can show your bank in the visible From while the envelope quietly points at their own server.

SPF looks at the envelope address. DKIM's signature carries a domain of its own. DMARC's job is to take whichever of those actually authenticated and check it against the From line you can see, because that is the address a human trusts.

What these actually look like

All three live as text records in your domain's DNS, the same place your website's address is configured. You do not need to memorise the syntax; it helps to recognise the shape.

An SPF record lists who is allowed to send. This one authorises Google Workspace and a marketing tool, and says anything else should be treated as suspicious:

example.com.  TXT  "v=spf1 include:_spf.google.com include:sendgrid.net -all"

The include: entries pull in each provider's own list of servers, and -all means "if it is not on those lists, it is not us".

A DKIM record publishes the public half of the signing key, so receivers can check the signature on your mail. The long string is the key itself:

selector1._domainkey.example.com.  TXT  "v=DKIM1; k=rsa; p=MIGfMA0GCSqGSIb3DQEBAQ...AB"

Finally the DMARC record ties it together and tells receivers what to do when a message fails. This one asks them to reject failures and to send you reports:

_dmarc.example.com.  TXT  "v=DMARC1; p=reject; rua=mailto:reports@example.com"

p=reject is the strict setting: fail authentication and the mail gets turned away. rua= is simply where your aggregate reports land.

How a pass is actually decided

DMARC sits on top of SPF and DKIM, and it evaluates them independently. There are two separate ways a message can pass: SPF passes for the hidden envelope domain and that domain aligns with the visible From domain, or a DKIM signature validates and its signing domain aligns with the visible From domain. If either aligned path succeeds, DMARC passes. If neither does, it fails. "Aligns" simply means the two domains match closely enough to count as the same organisation.

flowchart TD
  A[Incoming email] --> S{SPF authenticates and aligns with From domain?}
  S -->|yes| P([DMARC pass])
  S -->|no| D{DKIM validates and aligns with From domain?}
  D -->|yes| P
  D -->|no| F([DMARC fail])

DMARC passes if either SPF or DKIM both authenticates and aligns with the visible From domain. It fails only when neither does.

A quick word on what "align" means, because it catches people out. DMARC has two modes. In relaxed mode (the default) the two domains only need to share the same organisational domain, so a DKIM signature from mail.example.com aligns with a From of example.com. In strict mode they must be identical, and that same signature would fail. If you have seen "SPF pass, DMARC fail" and wondered how both can be true at once, it normally means SPF confirmed the hidden envelope domain successfully, but that domain did not match the visible From address closely enough to align. Strict versus relaxed mode then decides whether closely related domains count as the same.

Notice what the test never looks at: the words in the message, the links, the attachments, whether the sender means you harm. It only asks where the mail came from.

Where DMARC helps

The case DMARC was built for is exact-domain spoofing. If someone places your-bank.com in the From address without producing an aligned SPF or DKIM result, a policy of p=reject asks participating receivers to reject the message, although the receiver retains final control over its disposition and may apply local policy or exceptions. This is real protection, and for this specific attack it is strong. It also gives you something you did not have before: aggregate reports (defined in RFC 9990) revealing many of the systems participating receivers have observed sending as your domain, which is how you find the forgotten marketing tool or misconfigured relay before an attacker does.

Where it falls short

Lookalike domains. An attacker registers your-bank-support.com, sets up valid SPF and DKIM, and passes DMARC on their own domain. Your policy has no reach over a domain you do not own. To every receiver, that mail is fully authenticated.

Display-name impersonation. The name shows "Your Bank Security"; the actual address behind it is alerts@some-unrelated-domain.com. DMARC only ever checks the domain. It has nothing to say about the friendly name, which is the bit most people actually read, so this sails straight through a passing check.

Compromised mailboxes. When an attacker signs into a real account through phished credentials and sends through the legitimate provider, the message will usually pass SPF, DKIM and DMARC because, in the protocol's terms, it was sent through authorised infrastructure. Authentication cannot tell a real user apart from an attacker controlling that user's account.

Authenticated but malicious domains. Anyone can register a domain and configure flawless authentication; senders of unwanted mail do this routinely. A pass on totally-legit-invoices.com confirms the owner authorised the mail. It says nothing about whether the owner is honest.

Spam and inbox placement. DMARC is not a spam filter and does not decide whether mail reaches the inbox. Filters may weigh it as one input, but authenticated spam is still spam, and placement is a separate system with its own logic.

Forwarding and mailing lists. Legitimate intermediaries can break authentication. Forwarding commonly breaks SPF because the forwarding server is not authorised by the original sender's domain, while mailing lists may modify the subject or body and invalidate DKIM. A legitimate message can therefore fail DMARC even though nobody is impersonating the sender. This is one reason enforcement should follow careful monitoring and remediation rather than a blind switch to p=reject.

Authentication is not trust

So a DMARC pass tells you one thing and one thing only: the domain in the From line authorised the mail, and SPF or DKIM proves it. Useful. It shuts down exact-domain spoofing and shows you who is sending as you. But whether the message is honest, whether it is safe to click, whether the account behind it has been hijacked, DMARC has no opinion on any of that. When a vendor tells you DMARC "stops phishing", that is the gap they are papering over.