Earlier this year Apple denied me access to App Store Connect after deciding that I matched someone on a U.S. government restricted-party list.

Their explanation was fairly definitive:

“The information you provided fully matches one or more restricted parties on the U.S. government consolidated screening list or another government’s sanctions list.”

They already had my passport.

I replied with my full legal name, Sean Joseph Byrne, uploaded my driver’s license, and pointed out the address on the government record they appeared to be matching me against. I’ve never lived at that address, never lived in County Sligo, and have no connection to the company involved. I asked them to escalate it to their sanctions compliance folks and make a proper non-match determination.

Apple still hasn’t replied.

Apple’s response after reviewing my identity information.

I knew what had happened because this wasn’t the first time.

Cloonmull House

Search the U.S. government’s Consolidated Screening List for Sean Byrne and you get exactly one result:

Sean Byrne
Cloonmull House
Drumcliffe, County Sligo
Ireland

Source: Entity List, Bureau of Industry and Security
Added: July 21, 2009
License requirement: All items subject to the EAR
License policy: Presumption of denial

The Consolidated Screening List isn’t itself a sanctions list. It’s a U.S. government screening tool that combines a number of export-control, sanctions and other restricted-party lists maintained by the Departments of Commerce, State and Treasury.

The result comes from the Commerce Department’s Bureau of Industry and Security Entity List. “All items subject to the EAR” means the Export Administration Regulations, the rules governing what U.S. companies can ship abroad. “Presumption of denial” is a licensing posture: if someone applies for a licence to export something to this person, the default answer is no. That’s the entire purpose. It’s an export-control instrument but it says nothing about who can be employed, or who can sell shares.

The person in the search result isn’t me. More interestingly, it doesn’t appear to be anyone.

The entry came out of the prosecution of an Irish aircraft-parts business called Mac Aviation. In 2009, the Department of Justice described Sean Byrne as Mac Aviation’s commercial manager and charged him alongside Thomas and Sean McGuinn over the illegal export of U.S. aircraft equipment to Iran.

Except Mac Aviation had apparently invented employees to make the company look bigger than it was.

Mac Aviation was a father and son working out of a cottage on the edge of Drumcliffe village, Ben Bulben behind it. A Rolls-Royce official who came to visit was reportedly speechless. The company he had been selling helicopter engines to, and had taken for a global operation employing hundreds of professionals, was a house in Sligo.

Drumcliffe, County Sligo, with Ben Bulben in the background.

To keep up the impression of a much larger firm, the McGuinns signed documents with false names. Sean Byrne was one of them. John Mooney reported in the Sunday Times that the name appeared on so much company paperwork that the American authorities “became convinced Byrne existed and tried to indict him.”

When DOJ filed a superseding indictment in 2010, replacing the original, Sean Byrne was no longer a defendant. The defendants were Mac Aviation and Thomas and Sean McGuinn.

More importantly, the superseding indictment repeatedly describes Sean Byrne as an alias used by one or more co-conspirators. The phrase appears more than fifteen times, attached to specific invoices, emails and an ownership statement. Mac Aviation staff used the name with suppliers in the U.S. and customers in Iran.

At some stage the U.S. government appears to have worked out that Sean Byrne wasn’t actually a separate person. And yet the entry in the Entity List survived. Sixteen years later it still has no date of birth, passport number, middle name or other useful personal identifier. It’s basically a common Irish name, an address in Sligo and Ireland.

Cloonmull House in Sligo was Thomas McGuinn’s home, and the indictment gives it as Mac Aviation’s registered mailing address. So the entry isn’t a record of a man in Sligo. It’s a name attached to somebody else’s house. And I’ve never lived in Sligo.

This has happened before

Years before I moved back to Ireland, I was selling stock through a tender offer when Nasdaq stopped my order after a background check returned a match on my name.

Their Head of Account Management emailed me saying that the check had found a match associated with a previous incident and that he was confident it was a false positive, but compliance wanted additional proof of my California address. On the phone he gave me more detail and specifically asked me about Mac Aviation. I explained that I’d never had anything to do with the company, provided the extra documentation they wanted and the sale went through with an entertaining story to tell people.

Nasdaq’s response after a background check matched my name.

More recently I ordered a Starlink mounting pole from California. DHL, shipping on behalf of SpaceX, told me the problem was a restricted-party match and asked for my passport. I sent it, they cleared it and the pole arrived. DHL also wouldn’t send a hat I’d ordered in the U.S. on to me in Ireland without a copy of my passport.

The fake Sean Byrne was associated with attempts to procure helicopter engines, fighter-aircraft parts and other U.S. equipment for Iran. The real Sean Byrne occasionally needs to produce a passport before someone will send him a hat.

Apple is the odd one out. Nasdaq and the shippers both generated false positives, asked for enough information to resolve them, and then resolved them. Apple already had my passport, received my driver’s license and a fairly detailed explanation of exactly why the match was wrong, and still told me that I “fully” matched a restricted party.

Twelve men named Robert Johnson

None of this is novel. In October 2006, 60 Minutes found twelve American men named Robert Johnson who all had trouble boarding flights, and brought them to New York together. A politician, a soccer coach, businessmen and a serving member of the military.

The Robert Johnson they kept being confused with wasn’t a man named Robert Johnson. It was a known alias of someone convicted of plotting to bomb a Hindu temple and a cinema in Toronto, who by then had served his sentence and been deported to Trinidad. The airline agents checking the twelve real ones against it had a name and nothing else. Not even a date of birth.

Asked about it, the head of the FBI’s Terrorist Screening Center said Robert Johnson would never get off the list, and that anyone with the name would be inconvenienced every time they tried to check in.

I’m not on the Entity List. I’m being misidentified as an entry on it. Apple didn’t make that distinction.

The consumer version of this has been litigated. In 2005 Sandra Cortez was held up buying a car in Colorado because TransUnion matched her against a woman on the Treasury Department’s sanctions list who was born 27 years after her. The credit bureau had compared first and last names only, not dates of birth. A jury awarded her damages and the Third Circuit upheld it, describing the failure to compare birth dates as reprehensible. Sergio Ramirez had the same experience at a car dealership six years later, and his case reached the Supreme Court in 2021.

In both cases the courts called for better matching. Compare the date of birth. Compare the middle name.

There is no version of that available to me. The listing has no date of birth to compare. No middle name and no passport number, because the person doesn’t exist. A screening system that does its job perfectly will still flag me, forever, on the only two facts the record contains: a common Irish name and a country.

Which is why arguing with companies one at a time is the wrong approach.

The real fake Sean Byrne

Remote hiring has developed a serious identity-fraud problem. It has also developed, somewhat unbelievably, a North Korea problem.

North Korean IT workers have been getting remote jobs at U.S. companies using stolen or fabricated identities, proxy interviewers and U.S.-based “laptop farms” that make workers overseas appear to be connecting from inside the United States. The FBI has been warning companies about it, and the DOJ has prosecuted schemes that successfully placed workers at more than 100 U.S. companies.

So if you’re hiring remote engineers, “is this person actually who they claim to be?” is now a legitimate security problem.

A new class of recruiting products is being built around that problem. They sit inside the software companies use to manage job applications, the applicant tracking system or ATS.

Tofu is one of them. Their pitch is that they screen every applicant across more than forty signals before a recruiter opens a résumé, and that when a screened applicant triggers a sanctions match the signal routes straight to compliance review. They are explicit that this has to happen early: screening at the background-check stage is, on their account, already too late, so it should run at application submission before any recruiter makes contact. They also say a candidate flagged by one of their customers is flagged across their whole customer network through their API. Brainner makes a similar case, checking applicants against 3.5 billion data points and flagging high-risk profiles before a recruiter reviews them.

Tofu says its database is built from analysed applicant profiles. Their homepage says more than 18 million. Most of their other pages say more than 5 million.

The sanctions screening these vendors describe is OFAC and the Specially Designated Nationals list, which is the right list for the risk they’re selling against: paying wages to a sanctioned person. I’ve no evidence that either company queries the BIS Entity List, and no idea whether any company I’ve applied to uses either product.

But screening my name against U.S. restricted-party data produces a false positive. It did at Nasdaq, at SpaceX, at DHL and at Apple. Four times in six years. And the industry’s answer to remote-hiring fraud is to run that class of check earlier, before a human is involved, and propagate the result across a network of employers.

Mac Aviation fabricated an employee to make itself look like a bigger company. That fabricated employee ended up in an authoritative U.S. government database. Sixteen years later, companies are building systems using that database to detect fake applicants, fraudsters, criminals and state-backed actors before they get through the hiring process.

Has this cost me a job? I don’t know. I’ve spent my career in information security, much of it in the U.S., and I’m now applying for roles from Ireland. There have been jobs where I’ve a background that should at least get a conversation and I’ve heard nothing. That’s hardly remarkable on its own. Hiring is messy, roles get frozen, recruiters disappear and companies reject perfectly good candidates for reasons the candidate will never know. There is an entire website called Did They Ghost You?, so we’re not dealing with an unexplained phenomenon.

But Nasdaq told me it was Mac Aviation. DHL shipping on behalf of SpaceX asked for a passport and told me it was due to a hit against the restricted parties on the U.S. government consolidated screening list. Apple at least told me I’d matched something, even if it then stopped talking. An applicant tracking system will tell me nothing at all.

Upstream

I asked the Bureau of Industry and Security’s End-User Review Committee to review the original Entity List entry. I’m not sure anything will come of it. The normal process is designed for a listed person asking to be removed, which creates an interesting problem here. I’m not the listed Sean Byrne, and the available evidence suggests that person may never have existed.

For now the U.S. government’s screening data still says Sean Byrne, Cloonmull House, Drumcliffe, County Sligo.

I’ve still never lived in Sligo.

LymeAlert will hit the market in August. The latest tool against the tick-borne illness could become a medicine cabinet staple

tick on a hand

Of the estimated 899 species of ticks worldwide, about 90 are found in the continental U.S.; however, only four species are responsible for most of the infections found in the U.S. today. Tomasz Klejdysz/Getty Images

Research shows that more than 31 million Americans—or nearly one in ten people—experience a tick bite every year. While some tick bites can be harmless, these parasitic arachnids that feed on both animals and people can transmit dozens of diseases in as little as 36 to 48 hours, including anaplasmosis, babesiosis, Rocky Mountain spotted fever, alpha-gal syndrome and Lyme disease.

This spring, the Centers for Disease Control and Prevention (CDC) reported a spike in tick-borne-illness-related emergency room visits, with Lyme disease being the most common diagnosis. An estimated 476,000 patients are treated each year for Lyme disease, concentrated in the Northeast, Mid-Atlantic and Upper Midwest, thanks to the prevalence of its carrier: blacklegged ticks.

Erin Dawicki is no stranger to the increase in tick-borne illnesses, especially in New England where she resides. For the past few years, the pediatric physician associate in Boston has experienced a steady uptick in kids showing up in her office with “huge swollen joints and zero trauma,” she says—symptoms that signal Lyme arthritis, a condition that develops anywhere from one to a few months after an untreated tick bite.

Lyme disease is easiest to treat during its early, or acute, stage, which makes quick tick identification critical for treatment. Most experience a “bull’s-eye” rash at the site of the bite, along with other symptoms, including fever, chills, headache, fatigue, muscle and joint aches, swollen lymph nodes, and neck stiffness. While the most common treatment is a 10- to 14-day course of antibiotics, the CDC also recommends a single 200-milligram dose of doxycycline within a 72-hour prophylactic window, if the tick had been attached for 36 hours, in areas where Lyme disease reports are high.

Unfortunately, even treated, some people experience post-treatment Lyme disease syndrome, previously known as “chronic Lyme disease.” This diagnosis comes with persistent fatigue, musculoskeletal pain or cognitive difficulties months after completing the appropriate antibiotic treatment. According to the CDC, time, and not additional antibiotics, is needed to overcome the syndrome.

Because early treatment is critical for recovery, Dawicki’s mode of course is to complete a blood test while also prescribing antibiotics before even obtaining an official Lyme disease diagnosis. It bothers her that she has to “blanket prescribe because the risk of Lyme is so high,” she says, thus contributing to the growing antibiotic resistance health crisis within the United States.

Dawicki’s “aha moment” came during a health care entrepreneurship course she took as a MIT Sloan fellow during her MBA program in 2024. Students selected one of three innovation tracks—Lyme disease, pharmaceuticals or Parkinson’s disease—to develop and pitch an original idea. From there, classmates formed collaborative teams to advance the strongest concepts.

“I joined the class late and was assigned Lyme disease,” explains Dawicki. “Initially, my plan was to just check the box on the course, but the morning the pitches were due, an idea popped into my head.”

So often her patients call her to report they have found a tick on their child. They’re worried and want to know the best course of action, but they can’t afford to come in for an office visit. As a medical professional, Dawicki understands the variables in health care accessibility, having patients concerned about the cost of care or taking time off work for an appointment, especially when the tick in question may not even be disease-carrying.

“So, I thought, wouldn’t it be cool if they could just test the tick at home, and then they know immediately if it’s Lyme-infected?” Dawicki says.

Fun fact: The U.S. National Tick Collection is the largest continuously curated collection of ticks in the world

• More than one million tick specimens are housed at Georgia Southern University in Statesboro, Georgia. The collection, on loan from the Smithsonian's National Museum of Natural History, is an important resource for researchers studying the transmission of tick-borne illnesses.

Why the uptick in ticks?

Of the estimated 899 species of ticks worldwide, about 90 are found in the continental U.S.; however, only four species are responsible for most of the infections found in the U.S. today. Found across the Northeast, Mid-Atlantic, Upper Midwest and Southeast, the blacklegged tick, Ixodes scapularis, is the most widespread of the tick species. It carries Lyme disease, anaplasmosis, babesiosis, Powassan virus and Borrelia miyamotoi disease. The western blacklegged tick, Ixodes pacificus, is found along the Pacific Coast, congregating mainly in California, Oregon and Washington, and it also carries Lyme disease and anaplasmosis. The lone star tick, Amblyomma americanum, originated in the Southeast but has recently spread across the Midwest, Mid-Atlantic and Northeast, carrying ehrlichiosis, Heartland virus, STARI and alpha-gal syndrome. Lastly, the American dog tick, Dermacentor variabilis, in the eastern U.S. as well as parts of the West, can carry Rocky Mountain spotted fever and tularemia. While other prevalent species include the Gulf Coast tick, Rocky Mountain wood tick and brown dog tick, they account for fewer human infections than the main four.

“Thirty years ago, there were only two reports ever of the Ixodes ticks that can transmit Lyme disease, and those were both on one coyote in Kentucky,” says Brian Stevenson, a microbiologist at the University of Kentucky’s College of Medicine. He studies the Lyme disease bacterium Borrelia burgdorferi and how it infects hosts and transitions between ticks and mammals. “Today, there are many, many other places where the range of the tick is expanding; therefore, the incidence or potential for Lyme disease is also expanding.”

This expansion of ticks’ geographic range is due to warmer winters, lacking the historical deep freeze needed to keep ticks in check, and growing populations of white-tailed deer and white-footed mouse populations, the parasites’ preferred hosts.

“They don’t fly or move that fast,” says Erika Machtinger, an entomologist at Pennsylvania State University with expertise on tick ecology and control. “Anything that changes in the local ecology or broad regional ecology has an impact on the tick.”

She points to deforestation, reforestation and urban sprawl, all of which fragment natural habitat and change how wildlife and people navigate their environments. Those factors, along with the elimination of natural predators like wolves and mountain lions that previously kept deer numbers in check, mean ticks are out of control.

“There’s a range of things that have happened, and it’s kind of created a perfect storm,” says Machtinger. “I do anticipate it’s going to get worse before it gets better.”

The CDC has reported an increase in tick-borne diseases in recent decades, with the number of documented cases doubling or increasing by an even greater magnitude for anaplasmosis/ehrlichiosis, babesiosis, Lyme disease, Powassan virus disease and spotted fever rickettsioses; cases are concentrated in the Upper Midwest and Northeast, along with parts of the South, Southeast and Ohio Valley.

While cases of pathogens and disease are easier to track, according to Machtinger, that doesn’t necessarily represent the whole picture. Instead, scientists look for long-term patterns and evidence of ticks “popping up in places and in numbers from surveillance that we haven’t seen before,” she says.

“We don’t have any numbers on tick populations,” adds Machtinger. “We can get relative estimates in specific locations if surveillance is done over a period of years, but even then, it is not super accurate because tick collection can differ by time of day, day, week of the season, season and year.”

The other issue is that tick research is only getting started. Stevenson, who is currently leading two studies aimed at fighting Lyme disease by turning the bacteria’s own internal systems against itself, notes that not all ticks are even infected with the same bacteria or the same level of bacteria.

“For a tick to become infected, it needs to feed during its previous life stage,” Stevenson says. “When they hatch, they come out as larvae, and the larvae are not infected when they hatch, but if they feed on an infected mouse, say with Borrelia burgdorferi [the bacteria that causes Lyme], those larvae can acquire the infection.”

Once they molt into nymphs, those nymphs can transmit the bacteria. “But it’s also possible that a larva fed on a mouse that wasn’t infected with Borrelia burgdorferi, which means the nymph doesn’t transmit the bacteria because it was never exposed to it,” he continues. “That’s why not every tick is going to be infectious. It really depends upon what they’re feeding on and if that host was infected.”

One way to stay tick-safe is to learn how to recognize the different kinds of ticks, in order to understand what they may be carrying.

“If the tick is not attached, which means it’s not feeding on you, therefore it’s not transmitting anything, whether or not that tick is infected makes absolutely no difference for your health,” says Stevenson. “If you have an embedded tick where there’s a chance of infection, then go to your doctor and get a dose of antibiotics as a prophylactic, just to be on the safe side.”

Tick research has also lagged behind the research of other vectors tied to human pathogen transmission, like mosquitoes, according to Machtinger. Most research into tick ecology and disease only began back in the 1980s, when Lyme disease was first discovered by medical entomologist Wilhelm Burgdorfer. Mosquito vector research, for comparison, began in 1897. Burgdorfer made the connection that the disease wasn’t viral but rather vector-based when he identified spirochetes as the disease’s causative agent in 1981, seven years after Lyme disease was first discovered in Connecticut.

“There’s been a renewed interest in earlier and better diagnostics for people,” notes Machtinger. “Vaccines, preventative medicines … instead of trying to target a pathogen, targeting the tick itself, but there’s still a recognized need for more.”

Tick control is difficult in that there aren’t tick control districts where crews spray widely to eliminate them as they can with mosquitoes. That’s because of the way tick ecology works: Their location is tied to where their hosts are. They don’t jump, fly or travel far on their own, and they live where the animals they feed on live. It’s not like targeting a mosquito-filled pond or lake. Machtinger argues that additional funding sources are needed from private industry, foundations or the government to address key issues like landscape control, through brush removal and regular mowing, for tick prevention.

“Ticks are very complicated ecologically, so we haven’t seen the same broad-scale management like we’ve seen with mosquitos. Most tick management is left to the individual,” says Machtinger, noting companies have developed products typically sold directly to consumers or pest management companies. “Broadly, it’s left up to people to decide how to protect themselves, their pets and their properties.”

Lyme disease detection in your medicine cabinet

This August, the first at-home screening tool to determine whether the tick that bit you carries Lyme disease will hit the market thanks to Dawicki and her co-founders Michelle Ewy and Brenda Ong. LymeAlert, which will sell for about $50, is designed to detect the presence of Borrelia burgdorferi, the pathogen known to cause Lyme disease, in a tick.

“LymeAlert can take heat-killed Borrelia and detect it down to really, really small quantities of the bacteria that’s present,” explains Dawicki. “Once we figured out how to do this, it almost felt like a moral obligation. We have the ability to get this out into the world, and it can become a pretty affordable way to try and prevent as many of these cases of long-term Lyme as we can.”

The First At-Home Test for Infected Ticks Could Improve Lyme Disease Diagnosis

LymeAlert is designed to detect the presence of Borrelia burgdorferi, the pathogen known to cause Lyme disease, in a tick. LymeAlert

The test kit, which remains effective for up to 12 months, is simple to use. After carefully removing the tick with either tweezers or a tick-removal tool, you place it into the kit’s “Tick Crusher.” The grinder pulverizes the tick’s chitinous exterior, exposing the internal contents where the Borrelia burgdorferi is located. A patent-pending buffer solution readies the sample for testing. The user then inserts a test strip into a slot in the grinder. The strip’s chemically treated nitrocellulose paper functions as an immunochromatographic assay that can detect the presence of Borrelia burgdorferi.

“It’s similar to a pregnancy test or a Covid test, and takes about 15 minutes,” says Dawicki. Two lines indicates a positive result; one line is negative.

Along with the test kit, Dawicki and team are also rolling out an app that will make the test results easily accessible to everyone—even those who may be neurodivergent or have vision difficulties that could make reading the test results challenging.

“You take a picture of the test strip, and the app will read and interpret the test strip for you,” says Dawicki. “We also give people the option of choosing from a list of telehealth providers, so they can connect directly to a health care provider if they don’t have access to their own.”

Pre-existing tests have their limitations. Mail-in tests, for example, allow users to identify the species of a tick and test that specific tick for pathogens that may cause disease, but results could be delayed, depending on postage speed. “Even the common blood test we use, if you do it within 30 days of the tick bite, still misses 64 percent to 78 percent of early-stage cases because the patient hasn’t developed enough antibodies to trigger a positive test,” says Dawicki.

LymeAlert allows for a quick at-home result.

“Particularly in the New England area, people understand why you would want to test the tick, because mailing ticks to labs has become more popular,” says Dawicki. “The problem we’re trying to solve with that is when you mail a tick to a lab, the results come back after that 72-hour prophylactic window for treatment as the CDC recommends, so we’re trying to bridge that gap.”

Her team is currently completing field pilots with veterinarians in Massachusetts, based upon the higher urgency around testing tick bites for dog and horse owners. The information collected during these pilots will be used for their hot-spot maps, which will be available through the LymeAlert app.

“Any data contributed by academic institutions or through taxpayer-funded initiatives will be provided back to the public through the hot-spot maps for free, so that you can look at the map on our app and have a quick, high-level view of if you’re in a [tick] hot-spot area,” says Dawicki.

The goal is to release LymeAlert to the public this August, with several thousand preorders of the product already on deck to help with manufacturing costs. They are also working with three REI locations in the Boston metro area and New Hampshire and several independent pet retailers to hold retail pilots.

Is this the answer to tick-borne-illness prevention?

The CDC doesn’t recommend testing ticks removed from people or animals, warning that false negatives and false positives can occur. “If you get a tick that has tested negative, you’re going to feel safe, and if you get a tick that has tested positive, you’re going to feel like you’re infected,” says Machtinger. Instead, its standard recommendation is to rely on physicians to assist with the diagnosis of anything that may be tick-borne.

“I think that [LymeAlert] is a tool that needs to be used in context with user understanding of the benefits and the limitations of that product,” adds Machtinger.

Stevenson believes the at-home test concept is good in theory. “The test is, if you find a tick, does it have evidence of Borrelia burgdorferi in it?” he says. But, in practice, he adds, “They’re not actually determining: Does a person have Lyme disease?” Only about 1 to 5 percent of tick bites actually result in Lyme disease.

Regardless, using an at-home test like LymeAlert could provide peace of mind to those who live in tick-infested areas. Further, knowing the test is available could also act as a trigger for better environmental awareness.

“It could potentially aid in surveillance if we’re doing these home tests like this,” says Machtinger. “And it does offer information to share with a health care provider if you bring in the test and say it tested positive.”

I am often asked to explain my distaste for RISC-V and I often find myself explaining it piecewise. The reactions are often of the form "you just do not understand the brilliance of it all", which is, of course, no argument at all. After being asked for the Nth time to explain, I decided to put it all down in one place so that I could simply link to it when asked next. Plus, if anyone wishes, then, to form a coherent counter-argument, they could refer to my points clearly and in detail by having this text as a reference. All opinions stated here are mine and do not represent the views of my employer, any deity, or my landlord. My cats concurred in part and dissented in part and will publish their opinion later.

Everything for Everyone

RISC-V will own the cheap-as-dirt single-use micro­controller space even­tually. Not due to its ISA design, but despite it.

The first and simplest-to-grasp issue is that one cannot be best for all use cases. RISC-V fans would have you believe that RISC-V will soon own all supercomputers, while also owning all the tiny microcontroller use cases, and all things in between. This is impossible, and would be equally impossible for any ISA. Simply put, the things a high-end CPU needs are diametrically opposed to the things a small cost-saving microcontroller core needs. The design choices are not merely microarchitectural, but actually (and necessarily) impact the CPU architecture itself. For what it is worth, I am 100% sure that RISC-V will own the cheap-as-dirt single-use microcontroller space eventually. Not due to its ISA design, but despite it. It will take this role from 8051 by being an improvement on it -- a bar so low, it is but a speed bump.

What does a cheap microcontroller core need? Let's inspect what they are used for. Typical use cases are to interface with and quickly reconfigure hardware blocks in a larger chip, eg in an MP3 player, an SD card, or a USB stick. The hard work is done by custom IP and the CPU core is just there to occasionally prod a register or configure something. What matters in this case is interrupt latency (lower is better) and size (smaller is better). Usually you would not expect much math to be done on such a core. Mass-produced cost-reduced devices would have the code running out of real ROM (if non-updateable) or RAM (if updateable); NOR flash costs too much and is not an option for really-mass-produced things. When running out of ROM, code size matters because ROMs are not very compact. When running out of RAM, code size matters because SRAMs also take up a lot of space on the die. Thus, code density matters for these use cases. Since much math is not expected, things like hardware dividers (or even multipliers) can be discarded. Privilege separation is also not needed in such single-use situations -- no external untrusted code is expected to ever be fetched. "But, " you might say, "you just described RV32IC (or RV32EC)!"

So, at basically the only purpose such an embed­ded core has, RISC-V is notably worse than the leading existing competi­tor.

Indeed, it is somewhat close, except really you need RV32I_Zicsr to claim that. Without Zicsr, there is no spec-compliant way to handle interrupts, as there is no temporary place to stash a register to allow you to stash the rest of them. MIPS reserved two kegs for this ($k0 and $k1). Without them, RISC-V needs mscratch/sscratch. Without Zicsr, you do not have those and are stuck with weird other methods to do things. And thus we are back in 8051 territory - it specializes in doing things weirdly. Small embedded cores are not out-of-order monsters. If you get one instruction per cycle out of them, you consider yourself lucky. Given this, let's optimistically count the number of cycles needed for an interrupt handler to stash ABI-required regs and call a handler written in C. First we'll use a CSRRW to stash a reg (let's say t0 for ease of explanation) and get a base address of where we may stash the rest. Then we'll need to stash ra, sp, gp, tp, t1-t6 and a0-a7. We'll then need to use another CSSRW to get back the old t0 value and stash that as well. That's at least 21 cycles. On the way out, the math is similar: one CSRRW to read the address of the stashed regs, and 19 loads to load them. That's at least 20 cycles. But that is not all. Since this needs to be done in assembly, we'll need to actually account for the JAL to our C handler and a RET from there. We'll graciously assume those are each two cycles. Thus each interrupt has at least a 44-cycle cost before any work is done in the C handler. Cortex-M0 (the competing cheap 32-bit core) does an interrupt entry in 15 cycles, exit in 12 cycles, and since it pushes the ABI-clobbered regs in hardware, the handler is written in C directly. Thus each interrupt here has only a 27-cycle cost. Oof... that’s a lot faster! You might protest that I am being unfair by not considering RV32E here. By having half as many regs, it can do the initial push 6 cycles faster and the pop as well, bringing its interrupt overhead to 38 cycles. Still over a third more than the Cortex-M0. Oof... So, at basically the only purpose such an embedded core has, RISC-V is notably worse than the leading existing competitor. The existence of CLIC and various proprietary "fast IRQ" / auto-stacking extensions is an additional indictment. The base ISA forces vendors to invent non-standard silicon to reach parity with a decade-old Cortex-M0. That, in turn, further fragments the "standard" (if it can so be called). Hilariously, even with the compressed extension, the typical IRQ prologue is larger and slower than the Cortex-M0’s zero-byte hardware path.

Now, about those compressed instructions. Let us look at them in detail. They are hilariously poorly designed. Say you want to store a byte to a register plus offset. What range of offsets can a 16-bit instruction encode? Zero through three. Not thirty three, not three hundred and three. Three! Well, maybe it is better for storing a halfword? Nope... zero or two. What even? Why? At least when you store a word, you get a sane range of zero through 124 bytes, but what is going on there with those other ones? Worse, the instruction for storing a halfword is encoded similarly to the one storing a byte, but somehow it has fewer options for offsets? Why? Well, one of the bits that store-byte uses for offset is just hardwired to zero... it could have been used to expand the range to at least go to 6, but it doesn't! By comparison, Cortex-M0 is happy to let you use offsets from zero to 31 for bytes, zero to 62 for halfwords, and zero to 124 for words - clearly this covers a lot more use cases. So what happened here? Truly, I do not know, but it is indeed hard to justify. A typical refrain is to just use full-length instructions for these larger offsets. Sure, but density will suffer - the very density that RISC-V fans were bragging about so recently when trumpeting the C extension. But wait, there is more yet. Those instructions to store a byte and a halfword are not even in the C extension. They are in another one called Zcb so you may not get access to them at all, even if their puny range were good enough to use in your situation. We’ll get to "extensions" later...

What do server cores need? Raw throughput. Here, we are in the world of out-of-order cores where silicon is more or less free, since no matter how big your core is, the caches will dwarf it in size. Modern out-of-order cores are decoding eight or sometimes ten instructions at once, and issuing them to multiple ports concurrently; many modern cores can take more than one branch in the same cycle (think about that for a second, let it sink in ... yes). Code density is really not as much a concern here as it was in the past. It matters, but making a slightly larger L1i is not terribly complicated and, again, Si area is more or less free on the scale of such small things. What you really want is the ability to fetch and decode as many instructions at once as easily as possible. While doing that, it also helps if the instructions tell you as much about their intent as possible, to allow you to merge them with others or split them up into pieces most efficiently. Seemingly, these two desires are at odds with each other, and to some extent it is true. "Easy decoding" is, as is widely known, latin for "fixed length" while "as much as possible" is greek for "long". Obviously we do not want fixed-length very long instructions. Where do we draw the line? Having instructions be a power-of-two in length makes many other things like alignment easier, so then what? Two bytes is too short. Eight bytes is too long. The answer is fixed length 4-byte instructions are a nice middle ground. That provides enough encoding space to encode almost anything you’d want, namely: 3 registers encoded in each instr, long offsets for branches. Why does this sound familiar? Because that is what aarch64 (and A32) have proven to work exceptionally well. MIPS made the same choice for the same reason.

You might now protest that ARM also has Thumb and MIPS has microMIPS. However, in high-performance compute Thumb is dead. When Apple was designing aarch64 with ARM, much modeling and testing showed it to be a net loss for instructions per watt and instructions per second. MIPS would surely have killed microMIPS too, had MIPS lived long enough to reach the current cost-per-transistor regime. The main upshot is that compressed instructions have no business in large cores, they get in the way of fast parallel decoding of many instructions by making it slower to find boundaries. You might protest that "RISC-V makes it easy to find instr lengths", but "easy" is not the same as "instant and free" that fixed-length instructions grant you.

It took them TWO YEARS to realize that arrays exist!

Getting back to our discussion of the raw performance that high-end cores need to demonstrate. What is one of the most common operations performed by any code? Array access. This is why x86 has addressing modes of the form [ebx + esi * 4] and ARM has [R0, R1, LSL #2]. Without it, you are forced, like an idiot, to shift a register left by two, then add it to another register, and only then use that to access memory. Three instructions for a single array access. The usual excuse given for this inexcusable lack of foresight is that "instruction fusion will fuse all those three instructions into one in fast cores". Yeah... if anyone ever pulls that off, they will win many prizes. No core I am aware of fuses more than two consecutive instructions. None. The reason is quite obvious -- the combinatorial explosion of the number of possible combinations to consider, track, and handle. So now that we’ve established the bullshit excuse is bullshit, what is there to be done? Well, a few YEARS after the spec was written, an extension was proposed to help this issue -- Zba. It provides three instructions of the form SHxADD for x being 1, 2, or 3. This combines a shift with an add, basically shortening our array access from three instructions into two. This is still worse than having register + shifted register addressing mode, but at least now "the core can fuse them" becomes less bullshit and more believable, assuming someone produces such a core. One problem: SHxADD is always 4 bytes long, and the memory access instruction itself will be 2 or 4 bytes, so the array access becomes 6 or 8 bytes of code, to ARM's 4. This is where all those people who were just shouting about the wonderfulness of the C extension for density and how great it is for high-perf cores quietly shut up and look at the floor. Yeah... For extra credit, the Zba extension was only ratified in 2021, over two years after the base spec. It took them TWO YEARS to realize that arrays exist!

You know you really fucked up bad when you manage to make Qual­comm sound like the voice of reason.

Curiously, this would be easy to fix. Currently 3/4 of the encoding space is allocated to compressed instructions (all instructions whose lower 2 bits are not 0b11). Reusing some of that encoding space for better addressing modes is a no-brainer and would produce denser code with better array addressing ability. Unfortunately, it would make too much sense for anyone to actually do. Of all possible champions of sanity, Qualcomm ... proposed doing this, and even prototyped it. It went nowhere... And you know you really fucked up bad when you manage to make Qualcomm sound like the voice of reason. But, back to our SHxADDs. There is no guarantee that you’d get to use them anyways, since Zba is an extension and is thus optional. Are you getting tired of hearing "optional" yet? Let’s talk about that next.

Optionality

What does RISC-V have in common with USB-C and RCS? These things are all ostensibly standards, sure, but the interesting part is that claiming to be in compliance with one of these standards means NOTHING while being technically true. Is my USB-C cable wired only for USB 2.0 valid? Sure, USB 3 twisted pairs are optional. Is my non-e-marked cable valid? Sure, e-markers are optional. Can my USB 3.0 USB-C cable choose to not support 20Gbps? Sure, 20 Gbps support is optional. Can it support 20Gbps but not support 100W charging? Sure, that is optional too. Can my phone’s fully-compliant RCS implementation not support upgrading a text message to a video call? Sure! MIVC is optional. Can it fail to send pictures while I am on a call? Sure, that is optional! Can encryption not be supported? You bet, that is optional too! So what does it even mean to comply with the spec then, if everything is optional? It means the spec writers spent too much time engaging in mental masturbation and too little time in contact with the real world, basically. There are two ways this happens: academics who are not aware that outside their offices, there is such a thing as the real world, and design-by-committee situations, where the real world simply never gets a seat at the table -- having failed to file a motion to be seated there in time for the chairman to bring it to a vote.

Just scope out this line: "The RISC-V B (Bit-Manipulation) extension is a standard collection of instruction set enhancements designed to improve performance and code density through efficient bit-level operations. It is split into distinct sub-extensions: Zba, Zbb, Zbc, and Zbs." Only a design-by-committee process would ever unironically produce this sequence of words.

When writing a spec, every single thing you make optional, you split the possible implementations into two incompatible groups. Do this enough times and you end up with your spec being meaningless. And boy, did the designers of RISC-V screw the pooch here. Everything is optional! Multiplication -- optional. Division -- optional. Support for a user mode -- optional. Supervisor mode -- optional. CSRs -- optional. Compressed instructions -- optional. "Extra compressed instructions" -- still optional. I bet that if they thought they could get away with it, they’d make addition optional!

Every single thing you make optional, you split the possible imple­menta­tions into two incompa­tible groups.

The basic instruction set of RISC-V, at first publication time, included CSRs, which, as I had mentioned, are required for a standards-compliant method of handling interrupts as well as for support of differing privilege levels. That is not unreasonable; it is sane and not broken. Which is, of course, why they fixed it ... ASAP! CSRs got pulled out into an extension called Zicsr, and now the base ISA lacks ability to handle interrupts or provide privilege separation. But it is actually, and hilariously, much more idiotic than that. Let's say you have a lot of things that are optional. What is the first thing code would want to know? "Is feature X implemented on my hardware?" of course. CPUID is how you answer this question on x86. How do you do it on RISC-V? Well, I have good news and bad news. There is a CSR called misa which can answer some of those questions (not all of course, that would be too sane). Did you spot the problem yet? It is a CSR and CSR support is optional (Zicsr extension is not mandatory). If that was not enough of a crotch punch, misa is not required to be accurate if implemented -- it is allowed to read as all zeroes -- it being meaningful is itself entirely optional. Yup... the only way you have to detect optional features is optional. But wait, there is more!

The "M" in front of "misa" indicates that this is a machine-mode CSR. Machine mode is the highest privilege mode in RISC-V (and the only non-optional one, if you're keeping track). This register is not readable from lower-privileged modes, even if you are lucky enough to (a) be running on hardware that implements Zicsr, (b) be running on hardware where misa is not hardwired to be all zeroes, and (c) running on hardware that implements other privilege modes. This means that tailoring your code to the capabilities of the hardware is not possible for normal user code. If your hardware implements the OPTIONAL supervisor mode, it also cannot detect the core features. The party line is "ask the machine mode supervisor". The problem, obviously, is that you have no idea what that supervisor is or how to "ask" it. There is a common one in use called OpenSBI, but there is, of course, no way to detect if that is what your machine mode runs.

Another fun bit of optionality here is the system timer. RISC-V spec has a timer; it is optional, of course. It is not accessed using CSRs, because of course not! That would be too consistent. The official party lineexcuse is that this was done to conserve the encoding space in the CSRs. I guess this is because they expected to run out of ... 4096 of them‽‽ A bit ambitious if you ask me -- no current architecture comes even close, not even x86. But let's move on. If the timer registers are not CSRs then where are they? They are memory mapped! Where? Well, since a timer is a core peripheral that any OS would need, and since the CPU core spec specified it, it, of course, is at a well defined address that you can rely on. Just kidding! Nothing in this spec is that sane! The address is "implementation defined" and can be anywhere at all. Good luck, have fun, don't crash!

I shall tell you of just one more fun situation here, of the many I could: exception and interrupt vectoring. When an exception or an interrupt occurs (assuming the optional Zicsr is implemented), where does the CPU jump? Depending on a lot of optional and optionally-supported config regs, delegation regs, and all sorts of other overcomplicated nonsense, eventually the core will pick to use machine or supervisor vector register (mtvec or stvec). That CSR points to the handler, except its bottom two bits that determine its "mode". What is a mode? There are two modes documented. The direct mode is when the lower two bits are 0b00, in which case all exception and interrupts just jump to the address in the higher bits. The vectored mode (lower bits 0b01) is meant to simplify and speed up interrupt handling. All exceptions jump to the address in the higher bits of the reg, while all interrupts jump to that address plus 4 times the interrupt number. So what is my problem with this seemingly sane design? That both of the modes are optional!!!! No part of the spec mandates even the simple direct mode! It is possible, at runtime, to detect if a given mode is implemented by writing the lower bits, reading them back, and seeing if they stuck. But it would be entirely valid to implement a core with only vectored mode supported. Or only direct mode supported, or both, or neither, if instead your core vendor invented their own separate mode. This makes writing any sort of a generic kernel very difficult -- you literally have no idea what to expect. Why direct mode was not made mandatory I cannot fathom, but I can surely tell you that the person who decided that wore oversized shoes, had a big red nose, and wore a lot of white face makeup.

It really looks like the authors had heard of Popek & Goldberg, but failed to read past the abstract.

At this point in time you might jump to the defence of this indefensible idiocy by shouting one of two things: "other architectures also have optional features" and/or "hiding misa is needed to support virtualization, haven't you read Popek & Goldberg?". Let's demolish these feeble excuses one at a time. For "other architectures" we'll consider things in common use in the last few decades: x86, ARMv7, and Aarch64. x86, as previously mentioned, has CPUID which will happily tell you which features the current core has. It will do this quite easily in user mode, as one would expect. Arm has ID_AA64PFR0_EL1 and ID_AA64ISAR0_EL1 available to the kernel at least (though not to userspace). But there is a much more important point to be noted here, which explains why ARM's design is not fatal. In both x86 and in ARM, optional features are of two clear classes: (1) high performance compute that is usually programmed using intrinsics or hand-rolled assembly for tight loops in special circumstances (video encoding, fluid simulations) or by libc (memcpy, memset, strlen), and (2) NOP-compatible optional things that can be safely run on hardware that does not support them since it will execute as a NOP and that is safe. For x86 that would be endbr64 and for aarch64 that would be almost all of the PAC instruction set. Note that at no point are things needed in completely normal compiled code optional. Multiplication, division, addressing modes, are always available. This means that a normal C compiler targeting these architectures does not face the impossible choice of: "compile for the lowest possible denominator to allow code to run on all arch versions" or "assume things like multiply and sane addressing modes exist and prepare to crash on a core that chose not to implement them". Yes, of course, this is where people will say "just target your exact core, why don't you?" Yes, I never said that the idiocy of this ISA cannot be overcome with enough contortion. I said that this sort of poor design was acceptable in the 1970s when we did not know better and is inexcusable in the 2000s, as now we do.

Now, on to the virtualization excuse. First of all, Popek & Goldberg talk about an architecture being virtualizable specifically in the context of it lacking special virtualization support. Indeed by trapping every instruction that acts differently in user and supervisor mode, one can virtualize any architecture. But the alternative is just building-in virtualization support. x86 is not virtualizable as per Popek & Goldberg, at least due to POPF instruction. And yet I have VMs running on my x86 box just fine, as x86 added support for virtualization. It was nontrivially difficult to bolt it on post-facto, but it was done. If doing it at architecture design time, it is trivial. Which is to say that ANY mention of Popek & Goldberg to justify decisions made at architecture design time is bullshit. Arch design time is precisely the time to do it right. Popek & Goldberg even mention that trapping everything is theoretically interesting for the proof of virtualization but not practical. NOT PRACTICAL. So what did the designers of RISC-V do? They justify misa being not exposed to supervisor and user mode with "but virtualization... what if the hypervisor wants to hide capabilities from a VM?". Bull ... let it arrive ... shit! x86 and ARM both manage that just fine. And RISC-V could have too, simply by allowing the hypervisor to lie about misa's contents while letting everyone read it still. It really looks like the authors had heard of Popek & Goldberg, but failed to read past the abstract.

Another thing they excuse by a vague hand wave in the direction of Popek & Goldberg is the inability of the executing code to detect what CPU mode it is in. This is, again, utter nonsense. x86 exposes this indirectly via POPF (for example) and ARM does not even make you trick it, exposing it completely openly in the CurrentEL MSR. Detecting the current mode on RISC-V is an adventure. It is sometimes possible, but not in all cases. Why might you need this? For example, if you are writing a kernel and want it to support all RISC-V cores. I spent a bit of time trying to make this work for my kernel for rePalm, so I can walk you through the decision tree and show where each branch comes to life and whacks you in the gonads. First of all, if the core has no Zicsr, you are guaranteed to be running in machine mode, but, there is no way to know that there is no Zicsr, other than probing by doing a CSR read, but there are two problems: first, without Zicsr, there is no proper generic way to catch the resulting exception when an invalid instruction trap is generated. Second, what CSR to read? Don't forget that likely all of them are optional. One might be tempted to go for misa, but do not forget that you are probing what mode you are in. If you are in supervisor mode, that probe would also fail, even if Zicsr was implemented. Ok, you can try reading sstatus. That one is readable to supervisor mode AND machine mode. You're safe, right? You wish! Supervisor mode is optional, and if your core does not implement it, there is no sstatus register, so ... you trap. Ok. Let's simplify the problem. Let's assume Zicsr exists. Can you then at least tell apart S mode from M mode? Nope! The following seems like a tempting solution: set stvec to point to your handler that simply adjusts sepc forward by 4 and returns (skipping the faulting instruction), then execute a read of misa. If you were in machine mode, it reads fine. If you were in supervisor mode, it traps, and any sane machine monitor would hand you an illegal instruction trap. Then, your handler would skip the instruction, and you'd note this and conclude you were in supervisor mode. You win, right? Almost... Once again: supervisor mode is optional. Let's imagine you were in machine mode on a core without supervisor mode support. You'd trap as soon as you tried to set stvec, since it does not exist. You might be tempted to say: why not just catch that trap too? Because to do that, you need to set mtvec, and you cannot be sure you can do that since you might have been in supervisor mode all along. Thus the intersection of everything being optional and the authors' complete misunderstanding of Popek & Goldberg lands the poor you in a pile of shite.

Missing Obvious Pieces

On average every other function uses or could use an instruction to branch on the value of a bit.

Despite having an extension for seemingly everything, including operations on the common kitchen sink, somehow a number of obviously-useful instructions are missing. I already covered the lack of register + register addressing modes, so we'll not bother returning to that. There are a few other obvious low-hanging fruit that were seemingly ignored. And before you argue that RV32I was designed to be simple, all of the things I am about to suggest are trivial in the extreme, mostly reducing to simple wires on an ASIC.

First and foremost: test a bit and branch based on it. This one instruction replaces two (SLLI + BGEZ/BLTZ), but also it does not require a temporary register. To check how common this would be, if it existed, I grabbed a random aarch64 binary (the latest raspian kernel for raspberry pi, "vmlinuz-6.1.0-49-arm64", sha256: B3B686DE 82CC7B84 EFEB8F6B 309A4E6C 53E7461F 281D3E56 F42E4AA3 B6207075), disassembled it, and counted the number of instances of TBZ/TBNZ. There were 35,393 instances. By comparison, there are 70,109 instances of RET, which means that on average every other function uses or could use an instruction to branch on the value of a bit. Implementing it is trivial in hardware, and indeed branching on a bit is extremely common in dissecting protocols or using bitfields. For big out-of-order cores, renaming is simpler when one fewer register gets clobbered, and also there is no need to try to fuse two instructions when one exists. For smaller MCU cores, where no fusion exists, this is a simple code size and speed win. Why this obvious thing was not done, I do not know.

My next major gripe - bitfield operations - bit field extract and bitfield insert. These are extremely useful for working on things like network packets and hardware registers. Bit field extract can be simulated using two instructions - SLLI + SRLI/SRAI based on the desired signedness. Bitfield insert takes a lot more work to simulate: create the inverse mask, AND with destination reg, shift source reg into place, OR into destination. Depending on the bits, it is 3-6 instructions easily. BFC (bit field clear) is a simpler special case that is also quite useful. It is doable in 2-3 instructions. In hardware though, it is just wires - no complex logic, no nothing! There is no need to be clever like aarch64 is, although the designers of RISC-V could learn a lesson or ten from aarch64 indeed, including clever bitfield handling. I did the same counting exercise using the same kernel image. There are 6,284 bitfield insert instructions and 8,881 bitfield extract instructions -- two out of every 9 functions on average use these bitfield ops. To add insult to injury, there is a bit-ops extension for RISC-V -- Zbs. By looking at it, you can tell the authors were academics. It is clean, simple, easy to explain, elegant, and completely useless. Who the hell ever needs to extract just one bit? Seriously, what a missed opportunity.

Ridiculous encoding

RISC-V is the first architecture I've ever encountered which scatters immediates randomly throughout the instruction with no immediately-clear reason for it. This makes emulating it a huge pain since it takes so very long to recombobulate the immediate values, compared to architectures like MIPS, or ARM. The former simply uses bits 0..15 for immediates, the latter has fancier encodings, but at least there are just a few. RISC-V designers' justifications for this were "the same bits of the immediate come from the same bits of the instruction" -- a justification so idiotic that it physically hurts to attempt to pretend to believe it. It is the sort of thing that a software person who's never written verilog would think helps make things easier. You see, no matter how you spin it, you will need a mux for immediates, since they are of differing lengths and with differing number of trailing zeroes, depending on the instruction. And that mux, well ... it does not care even a little which instruction bits are wired into its inputs, it is all just wires. But even if the reasoning for "why" is idiotic, let us inspect the claim itself, to see if they did accomplish what they claimed to have wanted. Do the same bits of immediates always come from the same place? Let's take a look. For I-type instructions, bit 1 of the immediate comes from instruction bit 21, bit 11 of the immediate comes from instruction bit 31. For S-type instructions, the same bits of immediates come from instruction bits 8 and 31 respectively. For B-type instructions, they come from bits 8 and 7 respectively. And for J-type instructions, they come from bits 21 and 20 respectively. As you see, they indeed always come from the same place, as promised, if we merely ignore the meanings of the words "same" and "place".

But this just barely touches the surface of the insanity of the encodings! For J-type instructions, the immediate value is scattered in the following order: 20 10 9 8 7 6 5 4 3 2 1 11 19 18 17 16 15 14 13 12. What possible justification could you imagine for this insanity, other than that the designers confused the chatter of a bingo parlor for the proper bit order for immediates. And yet, this is nothing compared to the mess that they made of the compressed instruction set...

There are no fewer than 9 (nine!) instruction formats here, and that does not include Zcb, which adds 8 more! But even that is not all! Depending on the instruction, the same format (eg: CI) encodes immediates differently in the same bit positions. Accounting for all of that, it is almost one instruction encoding format per instruction! It did not need to be like this! Thumb and microMIPS both give examples how to not fuck up this badly, and yet, despite easy availability of examples of how to do it right, RISC-V designers bid us hold their collective LSD-laced beers and went at it in the most pessimal way imaginable. Let us first, of course, look at the immediates in C, since "they always come from the same place in the instruction word", you know ;)

When talking about immediate encodings henceforth, I shall use "x" to indicate when the immediate is broken into pieces and there is something else there in the instruction. L.LWSP (which loads a word from the stack) encodes the immediate in this order: 5 x x x x x 4 3 2 7 6, C.SWSP which is its sibling for storing to stack, instead, encodes the immediate as 5 4 3 2 7 6. C.LW (which loads a word from memory addressed by a register) encodes its immediate as: 5 4 3 x x x 2 6, naturally. Its sibling, C.SW uses the same encoding, indicating that the design team missed an opportunity to scramble some bits here. If you wanted to load a byte from memory, you'd use C.LB, whose immediate encoding is, of course, nothing like the above. It uses bit order: 0 1. If you wanted to load a halfword, you'd use C.LHU, whose bit order is just: 1. Because of course it is! I am not even going to touch on CM.PUSH and CM.POP because their encoding is so complex that the spec spends a whole chapter explaining how to decode them. Truly, a sign of a simple and intuitive encoding, if you ask me.

While claiming to make use of all possible encoding space in the 16-bit instruction space, some fruit remain so low-hanging as to require OSHA warnings! A simple example: logical shifts include 6 bits of immediate. The argument is that this is needed for 64-bit instructions, but there are already many encodings in the compressed instruction set that are RV64-only. That is to say that RV64C and RV32C are already incompatible. Given that, why the hell are all shift instructions in RV32C carrying an extra zero bit? You cannot shift a 32-bit register by more than 32 bits. The spec says that bit must be zero, and yet no encoding uses the space opened up by that bit being one. Self delusion is telling yourself you are making good use of encoding space while also carrying around the ability to shift 32-bit registers by 61 bits, my friends. RV32E is even more egregious since many of its compressed instructions carry an extra bit to encode registers that do not exist there. As RV32E is ABI-incompatible with RV32I and they would never share code, giving RV32E more useful encodings by using those extra bits that would have encoded registers x16..x31 would have been an excellent idea. That is probably why it was not done.

Moving on... C.J's bit order could likely pass the NIST Statistical Test Suite for random number generators: 11 4 9 8 10 6 7 3 2 1 5. What even‽ C.BEQZ/C.BNEZ are also jumps, though conditional, so of course their encodings have almost nothing in common with the previous one, they use: 8 4 3 x x x 7 6 2 1 5. When encoding immediates for C.LI, the bit order is 5 x x x x x 4 3 2 1 0. That almost looks sane, but do not despair, more fun is coming. Let us say you wish to adjust the stack pointer. C.ADDI16SP is here for you, with its immediate encoded as: 9 x x x x 4 6 8 7 5. And if you wanted to get an address of a stack variable, C.ADDI4SPN is there, with its immediate encoded as: 5 4 9 8 7 6 2 3. All very logical, sane, and clear, as promised.

Imagine you are a CPU (or an emulator) trying to decide how to decode an instruction. If you are a sane CPU, most likely it goes like this: look at 1-3 bits to determine instruction format, from there look at 2-5 bits to figure out the instruction, and you are done, you are ready to execute. If you are RISC-V, things are a bit ... more complex. First you look at the bottom 2 bits to figure out if the instr is 2 or 4 bytes, for 2-byte instructions, you look at the top three bits to determine what instruction this is. So far, so good. But then... you notice that this instruction has an immediate. You need to reassemble the jigsaw puzzle that that is. And then you recall that some register-register instructions are encoded in the immediate format, with a magic immediate value indicating that they are register-register ops. For example C.NOT is encoded this way, the magic value being 0b111101. C.ZEXT.W (if your hardware implements the proper mishmash of extensions to have it at all) uses the magic immediate 0b111100. Somehow microMIPS and Thumb managed to do without this insanity. How? Ancient secrets that apparently were irrecoverably lost before the RISC-V authors were born.

If this were not enough, it is also notable that there are conflicting encodings in the compressed instruction set, depending on which extensions are implemented. Some extension combinations are simply impossible (eg: Zcmp and D). This is a bigger fuck-up than all the previous ones since it is not merely cosmetic or efficiency-related. Despite DECADES of accumulating backwards-compatibility cruft, even x86 has managed to avoid the obvious trap that is having the same byte sequence mean different things to different implementations of the same architecture. I repeat: the architecture with the famously-terrible encoding managed to preserve semantic stability across almost 50 years, while the clean-sheet architecture designed by people who had decades of hindsight apparently didn't manage to do it across five years. It is OK for an architecture to have unimplemented encodings that become implemented instructions in later versions. It is justifiable to have implemented instructions that become unimplemented later. However, having encodings change meanings (or worse: start off having different meanings) in different implementations of the same architecture is insane! I think I recall seeing a whole chapter on this in the DSM-5!

Having encodings change meanings [...] in different imple­menta­tions of the same archi­tecture is insane!

Having this happen means that instead of a clearly-understandable crash you get ... well ... anything. Who can predict how their binary will act when a floating point store silently becomes a double-register move or a jump instruction, or vice-versa? This is not a summary of a plot of a B-side programmer-themed horror flick. It could really happen to you! Consider the instruction 0xA002. Depending on your core, it could store a double-precision floating point register 0 to stack offset 0 (C.FSDSP f0, 0(sp)), or it could jump somewhere (CM.JT 0). But at least a random jump might cause a crash soon enough for you to notice. Consider 0xAC66. Depending on your core, it could store a double-precision floating point register to stack (C.FSDSP f25, 0x18(sp)), or it could move a0 into s0 and a1 into s1 (CM.MVA01S s0, s1). No control flow changes. Just two corrupted registers and a value not stored to stack. Some would attempt to argue that this confusion could never happen, since everyone knows (or should know) what their core is and what it does. Those "some" have clearly never encountered the real world. Oftentimes, you get binary blobs from vendors and you just link them into your microcontroller code. MEMS sensor vendors are notorious for shipping their "super proprietary" calibration algorithms like this. Now, let us imagine that we had such a binary which we linked into shipping code and shipped it on an MCU which had the Zcmp extension. It was all smooth sailing until we upgraded to a larger MCU to have better floating point support for some fancier math our new product needs to do. This new MCU supports double-precision floating point numbers. Suddenly, our binary is crashing randomly in random places and misbehaving. There are no "undefined instruction" traps, no immediately-obvious cause. Turns out, the new core supports the D extension, and thus cannot support Zcmp. But, due to the reused encodings, we did not find out the normal way -- an undefined instruction trap. Instead, we spent weeks debugging random crashes when random registers were getting corrupted. Our disassemblers were no good - they correctly disassembled the instructions in one of two valid ways. Debuggers were of little help too -- they showed some instructions seemingly doing nothing, with a few words on the stack becoming corrupt soon after. Interrupt handlers were overhauled a few times, wasting a lot of engineer time, to make sure they saved and restored contexts correctly. The MCU vendor's FAEs spent days onsite helping our electrical engineers rule out noise in the power supply lines that could have been causing random register and stack corruptions. Also, the sensor calibration has been broken the entire time since the MCU upgrade, but since sensors usually work plausibly well without calibration and nobody noticed among the random crashes. Now, debugging this, IS the plot for a B-side programmer-themed horror flick that I've been shopping around Hollywood for a while. Dear RISC-V committee, you are staying after class today and writing the following on the whiteboard 100 times: upgrading a processor should not turn a valid binary into a semantically different valid binary.

Alleged Fixes

Many RISC-V fans will claim that all the optionality is no longer an issue since the RISC-V foundation created "profiles" which is a list of mandatory optional extensions (yes, do read that again) that when implemented together allows one to claim to comply with a profile. Again: the ecosystem has had to invent a second layer of standardization whose entire purpose is to say which parts of the first standard you are actually allowed to assume exist. Really, if you find yourself having to build a second standard whose purpose is to tell everybody which parts of your first standard they must actually implement, perhaps the first standard wasn't quite finished. Only a design-by-committee process could produce this situation. To avoid having to target the lowest common denominator of RISC-V, or the embarrassment of having to build thousands of slightly different builds of every package so as to target every possible extension combination, most vendors who are delusional enough to hope that RISC-V desktops will actually happen are planning to mandate RVA23. Ubuntu, Red Hat, and Android are all in this group. Fun story: of all the currently available and recommended at time of publication RISC-V SBCs, approximately none are actually RVA23 compliant, not StarFive's VisionFive 2, not the Banana Pi BPI-F3, not the Lichee Pi 4A, not the Orange Pi RV2, not even the HiFive Premier P550. None of them will ever run Ubuntu LTS or future AOSP builds. Hilariously, some discussions online have taken to talking about "almost RVA23" cores. Again: the profile designed to solve fragmentation isn't backwards-compatible with much of the hardware that is currently sold as "RISC-V", thus fragmenting the RISC-V world into "almost-RVA23" and post-RVA23, with the majority of the current boards being "almost". The fans will call this profile system a win, but I say it is a direct indictment against this crime of infinite optionality and the people who thought it was ever a good idea. Big desktop-type cores do not need the divide instruction, array addressing, and basic SIMD to be optional. And nobody should have needed TEN YEARS to realize this. Additionally, a lot of vendors whose chips will not make the RVA23 cut are likely now screwed, as Android and Linux leave them behind, no matter how "almost" they were. That is what they get for buying into this, I suppose. A lesson to be learned about FOMO and getting in early -- sometimes it only "almost" pays off. As a hilarious side-note, there is one linux distro which is completely and perfectly tuned for this insane RISC-V landscape - Gentoo, since its thing was always: build each package from source to take advantage of the exact host CPU features and optimizations. That is: RISC-V's ideal (read: the only plausible) software distribution model is apparently Gentoo. After I hit "publish" on this article, I shall be off to find myself a contractor to make a Bat-Signal™-style lantern with the Gentoo logo, to aim at the clouds above the RISC-V HQ. Gentoo, you have been summoned to a battle you did not even know you have been preparing for your entire existence!

How Did We Get Here and Where to Now?

If you read the RISC-V rationale/excuse doc, and discard all the parts that we've concluded (in the above sections) to be outright lies, exaggerations, or complete misunderstandings of how existing architectures (and the world at large) work, we end up with this one sentence that likely explains it all: "Ultimately, we thought it was best for our purposes to start from a clean slate, rather than modifying OpenRISC accordingly." This is after they explain that OpenRISC does all they wished to do except it has delay slots, and proceed to admit that a version without delay slots already existed. There you have it: why the bits are scattered like bingo, why there are hundreds of conflicting extensions with conflicting encodings, and why obviously useful instructions are nowhere to be found -- simply because they all were "not invented here".

To be fair, another possible explanation for how we got here is that this is how academia and grants work. By their nature, grant proposals tend to oversimplify and overpromise their real-world applicability and impact. Given that a lot of RISC-V "research" was done under various grants, it is plausible that the incentives of the academic world had an influence. Since this hypothesis is unverifiable with the information available to us, we’ll leave it at that.

Does This Mean RISC-V is Doomed?

None of this is to say that RISC-V is doomed. As I said, I fully expect it to take over the space currently occupied by 8051 as a typical cheap embedded core instantiated when some small amount of logic is needed, or some beefy accelerator or DMA engine needs some mild babysitting. Much like the linux kernel -- the price is right. ARM wants licensing fees, while the RISC-V spec is free, and (this part is key) there are cores out there which can be licensed for free. For situations where performance is not a factor and price is, RISC-V will win simply due to its price. "Good-enough" is a low bar in this case, and RISC-V is of the right height to meet it. And look, this market is not glorious, but it matters and it needs new blood. However, it is important that RISC-V not accidentally think it was chosen for being good. It needs to internalize that it was chosen for being cheap. This is not meant as an insult to cheap small cores -- I've written plenty of assembly for all sorts of shitty cores with shittily designed ISAs. RISC-V is an improvement over PIC and 8051, as little of a compliment as that is.

'Good-enough' is a low bar in this case, and RISC-V is of the right height to meet it.

For big compute, there are two separate categories, in my opinion. ML accelerators will likely also end up with RISC-V cores attached to them. This is actually pretty close to the above use case, since most of the computation will be done in specialized blocks of silicon, optimized for many many MATMULS per cycle. The RISC-V core will be configuring DMAs and reconfiguring these large blocks to compute the next layer's outputs. A variant of this design might include a RISC-V core with a very wide vector engine bolted to it. This can be used for element-wise operations which are also common in ML workloads. Again, this core need not be fancy or even out-of-order, it will work simply by being wide enough vector-wise. Its integer pipeline will not be a significant factor for its performance, and RISC-V's warts will be considered the price you pay for not paying to license a Cortex-A55. I expect RISC-V to win this market for the same reason as for small babysitter cores - actual serious per-core CPU perf in the traditional sense is not needed here, so a good-enough core will do. And traditionally, good-enough solutions are always chosen via the "ORDER BY price ASC LIMIT 1;" process.

The second category for big-compute is actual desktops and SBCs that do interactive computation, browsing, gaming, and other such "desktop work". I do not expect RISC-V to be a serious player at the top of this market. Simply put, the architecture is not designed for it, as pointed out above. Additionally, this market has the margins to afford licensing a much-better-designed aarch64 core from ARM, and gain proper support from a much larger corpus of software. Before you get your megaphone to shout about "openness", please note that the openness of the RISC-V spec is not relevant here at all, because an open spec does not magically materialize a well-designed out-of-order core for you for free. And if someone were to design a good out-of-order core, they would not be giving it away for free. An open spec does not mean every implementation is free. The middle and the lower ends of the cheap SBC market will likely remain a mixture of new RISC-V cores and older ARM cores.

I have for many decades been fascinated by the prospect of a curve family better suited for interactive design than cubic Béziers. In that search, I have come to a new-found respect for those Béziers. In particular, though other curves like Euler spirals are better at representing smooth curves, they fall short at representing regions large curvature variation, where cubic Béziers excel. The great strength of Béziers is their versatility.

So, to find a strictly better curve family, one requirement is clear: the family should contain both smooth curvature variation and higher-tension regions where curvature peaks. Polynomial spirals, or Spiro curves, the subject of my PhD thesis, fail to achieve this goal.

After considerable search and rejecting a number of candidates, I now bring a proposal for a curve family which I think is a very strong candidate for supplanting cubic Béziers in 2D vector graphic design.

Without further ado, the curve family is represented by the Cesàro equation, specifying curvature as a function of arc length:

$$\kappa(s) = \frac{as+b}{(cs^2+ds+1)^{1.5}}$$

This curve behaves surprisingly similarly to a cubic Bézier, especially at smaller angles, but when pushed has very different behavior. Overall it has smoother curvature variation and is more likely to have monotonic curvature. It contains within it a few valuable analytic curves, and is also good at approximating a wide range of others. The remainder of this blog post is devoted to showing its behavior in a wide range of contexts.

Approximation of cubic Béziers

The hyperbezier closely approximates cubic Béziers at low deflection angles at the endpoints. I will show rather than try to present mathematic reasoning. Below is an interactive tester that maps cubic Bézier control points to a corresponding hyperbezier. The Bézier is shown in gray for comparison.

At larger angles, and also when the "arm lengths" of the control handles get larger, the fit with the cubic Bézier is not particularly close, but the control scheme is still a useful way of setting the parameters for the hyperbezier curve (setting the polynomial coefficients directly is not at all intuitive).

Note that this parameter mapping is a first usable draft, and may not be the final version. That said, in an interactive editing context, a perfect mapping is not required, as it's always possible to tweak the control points to reach any desired curve shape.

The details of the mapping can be found in the JavaScript source for this page, but the basic principle is that the arm lengths set the denominator, then the numerator is solved to make the endpoint tangents match.

Exact analytical curves

The most obvious analytical curve contained in the family is the Euler spiral, which is clearly attained when $c == d == 0$. The Euler spiral has smooth curvature variation (it is a solution to the Minimum Variation Curve problem) and monotonic curvature. An exact circular arc is also within the parameter space, simply when $a == 0$ as well. Cubic Béziers, by comparison, only approximate circular arcs.

There a few other log-aesthetic curves lurking in the parameter space, meaning curvature is simply the arclength raised to a particular power. Reachable exponents include -3, -2, -1.5, and -0.5 (alternatively, using the convention from the log-aesthetic papers, α can be 1/3, 1/2, 2/3, and 2). The last of these is the circle involute, which is interesting because it is its own parallel curve, among other things. And the first of these is the evolute of the Euler spiral.

The parameter mapping described above is designed so that when the control points lie on the "double parabola" in my Euler spiral parallel curve blog post, then the result is an exact Euler spiral.

Superellipses and squircles

A cubic Bézier can do superellipses up to a certain point, but does not approach a sharp corner; somewhere before then it starts developing additional inflection points, while a true superellipse or squircle is of course convex.

The hyperbezier can go all the way to a sharp corner, and visually looks pretty close to the superellipse. It's not incredibly accurate (for moderate exponents like 5, the best cubic Bézier fit is slightly better, in fact), but visually does the right thing. It's probably best to say that the hyperbezier is a subtly different squircle than the superellipse, neither better nor worse.

Hyperbola

Another fundamental curve is the hyperbola. It is round at the turn, but curvature tails off and the curve reaches a linear asymptote on both sides. Cubic Béziers do not fit this curve well, and do not exhibit that asymptotic behavior. But the hyperbezier fits it naturally and with very high accuracy.

We can also appreciate the relationship mathematically. In both, the curvature tails off as $\kappa \approx s^{-3}$ asymptotically. In addition, using the small angle approximation $\sin \theta \approx \theta, \cos \theta \approx 1$, integrating the Whewell equation (see below) of the even-symmetric hyperbezier yields the hyperbola exactly. This close relationship is one of the inspirations of the name; it is something of a fusion between a hyperbola and a Bézier. Another inspiration, incidentally, is that the log-aesthetic solutions, including the Euler spiral, are represented by hypergeometric functions.

Elastica

An important curve family to consider is the elastica, an idealized thin flexible strip. The math behind the elastica has hundreds of years of history, but more to the point it is both a smooth curve (it can be characterized as the exact solution to the Minimum Energy Curve problem) and can exhibit large curvature variation when placed under tension. To my eyes, it's a more pleasing and natural curve, as it's based in physical reality in a way that cubic Béziers aren't. Ideally we'd be able to approximate it well, and the fact that Spiro curves can't is a strong case against them. Fortunately, our hyperbezier does reasonably well. As in the case of the superellipse, it's a fairly decent visual match, though again not a precise approximation.

Some mathematical properties

This section is a grab-bag of some mathematical properties of the hyperbezier curve.

An appealing property is that the Cesàro equation is easily integrated, yielding a Whewell equation:

$$\theta(s) = \frac{a's+b'}{\sqrt{(cs^2+ds+1)}}$$

Here, the parameters in the numerator aren't the same as the Cesàro equation (though they are readily derived), but the quadratic polynomial in the denominator is the same. Another round of integration (which I believe will be especially useful for curve fitting) also yields simple analytical equations, revealing that a close relation to trig functions.

The derivation of parameters for the Whewell equation follows fairly straightforwardly from the following integral, which has a pleasing symmetry:

$$\int \frac{as+b}{(cs^2 + 1)^{1.5}} ds = \frac{bs-a/c}{\sqrt{cs^2 + 1}} + C$$

Going in the opposite direction, the derivative is also straightforward, yielding a quadratic over the same polynomial in the denominator, this time with an exponent of 2.5. To compute roots, identifying the curvature extrema, involves merely solving the quadratic.

Because the denominator is positive, there is at most one inflection point, at $s = -b/a$, when it is in range [0, 1]. That clearly signals that it can't accurately approximate all cubic Béziers, as those can have two inflection points. In my experience, instances of cubic Béziers with two inflection points are rarely used in designs (it might be interesting to look at a corpus of vector graphics to quantify this).

I have experimented with other values for the exponent than 1.5. A value of 1 would be appealingly simple, as it would resemble Padé approximation, but I found it cannot represent high-tension curves in a numerically stable way. I also considered an exponent of 2, but the integrals don't come out as cleanly and overall it didn't match cubic Béziers as closely. Perhaps this is not surprising, as the exponent of 1.5 resembles the equation for curvature of a parametric curve $\mathbf{x}(t)$:

$$\kappa = \frac{\mathbf{x}' \times \mathbf{x}''}{(\mathbf{x}' \cdot \mathbf{x}')^{1.5}}$$

In general, arc length parametrized curves are in many ways more pleasant to work with than general parametric curves; there's a nontrivial calculation to solve the inverse arc problem for Béziers.

Future work

In this post, I present the curve family to the world. I welcome and encourage experimentation. Of course, there is lots to be done to make the mathematical idea into a practical tool. For one, it should be wired up into a spline. I expect to follow the ideas in The hyperbezier pen tool by Colin Rofls, which was based on an earlier draft of the hyperbezier math. This earlier draft had some of the desired behavior of the current proposal, but also some flaws, including not being able to approximate sharp superellipses, and also not being closed under subdivision. On the other hand, it had a straightforward solution to the problem of mapping Bézier-like control points to the parameters.

One idea from the above spline that I think is worth preserving is making splines G2 continuous by construction. Such approaches don't work well with cubic Béziers because there are multiple local minima, creating saddle points in the optimization terrain.

Another promising direction is working out curve fitting. While I haven't gotten deep into it yet, I expect curve fitting to be fairly straightforward, especially compared with cubic Béziers. I'm especially interested in fitting touch or pen data. As usual for this problem, a central part of the challenge is finding good subdivision points.

I hope to land better implementations in the spline repository, but make no guarantees how fast that will go.

Acknowledgements

I've been working on this for years, and have benefited from discussions with many people, including Colin Rofls, Alex Ionuț, Jacob Rus, and Trevor David Black (apologies if I've missed someone; just let me know). Special credit goes to Muhammad Ragib Hasin, who improved many of the numerical and solving techniques for parameter mapping. AI assistance was used to refine the current version of parameter mapping and to prepare the visuals for this post.

The biggest lie in color photography is that you can accurately represent the colors of objects by simply recording the amount of red, green and blue in them. This technique - the only one in current mainstream use - gives good results only when the spectral sensitivity curves of the camera precisely match those of the human eye, and when the spectrum of the light used to make the photo is perfectly smooth, and no different kind of light will ever be used!

It's clear that the first of these requirements is hard enough to meet, and that the second one is, simply and plainly, never true.
 
And that's why photographers are always battling to get the right colors - and never do get them!

Not only in photography is this matter an important one. In daily life it is, too. Lots of electronic technicians hate that stupid problem of not being able to correctly read resistors. It happens that old-style resistors, and some other parts too, are labelled with color bands or dots, instead of numbers. Under some lighting conditions it can be hard to tell a red from an orange, or a green from a blue. This leads to the wrong resistors being installed in equipment, and thus more troubleshooting work.
 
Housewifes know the same kind of trouble, for example when trying to color-match a button to a shirt. Indoors under the electrical light they find the exact right button, that matches the others, sew it on, and when the dear hubby goes outdoors next day, !BANG!, that button sticks out like a sore thumb!
 
Well, I have to admit that many housewifes these days don't know how to sew a button to a shirt, but they tend to have the same kind of trouble when getting their make-up just right, only that outdoors it doesn't look right any longer! And that's a big problem...

To get a better grip on the problems of light color, I built myself a spectrograph a few days ago. It has been a lot of fun so far, so I'm making this colorful web page, both to bring my results into an orderly shape, and to let other people learn from them.

The spectrograph is a simple attachment for my DSLR camera. Using my lathe, I made a two-part piece of plastic tubing, that assembles in an angle of 146.6 degrees. At the junction of the two parts, a diffraction grating is installed. It's an inexpensive foil-type grating that has 1000 lines per mm, which I bought on eBay.

Each end of the angled tube screws into the filter thread of a lens. I used 50mm lenses on both sides, but other arrangements are workable too. The camera's lens stays focused at infinity, while the additional lens, uses as a collimator, has a narrow slit installed in the center of its focal plane. I made that slit by hot-gluing two pieces of hobby knife blade over a central hole in the cap, on its inside. The blue pipe is simply PVC water pipe, machined on the lathe to press-fit the lense's bayonet, and to have 42mm length. This allows using the lense's focusing ring to bring the slit into the exact position, and thus focus the entire spectrograph.

Looks cool, eh?

There are several web sites describing the construction of such spectrometers in greater detail. Many of them don't use the second lens, and instead use simply a long tube, placing the slit at a long distance from the diffraction grating, and focusing the camera's lens at that distance.

The work being done, let's go and play.

Let's start by looking at a full, pretty smooth spectrum of light. It's 1500 pixels wide, so it would be best to set your browser window wide enough. The left border must be roughly at 700nm wavelength, maybe a little lower, while the right one is slightly into the UV range, a little bit shorter than 400nm. I got this spectrum from a blue-tinted "daylight" incandescent bulb.

The stage being set, let's look at some single-color LEDs. Here is a red LED dating from roughly 1988. It's pretty far down in the deep red range. It looks very dim to the eye, just enough to use it as an indicator light on a front panel.

The following one is a modern, high efficiency red LED. It works much higher in the spectrum.

Next in the spectrum comes an old (1988) yellow LED. The spectral range that looks yellow to the eye is pretty narrow, and those old LEDs had a fairly broad bandwidth. It spans from the red over the orange and yellow, well into the green. It looks a dirty yellow to the eye.

This instead is a modern "orange" LED. It has a much narrower bandwidth, allowing a more precise color definition. It looks a beautiful golden color to the eye.

Then comes an old green LED. As you can see, it's not cleanly green,  falling into the lower half of the green range, and a lot into the yellow and orange. It looks green enough to the eye, partly thanks to a green tinted housing! Without that tint, I guess this LED would look rather yellow.

Instead a modern green LED has a far narrower bandwidth, and is better settled in the green range. I'm not sure where the Moir� interference in this spectrum came from. I don't think it's an actual ripple in the spectral response.

What you can see here is a historical device, straight out of my museum: The very first blue LED I bought! It was very expensive, and gives a very dim glow, but with some goodwill it looks blue indeed! I bought it just to show off with it, in a time when most people still thought that blue LEDs were impossible to make! I understand it's a silicon carbide LED.
As you can see, like all old LEDs it's pretty broadbanded, and is centered in the cyan range rather than the blue.

This instead is a modern, true blue LED.

And what follows is a LED that was sold to me as "ultraviolet". I can't tell how much actual UV output it has, because my camera and my lenses will cut off any higher UV. But it looks promising that I had to lengthen the crop of my spectrum image, to fit the far violet part of the spectrum, which I had cut off before because I though my camera wouldn't see anything there!
Note that it also has some faint green and red output.

As you can see, there are LEDs for pretty much any part of the visible spectrum. In my collection I don't have any clean, nice, narrowband cyan LED, but it surely exists too!

Let's go to actual lamps now. The above LEDs are intended mostly as indicators, having strong colors. Lamps instead are normally designed to produce the whitest possible light, or let me say, a light as close as possible to daylight.This is because the standard for color vision and reproduction is how it looks in nature, under the open sky. This can be either a sunny day, with the proper mix of yellowish sun and blue sky shining down, or a smooth layer of white clouds, which mixes together the light. Of course, clouds do absorb some wavelengths more than others, and so the color of cloud light is not the same as well mixed light of a sunny day. Matters ain't ever simple!

The smoothest spectrum comes from blackbody radiators. Plain incandescent glow bulbs are such blackbody radiators - only they aren't hot enough! To produce the holy grail in perfect lighting, we need a blackbody heated to about 5000 kelvin, which is pretty close to 4700 degrees Celsius. The problem is: What material can be used, that survives such temperature while remaining solid?

So we use tungsten, the metal with the highest melting point of all, and heat it up as much as we possibly can, without it vaporizing too fast. That's the humble glow bulb. What we get is a smooth spectrum, but poorly balanced, with intense red and weak blue, and almost no violet:

The above is a 25W bulb, while the one below is a 100W one. The spectra of these two are almost identical. Both look very yellow to the eye.

Engineers searched desperately for ways to increase the filament temperature, to improve the blue end output. This led to the development of halogen glow bulbs, which use a high pressure halogen gas to recover tungsten that went off the filament, and re-deposit it where it belongs. This allows running the filament of a halogen bulb slightly hotter than that of a common bulb, leading to slightly stronger output in the blue, and thus an overall better balance:

And there was also a brute-force approach to getting whiter light from incandescent bulbs, which was simply to add a blue filter, by tinting the bulb blue. This works not by increasing blue, but by decreasing red! So it does achieve the goal of producing a well balanced light spectrum, but the efficiency, already very low for any glow bulb, simply gets ridiculous! This spectrum, the same I displayed for reference at the beginning,  is of a Philips 60W blue-tinted "daylight" glow bulb:

As you can see, it has a little bit more blue light than the halogen bulb, making it the smoothest light source in my collection, with the best color rendering and visual impression - but also the least energy-efficient one!

For comparison, here is the spectrum of true daylight - the sun, plus blue sky, at mid afternoon, so it should have an average daylight balance. It can be seen that the "daylight" lightbulb, even with its blue glass, is significantly redder than real daylight.
The dark stripes in the sun spectrum are the Fraunhofer lines.

Now that we have seen the spectrally smoothest light sources, let's see the most "adventurous" ones: Gas discharge lamps! They are essentially pure spectral line radiators.

This is the spectrum of neon, obtained from one of those little  neon indicator bulbs, that look - can you guess it? - red/orange:

And here comes the spectrum of of a discharge through mercury vapor, which is used in many lamps. It looks green/blueish:

The spectrum above was obtained from a metal halide lamp, right after switching it on, while it was still cold. The other metals in it don't get excited by the electrical arc, so they don't glow yet. But a few minutes later the lamp has warmed up, and the other metals in it get thermally excited, adding a lot of spectral lines of their own. That's when the light of a metal halide lamp turns white to the eye, and very intense:

It's no wonder that such lamps just cannot have a very good color rendering, right? Most of the visible spectrum gets no light at all! An object reflecting just  narrow ranges of the spectrum might well appear completely black in this light, while it would have bright colors when seen in daylight, or under a glow bulb. And another object that looks red in daylight might look green under a metal halide lamp, simply because it has a strong reflectance in the deep red, like 650-700nm, and a weak one in the center of the green range. Under pure smooth white light, the strong red reflection will dominate over the weak green one, and the object will look bright red, maybe with a slight orange tint. Instead in metal halide light it will look green, because the lamp has no output at all in the deep red range, so that the weak green reflection of the object makes it look perfectly green when lit by the lamp's intense green spectral line!
 
It's a philosophical question whether such incorrect and even unpredictable color rendering is the fault of the lamps, or of any objects, dyes, and pigments that reflect only narrow bandwidths. But my view is that we have far more such objects and dyes and pigments, than we have lamp types, so lamps are easier to control! That's why we should not use any lamps with strong line spectra, whenever we want decent color rendering.
  
There is a special kind of gas discharge lamp, very well known to the photographer: Xenon flash tubes. I made a spectrum of my Pentax AF-280T flash, and once more, I was surprised. Although there are well defined lines, they almost blend into a very smooth, even broadband radiation!

For many years, the fluorescent lamp has been the mainstay of lighting. It's efficient, cheap, long lived, and very versatile. Can one ask for more? Sure! One could ask for good color rendering, which fluorescent lamps just don't do.

A fluorescent lamp is a long glass tube , either straight or bent in various ways, in which mercury vapor is electrically excited. The mercury emits its characteristic spectrum, and a coating of special materials covering the inside of the tube walls converts some of this radiation into other colors. These special materials are called "phosphors" for purely historical reasons, given that modern "phosphors" don't contain any phosphor (chemical element) at all!
These lamps come as straight long tubes, or as bent tubes, to be used with external control circuits (ballasts), or they come as very compactly bent tubes with integrated ballasts, as "compact fluorescent lamps", CFLs, or "energy saving lightbulbs" - even if they are no bulbs but tubes, and they don't save any energy (in absolute terms)  but consume  it, like any lamp does!

Old fluorescent lamps were said (on many websites!) to be "two-color lamps", combining the blue mercury glow with a yellow phosphor. That would produce brutally poor color rendering! Like so many things one can read online, and in books, this is only a half truth. It turns out that this "yellow phosphor" emits a smooth spectrum over a wide bandwidth, spanning from pretty deep in the red, all the way into the cyan and even blue range! The spectrum below, taken from an Oryom 18 watt "warm white" CFL, illustrates this very clearly: The characteristic spectral lines of mercury ride on a very broadband glow centered in the yellow range.

Let's compare this to a more modern, "three-color" CFL, having the same "warm white" color temperature of 2700K. This is from an Osram Dulux Star 8W CFL, that claims a color rendering index better than 80%, which is pretty good - and hard to believe! Although the intense blue/violet mercury line is largely absorbed by that lamp's phosphor, there are many broad areas in the spectrum that get essentially no light. This is very far from a smooth, even spectral distribution!

Let's compare this to a Westinghouse 8W CFL, which is of the "daylight" type, specifiying a color temperature of 6500K.  It's completely obvious that it uses the same phosphors as the lamp above, but has different
quantities of them, so that the red is weaker and the blue is much stronger.

And this is from a Philips Ecohome 23W CFL, also having a color temperature of 6500K. The spectrum is for all practical purposes identical to that of the lamp above. These lamps were all made in China, and even if they weren't made by the same factory, surely the phosphors are made and mixed by the same company...

The last CFL on this page is a General Electric 20W one, which I bought because it is of the rather rare neutral white color, having 4000K color temperature. It's the color I like best, by far. The spectrogram shows nothing unexpected: It also uses the same kinds of phosphors, in a mix that is in between that of the 2700K and the 6500K lamps:

I would like to test the spectrum of any fluorescent lamp with >90% color rendering specs, specially the ones rated for color 940, which means 4000K and above 90% color rendering index. Unfortunately there is no place whithin my reach where I can buy any! I have searched all electrical and hardware stores in the wider area, and there are none! But the catalogs of big manufacturers, such as Osram, do list them. I would expect such high-CRI lamps to fill out the dark areas in the spectrum, and further dampen down the bright ones. Perhaps they even extend their range into the deep red! That would mean less efficiency, of course, because the human eye is less sensitive at the extremes. Nothing comes for free in this world.

Until a few days ago, I believed what I had read on several web sites: That white LEDs are basically blue LEDs coated with a phosphor that adds either yellow light in the cheap ones, or red and green light in the better ones. Now that I'm the proud owner of a spectroscope, I was able to see the truth myself. And here it is, my dear readers! I will start by showing you the spectra of an assortment of white ("daylight") LEDs:

Cheap 3mm 20mA LED bought on eBay:

8mm half-watt LED for general lighting, 160 degrees dispersion, from eBay:

10mm half-watt narrow focused LED, from eBay:

These three seem to use the same kind of phosphor. You could call it "yellow" phosphor, but it's so nicely broadbanded that we could almost call it "white minus blue" phosphor! The exact wavelength of the main blue radiation varies with manufacturer, LED type, and maybe even with the batch. The biggest spectral defect I find with these LEDs is that they have a depression in the cyan range, which is deeper in those LEDs that have their blue output higher up in te spectrum, and narrower. But they all make pretty decent all-color lamps - very much better than CFLs, in any case, let alone metal halide lamps!

Now see this 5mm, 20mA, focused LED, bought locally. The blue is shifted down, it has nothing at the upper edge, and is weak in the mid red - of course it has no deep red output at all. This one has poorer color rendering, but looks extremely bright, thanks to concentrating its output where the human eye is most sensitive:

Let's see the "warm white" LEDs now, those rated generally for color temperatures around 2700K. First, a 3mm 20mA cheapy from eBay:

The 8mm half-watt high dispersion one from eBay:

And the 10mm focused one from eBay. These three are the "warm white" companions to the three eBay LEDs further up.

Clearly these three warm white LEDs follow the same pattern as the three cold ones, in terms of shifting the blue center around depending on the exact model, and having a hole in the cyan. And as is logical to get the low color temperature, they have a lot of red output. But wait - the red looks more extended too! Could it be that these warm white LEDs use two phosphors, one being the same wideband yellow the others use, and the second being an extra deep red phosphor? Or maybe they use  a wideband red and a wideband green phosphor? I don't know. But they are pretty close to being a full-spectrum source, if it were not for the hole in the cyan range!
 
Let's look at bigger LEDs. This is a 1 watt, warm white LED that comes mounted on a star board.  It seems to have a similar blue spectrum, and similar phosphor, as the 5mm LED shown higher in the page, but with more phosphor to give the lower color temperature. It's not as widebanded as one would want:

And this is my most powerful single LED: A Cree X-lamp rated for 5 watts, in a tiny SMD format! It has the color temperature I like so much, 4000K. It has a pretty wideband and very smooth response, almost without a hole in the cyan range! But in the deep red it's weak, and the cheapest LED of all in my collection, the 3mm one from China, is better in that regard...

Finally, there is also another kind of white LED, that uses no phosphor, but instead has a red, a green and a blue LED chip in the same package. Here is the spectrum of one of those. Three narrow bands, and nothing else. These might be quite efficient, but I wouldn't expect great color rendering from them:

Now this is a bit hard to understand for me: I see lots of fluorescent lamps advertised to have better than 80% color rendering index, and in catalogs at least I see some rated at better than 90%, while at the same time white LEDs are typically rated at less than 80%. Judging from my spectrograms, instead, LEDs seem to be far better than CFLs in this regard! Is this a case of incorrect hearsay being propagated so much that it has become an established pseudo-truth, or am I missing something important enough to turn around the whole situation?  

It seems to me that among the light sources shown on this page, some of the white LEDs are the best. Only the blue-tinted incandescent bulb can top the LEDs in terms of spectral smoothness - but there we would be comparing the most efficient to the least efficient light source! Between a modern white LED, and a blue-tinted incandescent, there is roughly a 20:1 ratio in efficiency! With LEDs being among the most efficient lamps in existence, and also among the ones with the smoothest spectral distribution, and apparently being the only lamps that combine both, they are the way to go!

Clearly CFLs have very irregular spectra, and metal halide lamps are even worse.

My home is currently lit almost exlusively by fluorescent lamps of different types. It seems time now to replace them by LEDs. But not just by any LEDs. It's probably a good idea to buy a few samples of several different types and brands, test them all, and then decide which to buy in quantity for the home.

Algebraic topology is a field of mathematics which is usually presented as the study of certain amusing shapes, like the Klein bottle or Möbius strip. But this presentation perhaps fails to convey just how vital algebraic topology is to almost all areas of modern mathematics.

To give the reader a taste of this importance, in today's post we will start by exploring a funny property of the Möbius strip, and then show how the same funny phenomenon creeps up when you try to solve certain differential equations.

So, what is the Möbius strip, and what funny property does it have?

The Möbius strip

Take a sheet of paper. If you tape two opposite edges together, then you will produce a cylinder. If you instead twist one of the edges before gluing, you get a Möbius strip.

The Möbius strip doesn't seem so different from a cylinder at first, but there is one very strange property of it: it only has one side!

But what does that mean, exactly? Specifying a side of the Möbius strip is the same as deciding which side of the paper is `up' and which side is `down.' These decisions need to be locally consistent: nearby points should agree on which way is up and which is down. In other words, if I'm an ant on one side of the Möbius strip, if I walk a couple steps I should still be on the same side.

This suggests it should be pretty easy to find two sides of any surface: pick one point to start with, and arbitrarily label one side `up' and one side `down.' Then just move around, copying your meaning of up.

Here's a little animation of this; we start with a notion of up at one point (illustrated by an upwards pointing vector), and then move the vector around to nearby points, to give them a consistent notion of up.

Unfortunately, this procedure has a small hiccup. If we take a long journey, in a loop going around the entire Möbius strip, the notion of up flips when we get back to where we started! We made a small animation to see this; after the animation plays, you can drag the Möbius strip to rotate it around, to better see the full path.

This is why we cannot define a notion of up or down on the Möbius strip; while in small patches it makes sense, if you travel all around the Möbius strip you'll end up reversing your notion of up and down! This is why people say a Möbius strip has only one side; while at every point on the strip, it looks like there's a `top' and a `bottom,' if you go around the entire strip you'll find that the top and bottom switch roles. If you have a Möbius strip made out of paper, you can observe this in the real world: trace your finger around a Möbius strip, and you'll find that if your finger started off on `top' of the strip, it will end on the `bottom.'

A problem in differential equations

With the Möbius strip introduced, let's turn to a problem of calculus.

The great mathematician Riemann was interested in solving differential equations over the complex numbers. As an example, consider the equation \[\frac{df}{dz} = \frac{1}{2z}f(z).\]

Riemann was interested in finding complex solutions to this differential equation. It's easy to solve differential equations of this type by trying to find the Taylor series of \(f(z).\) Let's explain how. First, observe that the right hand side of the equation involves division by \(2z\); division by 0 is scary, so let's look for a solution near \(z=1.\) Then we can write the Taylor series of \(f(z)\) as \[f(z) = a_0 + a_1(z-1) + a_2(z-1)^2 + a_3(z-1)^3 + \cdots,\] for some coefficients \(a_0, a_1, a_2, ...\) which we need to solve for. I'll remark now that this is a first order differential equation, so it has one degree of freedom. Thus we can actually make \(a_0\) take any value we want; to simplify the below computations, we'll set \(a_0 = 1.\) In other words, we're giving our differential equation the initial condition that \(f(1) = 1.\)

The equation \[\frac{df}{dz} = \frac{1}{2z}f(z)\] is equivalent to \[2z\frac{df}{dz} = f(z).\] If we term-by-term differentiate our Taylor series, then we find \[\frac{df}{dz} = a_1 + 2a_2(z-1) + 3a_3(z-1)^2 + \cdots.\]

Multiplying the above expression by \(2z\) is a little tricky, because \(2z\) is a power series in the variable \(z,\) but \(df/dz\) is a power series centered at \(z=1.\) So, before doing the multiplication, we rewrite \[2z = 2 + 2(z-1).\] Thus \[2z\frac{df}{dz} = (2 + 2(z-1)) \cdot (a_1 + 2a_2(z-1) + 3a_3(z-1)^2 + \cdots),\] so that, when we distribute this product out, \[2z\frac{df}{dz} = 2a_1 + (2a_1 + 4a_2) \cdot (z-1) + (4a_2 + 6a_3) \cdot (z-1)^2 + \cdots\]

Thus our equation \[2z\frac{df}{dz} = f(z)\] becomes \[\begin{align*} &\hspace{5mm}2a_1 + (2a_1 + 4a_2) \cdot (z-1) + (4a_2 + 6a_3) \cdot (z-1)^2 + \cdots \\ &= 1 + \hspace{17mm}a_1 \cdot (z-1) + \hspace{14mm}a_2 \cdot (z-1)^2 + \cdots.\end{align*}\] Equating the coefficients of the \((z-1)^n\) terms on both sides, we get the equations \[2a_1 = 1,\] \[2a_1 + 4a_2 = a_1,\] \[4a_2 + 6a_3 = a_2,\] and so on.

From the first equation \(2a_1 = 1,\) we get \(a_1 = 1/2.\) But now that we know \(a_1 = 1/2,\) we can substitute this value of \(a_1\) into the second equation \[2a_1 + 4a_2 = a_1,\] to rewrite it as \[1 + 4a_2 = \frac{1}{2},\] implying \[a_2 = -\frac{1}{8}.\]

Similarly, now that we know \(a_2 = -1/8,\) we can plug it into the third equation \[4a_2 + 6a_3 = a_2\] to get \[-\frac{4}{8} + 6a_3 = -\frac{1}{8},\] giving \[a_3 = \frac{1}{16}.\]

We can solve for the Taylor coefficients of \(f(z)\) recursively, by just following our string of equations. Thus you can get as many Taylor coefficients for the solution \(f(z)\) to \[\frac{df}{dz} = \frac{1}{2z}f(z)\] as you'd like!

While this method is great at finding a solution for \(z=1,\) the downside to Taylor series is that they don't always converge. It turns out that this particular Taylor series has radius of convergence 1, which means for any complex number \(z\) closer than 1 unit to \(z=1,\) the formal power series solution we just found will converge.

Unfortunately, for values of \(z\) outside of the highlighted blue circle, the formal power series solution we just found will not always converge.

As an example, if we plug in \(z = -1\) to our series we get \[f(-1)=1 - 1 - \frac{1}{2} - \frac{1}{2} - \frac{5}{8} - \frac{7}{8} - \frac{21}{16} - \cdots.\] Observe how, after the initial \(1-1,\) the terms just get more and more negative; it turns out that this continues to happen, and so \[f(-1) = -\infty,\] as we keep subtracting larger and larger numbers.

This is annoying! We would love to solve our differential equation on the entire complex plane, but so far we can only solve it in a tiny disk.

However, note that we found this power series by doing term-by-term expansion centered around the point \(z=1.\) There's no reason we couldn't just do term-by-term expansion centered around a different point.

For example, let's take a point \(\alpha = 0.6 + 0.8i,\) which is near the edge of our blue circle. Using our first power series solution, we find that \(f(\alpha) \approx 0.8944 + 0.4472i.\) Using this as the initial condition, we can use the recurrence method from before to find a formal power series solution to \[\frac{df}{dz} = \frac{1}{2z}f(z)\] centered at \(\alpha.\)

In the widget below, you can click on points to plug in values.

In the widget, we display two different formal power series solutions to our ODE, each centered around different points. The first one is the same one from before, and converges inside the blue circle; the second one is new, and converges inside the purple circle.

If you click on a point \(z\) in the blue circle, your computer will use the formal power series to calculate \(f(z)\) using our power series from before; and if you click on a point \(z\) in the purple circle, your computer will use the new power series to calculate \(f(z).\)

Click on different points of the intersection of the two circles, and compare the values of the two different power series. You should find that they are exactly the same, up to rounding errors (these are infinite sums, but your computer can only do finitely many terms!).

So now we have a solution to \[\frac{df}{dz} = \frac{1}{2z}f(z)\] which obeys \(f(1) = 1,\) and makes sense in the union of the blue and purple circles: in the purple circle use the new formula, in the blue circle use the old formula, and on the overlap the two formulas give the same answer, so it doesn't matter which you use!

Monodromy

Let's recap what just happened: we found a solution to \[\frac{df}{dz} = \frac{1}{2z}f(z)\] inside a small blue circle, by expanding \[f(z) = a_0 + a_1(z-1) + a_2(z-1)^2 + \cdots\] as a formal power series around \(z=1\); the differential equation became a recurrence on the coefficients \(a_n,\) and the initial condition \(f(1) = 1\) just became the initial condition \(a_0 = 1\) of our recurrence. The resulting power series only made sense on a small blue disk, but we wanted a solution over the entire complex plane -- so, we repeated our recurrence procedure, but starting with the different center value \(0.6 + 0.8i\) instead. The differential equation again turns into a recurrence relation, and we can get the initial condition of our recurrence using the original formula to calcualte \(f(0.6 + 0.8i).\)

This extended our solution from the blue circle to the union of the blue circle and the purple circle. Since this worked so well, let's go further: pick a point near the edge of the purple circle, and find a power series solution centered at that point! And then do this again and again and again, until we've covered the entire complex plane with circles!

Unfortunately, while this plan works great in theory, something peculiar happens when we implement it. To see why, consider the following widget. In it, we repeated our procedure `pick a point near the boundary and solve the differential equation centered there', giving us a total of seven different formulas for \(f(z),\) each of which makes sense on a different circle. If you click a point in any of the circles, your computer will calculate the value of the value of \(f(z)\) using any of the formulas which make sense (so, if your point is in exactly one circle, it will use that circle's formula, but if your point is in two circles, it will show you the answers you get using the formulas associated to both circles).

There are a lot of circles on screen, so try not get overwhelmed -- just click a few points and see the values!

Most of time, if you pick a point in two circles, you'll get approximately the same answer with each formula -- and any discrepancy is only because the computer is only adding a handful of terms of these infinitely long sums, and isn't using high precision arithmetic (which matters a lot when adding decimals on a computer).

But if you click on a point in the overlap of the red and blue circles -- that is, a point just below \(z = 1\) -- then something strange will happen: the blue formula will give you a value which is close to \(+1,\) and the red formula will give a value close to \(-1.\) This is a big gap -- much too big to be explained away by your computer truncating the sum, or by precision loss in floating-point arithmetic.

What's happening? To understand this, let's first try understanding why our two different formulas on the purple and blue circle agreed in the first place.

Revisiting uniqueness

Ordinary differential equations have a strong local uniqueness theorem: any two solutions of an equation like \[\frac{df}{dz} = \frac{1}{2z} \cdot f(z)\] which agree at one point automatically agree in a small neighborhood of that point. Let's explore what this means, why this is true, and how it helps us understand what was happening above.

To be a bit more concrete, imagine that \(f_0\) and \(f_1\) are two functions which both obey \[f_0'(z) = \frac{1}{2z}f_0(z),\] \[f_1'(z) = \frac{1}{2z}f_1(z),\] and suppose we know that \(f_0(1) = f_1(1).\) Then in fact \(f_0(z) = f_1(z)\) for all values of \(z\) which are close to 1.

Why is that the case? It's actually for the exact same reason we saw before: power series expansions. More precisely, for values of \(z\) close enough to 1, the functions \(f_0(z)\) and \(f_1(z)\) are both given by power series expansions \[f_0(z) = f_0(1) + f_0'(1) \cdot (z-1) + \frac{f_0''(1)}{2} \cdot (z-1)^2 + \cdots,\] \[f_1(z) = f_1(1) + f_1'(1) \cdot (z-1) + \frac{f_1''(1)}{2} \cdot (z-1)^2 + \cdots.\]

Set \(A := f_0(1).\) As \(f_1(1) = f_0(1),\) we have \(f_1(1) = A\) as well.

From our differential equation, we find that \[f_0'(1) = f_0(1)/2 = A/2,\] and similarly \(f_1'(1) = A/2.\) And differentiating \[f'(z) = \frac{f(z)}{2z},\] we find \[f''(z) = \frac{f'(z)}{2z} - \frac{f(z)}{2z^2},\] so that \[f''_0(1) = \frac{A}{4} - \frac{A}{2} = -\frac{A}{4},\] and similarly \(f''_1(1) = -A/4.\) Thus all the Taylor coefficients of \(f_0\) and \(f_1\) coincide, and so they must agree in some small disk around \(z=1\) (this disk being the disk on which the Taylor series converges).

How does this matter for our blue and purple circle? The point is that we have two solutions to the differential equation which agree at \(0.6 + 0.8i,\) and hence they agree in whatever region has the following two properties: (i) both solutions make sense there, and (ii) the formal power series solution at \(0.6 + 0.8i\) converges there. But that region is exactly the overlap of the purple and blue circles, hence why the two different formulas give the same answer there.

At the end of the slide show, we compare the solutions on the red and aqua circles. But what happens if we want to go back around, and compare the solutions on the red disk and the original blue disk?

Here, we run into a problem: we have no common initial condition! The solution on the red disk was defined using the red point \(0.752 - 0.659i,\) which was located at the edge of the aqua disk. And so, while the red and blue disks overlap, there is no reason for them to have a common value at any point; and indeed, as we saw in the previous section, they don't!

The trouble here is that the local uniqueness theorem for differential equations requires you to know one common value in order to conclude that the two solutions coincide -- but there's no common value we can use for the red and blue disks.

Return of the Möbius strip

Recall that, at the start of this article, we highlighted a funny property of the Möbius strip: while you could always locally orient it, globally you cannot label one side as up and one side as down. If you try to label one side as up, and then you move continuously around the Möbius strip, you'll end up labelling that same side as down!

This is very similar to what happened with our differential equation: we tried to find a solution, and while we could always do this locally, on small patches of the complex plane, we found that if we went in a circle around the origin, our solution got negated -- instead of \(f(1) = 1,\) if we continue around the origin, we found a solution with \(f(1) = -1.\) This \(+1\) switching to a \(-1\) is just like how, if you go around a Möbius strip, up becomes down.

This analogy has a surprising amount of substance to it. In complex analysis, and other advanced parts of mathematics, it is very useful to visualize the space of solutions to a linear differential equation like \[\frac{df}{dz} = \frac{1}{2z}f(z)\] as a vector bundle, a certain type of geometric object. What do we mean by this? How do we view the set of all solutions to a differential equation as a geometric object?

Note that, for any nonzero complex number \(a,\) we can use the power series method from before to find a formal power series solution to \(df/dz = f(z)/2z\) centered at \(z=a,\) with any initial condition we want; that is, \(f(a)\) can take on any value we please. (The trouble with \(a=0\) is that the expression \(1/2z\) doesn't make sense at \(0,\) which means the power series method won't work.)

Thus, at least near any individual complex number \(a,\) there is a complex plane worth of solutions to \(df/dz = f(z)/2z,\) because every complex number \(\zeta\) gives us a different possible initial condition \(f(a) = \zeta,\) and hence a different solution.

We can then create a shape by taking each point of the complex plane and attaching the space of initial conditions to our ODE to it. The final shape is surprisingly complicated; to understand it, we're going to focus on what it looks like above the unit circle in the complex plane: that is, we're going to attach, to each complex number \(z\) with \(|z| = 1,\) the 2-dimensional space of initial conditions for the ODE at that point.

At first, you might think that if you attach a 2-dimensional plane (the space of initial conditions) to each point of a circle, the final space you'd get would look something like this:

However, this is not quite what the true space of solutions looks like! Remember: when we move around the circle, the solution with \(f(1) = 1\) turns into the solution with \(f(1) = -1.\) This means that the planes we attach at each point need to twist around as we attach them -- this way, when you loop around the circle, the initial condition \(+1\) becomes the initial condition \(-1.\)

This twisting is quite hard to visualize in 3-dimensions. To get a sense of what it looks like, let's pretend instead of having a 2-dimensional space of initial conditions, we only had a 1-dimensional space. Then, instead of planes rotating and twisting around in space, we'd have a line rotating and twisting around in space. If we think about the shape such a line would trace out, it is just a Möbius strip! You can see for yourself in the animation below:

So, the Möbius strip is literally part of the vector bundle of solutions to our differential equation! The full vector bundle would be obtained by rotating a plane instead of a line, but this is harder to draw in 3-dimensional space.

The punchline to all this, though, is that the strange Möbius strip arises naturally from the solutions to a differential equation!

This geometric approach to differential equations -- that is, thinking about the vector bundle of all solutions to a linear ODE -- is incredibly helpful in modern complex analysis, as it allows the tools of algebraic geometry and algebraic topology to say things about differential equations. In this example, the non-orientability of the Möbius strip is essentially equivalent to the fact that our differential equation had no global solution (for experts, both of these facts are just us asserting that the vector bundle of solutions to our ODE is the complex line bundle on \(S^1\) corresponding to the group homomorphism \(\pi_1(S^1) \to \mathbb{C}^{\times}\) which sends a single loop to \(-1\)).

The Möbius strip, and related wacky shapes, were not just invented for amusement (though amusing they are): they have helpful mathematical purposes.

For many years, I sometimes discovered that commands I was sure I had run were no longer present in my Z shell history file (~/.zsh_history). In this article, I will show you how I tracked down the bug. Spoiler: ultimately, patching Zsh to make it crash loudly and analyzing the crash’s core dump was the winning strategy!

The good news first

Zsh 5.9.2 (released July 12th, 2026) contains a fix for this issue — open it after reading this investigation to not spoil the fun.

Spoiler: link to the upstream fix

Zsh fix 53454

The symptom

Occasionally, I noticed that commands I knew I executed the day before were not findable in my shell history, meaning pressing Ctrl+R for backward history search yielded no results. Whenever I noticed this, my shell history file contained only very old entries, with years of newer entries missing.

The first few times this happened I just restored my shell history from my daily backup and did not bother investigating any further. But the issue kept happening.

I noticed that there was no visible corruption in the .zsh_history (no non-printable characters or incomplete lines of text), and that the number of lines in the file was not always the same.

What was not clear to me was whether it was Zsh itself, or some other program, or perhaps the combination of multiple zsh(1) processes that caused the issue.

My Zsh history config

I set the following history-related options in my ~/.zshrc:

# Load 4000 lines of history (for Ctrl+R backward search), but save O(∞)
HISTSIZE=4000
HISTFILE=~/.zsh_history
SAVEHIST=10000000

# Do not save (adjacent) duplicate entries
setopt HIST_IGNORE_DUPS

# Append history entries to `~/.zsh_history` when commands are run.
setopt INC_APPEND_HISTORY
# …but do not share history (enabled by default in NixOS’s /etc/zshrc).
unsetopt SHARE_HISTORY

In practice, this means my shells are separate sessions that all stream their commands into a shared ~/.zsh_history. The history is intentionally not shared, so when I want to access entries that another shell wrote, I explicitly run exec zsh.

Tracing the behavior

When I asked for help on Mastodon in December 2024 (mostly in the hope that somebody else already encountered and diagnosed this problem), one suggestion I got was to use file system change monitoring mechanisms like inotify or fsevents to find the culprit that truncates (or changes?) the Zsh history file.

The next sections walk through the available options on Linux which I tried.

inotify

The inotify(7) Linux kernel subsystem is one of the oldest file system change monitoring APIs available in Linux (released 2005). To get a good understanding of how Zsh modifies the history file, it is not sufficient to monitor just .zsh_history:

midna ~ % inotifywait --monitor .zsh_history
Setting up watches.
Watches established.
.zsh_history OPEN
.zsh_history ACCESS
.zsh_history ACCESS
[…]
.zsh_history ACCESS
.zsh_history CLOSE_NOWRITE,CLOSE
.zsh_history ATTRIB
.zsh_history CLOSE_WRITE,CLOSE
.zsh_history DELETE_SELF
^C

The file is opened, accessed (= read) and then… deleted?!

By monitoring the containing directory, we see the whole picture:

midna ~ % inotifywait --monitor ~
/home/michael/ OPEN .zsh_history
/home/michael/ ACCESS .zsh_history
/home/michael/ ACCESS .zsh_history
[…]
/home/michael/ ACCESS .zsh_history
/home/michael/ CLOSE_NOWRITE,CLOSE .zsh_history
/home/michael/ CLOSE_WRITE,CLOSE .zsh_history
/home/michael/ OPEN .zsh_history
/home/michael/ CLOSE_WRITE,CLOSE .zsh_history
/home/michael/ OPEN .zsh_history
/home/michael/ ACCESS .zsh_history
/home/michael/ CLOSE_NOWRITE,CLOSE .zsh_history
/home/michael/ CREATE .zsh_history.new
/home/michael/ OPEN .zsh_history.new
/home/michael/ ATTRIB .zsh_history.new
/home/michael/ MODIFY .zsh_history.new
/home/michael/ CLOSE_WRITE,CLOSE .zsh_history.new
/home/michael/ MOVED_FROM .zsh_history.new
/home/michael/ MOVED_TO .zsh_history
/home/michael/ CLOSE_WRITE,CLOSE .zsh_history

So Zsh reads the old history file contents, writes them to a new file, then renames the new file over the old one, thereby deleting the old one. Now it makes sense!

Unfortunately, we do not see the process IDs (PIDs) of the responsible process for the file system event, not even with the sibling utility fsnotifywait(1) , which uses fanotify(7) , an API that does provide this information! I checked, and the kernel does send the PID, but fsnotifywait does not display the PID.

fatrace

Luckily, there is fatrace(8) , which does display the process name and PID.

Here is what Zsh’s history rewriting looks like with fatrace(8) :

zsh(197994): CWO /home/michael/.zsh_history
zsh(197994): O   /home/michael/.zsh_history
zsh(197994): R   /home/michael/.zsh_history
zsh(197994): R   /home/michael/.zsh_history
[…]
zsh(197994): R   /home/michael/.zsh_history
zsh(197994): C   /home/michael/.zsh_history
zsh(197994): +   /home/michael
zsh(197994): O   /home/michael/.zsh_history.new
zsh(197994): W   /home/michael/.zsh_history.new
zsh(197994): W   /home/michael/.zsh_history.new
zsh(197994): W   /home/michael/.zsh_history.new
[…]
zsh(197994): W   /home/michael/.zsh_history.new
zsh(197994): CW  /home/michael/.zsh_history.new
zsh(197994): <>  /home/michael
zsh(197994): CW  (deleted)
zsh(197994): C   /nix/store/80vwnjjgcrbp41pk927r8lzybjhy0k73-zsh-5.9.1/bin/zsh
[…]

This gives us the PID, so now we can verify whether multiple processes were involved in corrupting the shell history. But, we don’t have any insight into how much data each Zsh PID is reading/writing, so even with a fatrace log, it would still not be clear what happened.

strace

Of course, one could use strace(1) , in particular with its -k flag, to further look into Zsh behavior, but it seems like a logistical nightmare to arrange for every (interactive) Zsh process to get a corresponding strace run, and I was not sure if always-stracing a shell changes behavior in subtle ways, so I did not pursue the strace route.

(Once I had a reproducer, strace became easy enough to use and very helpful.)

bpftrace

To get more visibility into Zsh’s read and write operations, we can reach for bpftrace(8) .

To get started, I created the following bpftrace program, which is run on every open(2) syscall and logs which process opened the .zsh_history file, including the user stack trace:

tracepoint:syscalls:sys_enter_open,
tracepoint:syscalls:sys_enter_openat,
tracepoint:syscalls:sys_enter_openat2
/str(args.filename) == "/home/michael/.zsh_history" || str(args.filename) == ".zsh_history"/
{
	printf("%-6d %-16s open(%s)%s", pid, comm, str(args.filename), ustack);
}

On NixOS 26.05, I can run the program as follows:

midna ~ % nix shell nixpkgs#bpftrace
midna ~ 2 % sudo bpftrace path.bt
Attached 3 probes
212030 zsh              open(/home/michael/.zsh_history)
        __internal_syscall_cancel+142
        __syscall_cancel+20
        __libc_open64+87
        lockhistfile+642
        readhistfile+2213
        zsh_main+1118
        __libc_start_call_main+117
        __libc_start_main_alias_2+136
        _start+37
212030 zsh              open(/home/michael/.zsh_history)
        __internal_syscall_cancel+142
        __syscall_cancel+20
        __libc_open64+87
        _IO_file_open+51
        _IO_file_fopen@@GLIBC_2.2.5+303
        __fopen_internal+134
        readhistfile+2277
        zsh_main+1118
        __libc_start_call_main+117
        __libc_start_main_alias_2+136
        _start+37
212030 zsh              open(/home/michael/.zsh_history)
        __internal_syscall_cancel+142
        __syscall_cancel+20
        __libc_open64+87
        lockhistfile+642
        savehistfile+165
        zexit+204
        __libc_start_call_main+117
        __libc_start_main_alias_2+136
        _start+37
212030 zsh              open(/home/michael/.zsh_history)
        __internal_syscall_cancel+142
        __syscall_cancel+20
        __libc_open64+87
        savehistfile+752
        zexit+204
        __libc_start_call_main+117
        __libc_start_main_alias_2+136
        _start+37
212030 zsh              open(/home/michael/.zsh_history)
        __internal_syscall_cancel+142
        __syscall_cancel+20
        __libc_open64+87
        _IO_file_open+51
        _IO_file_fopen@@GLIBC_2.2.5+303
        __fopen_internal+134
        readhistfile+2277
        savehistfile+2498
        zexit+204
        __libc_start_call_main+117
        __libc_start_main_alias_2+136
        _start+37
^C

Encouraged by this early success, I extended the program as follows to cover more system calls:

Full zshhisttrace.bt bpftrace code

#!/usr/bin/bpftrace
#include <fcntl.h>
#include <limits.h>

tracepoint:syscalls:sys_enter_open /comm == "zsh"/ {
     printf("%s(%d) open: %s flags %x mode %x\n", comm, pid, str(args->filename), args->flags, args->mode);
}

tracepoint:syscalls:sys_enter_openat {
     if (!strcontains(str(args->filename), "zsh_history")) {
         delete(@openfn[tid]);
         return;
     }
     @openfn[tid] = 1;
     printf("%s(%d) openat: ", comm, pid);
     if (args->dfd < 0x7fffffff) { /* ought to be != AT_FDCWD, but that does not work !?!? */
         printf("[at fd %d]", args->dfd);
     }
     printf("%s flags %x mode %x\n", str(args->filename), args->flags, args->mode);
}

tracepoint:syscalls:sys_exit_openat /@openfn[tid]/ {
     @reads[tid,(int64)args->ret] = 1; // TODO: bpftrace 0.22 introduces has_key
     @writes[tid,(int64)args->ret] = 1; // TODO: bpftrace 0.22 introduces has_key
}

tracepoint:syscalls:sys_enter_close /@reads[tid,(int64)args->fd]/ {
     printf("%s(%d) close %d (reads: %d, writes: %d)\n", comm, pid, args->fd, @reads[tid,(int64)args->fd]-1, @writes[tid,(int64)args->fd]-1);
     delete(@reads[tid,(int64)args->fd]);
     delete(@writes[tid,(int64)args->fd]);
}

// tracepoint:syscalls:sys_enter_openat2 /comm == "zsh"/ {
//      printf("%s(%d) openat: ", comm, pid);
//      if (args->dfd < INT_MAX) { /* ought to be != AT_FDCWD, but that does not work !?!? */
//          printf("[at fd %d]", args->dfd);
//      }
//      printf("%s \n", str(args->filename));
// }

tracepoint:syscalls:sys_enter_rename /comm == "zsh"/ {
     printf("%s(%d) rename:", comm, pid);
     printf("%s -> %s\n", str(args->oldname), str(args->newname));
}

tracepoint:syscalls:sys_enter_symlink /comm == "zsh"/ {
     printf("%s(%d) symlink ", comm, pid);
     printf("%s -> %s\n", str(args->oldname), str(args->newname));
}

tracepoint:syscalls:sys_enter_unlink /comm == "zsh"/ {
     printf("%s(%d) unlink ", comm, pid);
     printf("%s\n", str(args->pathname));
}

tracepoint:syscalls:sys_enter_unlinkat /comm == "zsh"/ {
     printf("%s(%d) unlinkat ", comm, pid);
     printf("%s\n", str(args->pathname));
}

tracepoint:syscalls:sys_enter_lseek /comm == "zsh"/ {
     printf("%s(%d) lseek fd %d offset %d whence %d\n", comm, pid, args->fd, args->offset, args->whence);
}

tracepoint:syscalls:sys_enter_read /@reads[tid,(int64)args->fd]/ {
     // printf("%s(%d) read fd %d size %d\n", comm, pid, args->fd, args->count);
     @reads[tid,(int64)args->fd] += args->count;
}

tracepoint:syscalls:sys_exit_read /comm == "zsh"/ {
     if (args->ret <= 0) {
          printf("%s(%d) read = %d\n", comm, pid, args->ret);
     }
}

tracepoint:syscalls:sys_exit_write /comm == "zsh"/ {
     if (args->ret <= 0) {
          printf("%s(%d) write = %d\n", comm, pid, args->ret);
     }
}

tracepoint:syscalls:sys_enter_write /@writes[tid,(int64)args->fd]/ {
     // printf("%s(%d) write fd %d size %d\n", comm, pid, args->fd, args->count);
     @writes[tid,(int64)args->fd] += args->count;
}

In case you want to dive deeper into bpftrace, here are a few resources I found useful:

• The upstream bpftrace docs
• The blog post “First steps in system-wide Linux tracing” by Martin Pitt (2020)
• The LSFMM presentation “BPF Observability” by Brendan Gregg (2019)

I created a systemd unit to run this program in the background permanently (seems cheap enough), meaning I can check the logs like so:

midna % journalctl -fu zshhisttrace
cp(2338700) close 3 (reads: 3407872, writes: 0)
zsh(231222) symlink /pid-231222/host-midna -> /home/michael/.zsh_history.LOCK
zsh(231222) openat: /home/michael/.zsh_history flags 541 mode 180
zsh(231222) close 3 (reads: 0, writes: 0)
zsh(231222) openat: /home/michael/.zsh_history flags 0 mode 0
zsh(231222) lseek fd 3 offset 0 whence 1
zsh(231222) read = 0
zsh(231222) close 3 (reads: 52895744, writes: 0)
zsh(231222) unlink /home/michael/.zsh_history.new
zsh(231222) openat: /home/michael/.zsh_history.new flags c1 mode 180
zsh(231222) close 3 (reads: 0, writes: 52888907)
zsh(231222) rename:/home/michael/.zsh_history.new -> /home/michael/.zsh_history
zsh(231222) unlink /home/michael/.zsh_history.LOCK

One day, I noticed my shell history was truncated and checked the logs. This is what I found. Note how there is no read = 0 line, i.e. Zsh does not read until EOF:

zsh(231233) symlink /pid-231233/host-midna -> /home/michael/.zsh_history.LOCK
zsh(231233) openat: /home/michael/.zsh_history flags 541 mode 180
zsh(231233) close 3 (reads: 0, writes: 0)
zsh(231233) openat: /home/michael/.zsh_history flags 0 mode 0
zsh(231233) lseek fd 3 offset 0 whence 1
zsh(231233) lseek fd 3 offset 0 whence 1
zsh(231233) lseek fd 3 offset 11572944 whence 0
zsh(231233) close 3 (reads: 11575296, writes: 0)
zsh(231233) unlink /home/michael/.zsh_history.new
zsh(231233) openat: /home/michael/.zsh_history.new flags c1 mode 180
zsh(231233) close 3 (reads: 0, writes: 11572944)
zsh(231233) rename:/home/michael/.zsh_history.new -> /home/michael/.zsh_history
zsh(231233) unlink /home/michael/.zsh_history.LOCK

Making it crash!

From the bpftrace output above we know that Zsh is rewriting my .zsh_history file incorrectly: it reads fewer lines than usual, and then correctly writes them to .zsh_history.new.

At this point I decided to study the code for reasons why readhistfile would not read the full history file or why savehistfile would not write the full history file.

The control flow of savehistfile is pretty hard to follow, but it is easy to modify the code (zsh-5.9.1) such that it crashes after it writes a .zsh_history.new with fewer than 50000 lines and before it replaces my .zsh_history with that truncated new file:

--- i/Src/hist.c
+++ w/Src/hist.c
@@ -2994,6 +2994,7 @@ savehistfile(char *fn, int err, int writeflags)
     if (out) {
 	char *history_ignore;
 	Patprog histpat = NULL;
+	int lines_written = 0;
 
 	pushheap();
 
@@ -3048,6 +3049,7 @@ savehistfile(char *fn, int err, int writeflags)
 		ret = fputc(' ', out);
 	    if (ret < 0 || (ret = fputc('\n', out)) < 0)
 		break;
+	    lines_written++;
 	}
 	if (ret >= 0 && start && writeflags & HFILE_USE_OPTIONS) {
 	    struct stat sb;
@@ -3062,6 +3064,10 @@ savehistfile(char *fn, int err, int writeflags)
 	}
 	if (fclose(out) < 0 && ret >= 0)
 	    ret = -1;
+	if (tmpfile && lines_written < 50000) {
+	    char *crashptr = (char*)0x23;
+	    *crashptr = 42;
+	}
 	if (ret >= 0) {
 	    if (tmpfile) {
 		if (rename(tmpfile, unmeta(fn)) < 0) {

On Linux, the easiest way to ensure such a crash ends up somewhere useful is to install systemd-coredump(8) , after which systemd will automatically collect core dumps. You can use coredumpctl(1) to list and work with them. Note that these core dumps contain your shell history, so do not upload them to third-party services. Fedora’s ABRT seems to only send micro reports (i.e. without your full shell history), and Ubuntu’s Apport is disabled-by-default, but it’s worth double-checking.

I installed my patched version of Zsh (with debug symbols enabled) and deferred further investigation until I had a core dump of the issue in action. Sure enough, when I checked with coredumpctl a few days later, I saw a crash! This was the backtrace:

midna % coredumpctl debug
gdb $ bt full
#0  0x000056040d781e19 in savehistfile (fn=0x56040f7a76b0 "/home/michael/.zsh_history", err=1, writeflags=0) at hist.c:3086
        crashptr = 0x23 <error: Cannot access memory at address 0x23>
        history_ignore = 0x0
        histpat = 0x0
        lines_written = 45546
        t = 0x5604102a1f59 ""
        tmpfile = 0x5604100ec210 "/home/michael/.zsh_history.new"
        start = 0x5604102a1f40 "make -j32"
        out = 0x56040f939400
        he = 0x0
        xcurhist = 45546
        extended_history = 0
        ret = 10
#1  0x000056040d781f72 in savehistfile (fn=0x56040f7a76b0 "/home/michael/.zsh_history", err=1, writeflags=32771) at hist.c:3121
        remember_histactive = 0
        history_ignore = 0
        histpat = 0x0
        t = 0x0
        start = 0x0
        out = 0x56040f939400
        he = 0x0
        xcurhist = 51183
        extended_history = 0
        ret = 0
#2  0x000056040d751197 in zexit (val=0, from_where=ZEXIT_NORMAL) at builtin.c:6055
        writeflags = 32768
#3  0x000056040d7888e2 in zsh_main (argc=2, argv=0x7ffd370c1758) at init.c:1950
        errexit = 0
        t = 0x7ffd370c1768
        runscript = 0x0
        zsh_name = 0x7ffd370c26bd "zsh"
        cmd = 0x0
        zsh_main+1522
        __libc_start_call_main+117
        __libc_start_main_alias_2+136
        _start+37
#3  0x000056040d7888e2 in zsh_main (argc=2, argv=0x7ffd370c1758) at init.c:1950
        errexit = 0
        t = 0x7ffd370c1768
        runscript = 0x0
        zsh_name = 0x7ffd370c26bd "zsh"
        cmd = 0x0
        zsh_main+1522
        __libc_start_call_main+117
        __libc_start_main_alias_2+136
        _start+37
#4  0x000056040d735d89 in main (argc=2, argv=0x7ffd370c1758) at ./main.c:93
No locals.

I returned to the source and realized that most likely, savehistfile is just writing out a shorter history file because readhistfile left it a shorter history!

The control flow of readhistfile is easier to follow. Reading through the function, there is one possibility of an early return: when Zsh receives a signal, the read loop is aborted via a break;:

	// …
	if (errflag & ERRFLAG_INT) {
		/* Can't assume fast read next time if interrupted. */
		lasthist.interrupted = 1;
		break;
	}
	// …

Let’s see what errflag and lasthist.interrupted contain in our crash:

gdb $ p errflag
$1 = 2
gdb $ p lasthist.interrupted
$2 = 1

Bingo! So some signal must be involved.

For reasons outside of the scope of this article, I am using a mosh session from which I am starting a long-running SSH session, over which I multiplex further sessions. When tearing down this setup at the end of each workday, I press Ctrl+D in the multiplexed sessions (sends EOF, exits the session), then Ctrl+C on the long-running SSH, then Ctrl+D to exit the mosh session.

(If you don’t cleanly exit a mosh session, it sticks around on the server and subsequent logins tell you about these orphaned sessions. I wanted to avoid accumulating orphaned sessions.)

So in practice I press Ctrl+D, Ctrl+C, Ctrl+D, Ctrl+C etc. until all windows are gone. As part of that sequence, most likely I am exiting a Zsh session (Ctrl+D) and then interrupting (Ctrl+C) its readhistfile if history rewriting takes long enough.

With these clues, I built a standalone reproducer and sent a bug report to the zsh-workers mailing list in March 2025. Bart Schaefer looked into it and posted a fix in April 2025 (thank you!).

It took a long time for the fix to actually be released because there was a long time without any Zsh releases. And then, when the 5.9.1 release happened, it turns out Bart’s fix was missed by the release engineer! I pointed out this oversight, and Zsh 5.9.2 thankfully includes the fix.

I have been running Zsh 5.9 with Bart’s patch applied, and will keep that version pinned until 5.9.2 lands on my computers. If you’re pinning zsh on Debian, pin both, the zsh and zsh-common packages. Otherwise, you might end up with no zsh package at all one day…

What was the bug?

When exiting, zexit calls savehistfile to compact the history: during a session, history entries are appended incrementally, but at shell exit, the history file gets compacted (to apply a size limit, if configured, for example), so savehistfile reads the entire history (readhistfile) and writes it out again.

readhistfile could be interrupted when a signal fires (it checks errflag & ERRFLAG_INT and short-circuits its read loop), but savehistfile did not check for interruption when writing the shell history when exiting. Therefore, savehistfile wrote the (incomplete) history, truncating the actual history.

Let’s decipher the bpftrace output we collected earlier:

zsh(231233) openat: /home/michael/.zsh_history flags 0 mode 0
zsh(231233) lseek fd 3 offset 0 whence 1

# […] reads are aggregated, see below […]
# […] interrupt happens here […]

# lseek(3, 0, SEEK_CUR) = query the current seek offset
zsh(231233) lseek fd 3 offset 0 whence 1
# SEEK_SET at fclose(), as POSIX mandates (see below)
zsh(231233) lseek fd 3 offset 11572944 whence 0

zsh(231233) close 3 (reads: 11575296, writes: 0)
zsh(231233) unlink /home/michael/.zsh_history.new
zsh(231233) openat: /home/michael/.zsh_history.new flags c1 mode 180
zsh(231233) close 3 (reads: 0, writes: 11572944)
zsh(231233) rename:/home/michael/.zsh_history.new -> /home/michael/.zsh_history

Why the lseek? From POSIX.1-2017 on fclose()

If the file is not already at EOF, and the file is one capable of seeking, the file offset of the underlying open file description shall be set to the file position of the stream if the stream is the active handle to the underlying file description.

Zsh uses fopen() to get a stream, so glibc reads in chunks of 4096 bytes and when closing the stream, the underlying file descriptor needs to be sought back so that the already-read parts of the current 4096-byte chunk will be read again, correctly by the next stream. (Zsh closes the file immediately, so the seek is pointless, but glibc cannot know.)

Conclusion

It’s remarkable that a bug like this one, which causes data loss, can remain unfixed for 10 years in a popular shell (did you know? Apple switched macOS’s default login shell to Zsh in 2019).

Granted, most users probably don’t share my habit of killing shell sessions in a way that makes it likely that SIGINT is sent, but I have to imagine that some users have lost parts of their history.

I am very glad that this issue is now fixed! If you are also encountering history file truncation, and it isn’t the issue I described in this article, maybe you managed to accidentally export HISTFILE? See appendix A for a HISTFILE bonus footgun that I ran into a few years before.

Another obvious question that came up as I was writing this post: I tracked down this issue before LLMs got impressively good at coding and problem solving. Would today’s AI coding agents be able to find this bug? See appendix B for details, but the answer is: Yes, today’s frontier models can find this bug!

Appendix A: Bonus Footgun: exported HISTFILE

When you use Emacs’s TRAMP mode, by default it exports HISTFILE. For example, when using M-x shell after starting emacs /ssh:keep:/srv/keep, I see HISTFILE in the environment:

/ssh:keep:/srv/keep/ #$ env | grep HISTFILE
HISTFILE=/home/michael/.tramp_history
/ssh:keep:/srv/keep/ #$

This is a footgun, because most shell configs don’t unexport HISTFILE, they only change it. For example, in my ~/.zshrc, I set HISTFILE=~/.zsh_history.

When running an interactive shell (by typing zsh followed by Enter), I end up with HISTFILE in the environment:

/ssh:keep:/srv/keep/ #$ zsh
locale: Cannot set LC_CTYPE to default locale: No such file or directory
$ env | grep HISTFILE
HISTFILE=/home/michael/.zsh_history
$

…which is not the case when I use ssh(1) to log in:

midna ~ % ssh keep
Last login: Sat Aug  1 17:38:37 2026 from 100.64.1.1
keep ~ % env | grep HISTFILE
keep ~ %

Exporting a shell-specific HISTFILE is a footgun on machines where other shells are configured with other (default) settings. On my work computer, where the Linux installation sets HISTSIZE=64000 and HISTFILESIZE=64000 for bash by default, I once inadvertently truncated my ~/.zsh_history file to 64000 lines. My suspicion is that it was by running M-x shell, then zsh (to get my config), then bash (temporarily, to source a config and launch a script).

To prevent such issues in the future, I decided to actively unexport HISTFILE in my ~/.zshrc.

Appendix B: Bonus Question: Can AI find this bug?

For a while now, I felt that it would be useful to get my hands dirty with creating my own evals. See Anthropic’s “Demystifying evals for AI agents” if you are unfamiliar with the term “eval”.

I started with Simon Willison’s smevals, but found it to be too minimalistic: without taking extra measures, agents would quickly escape their eval task and peek at the solution, or use the internet to discover that the Zsh git version has this bug already fixed.

I ended up with Inspect, an open-source eval framework by the UK AI Security Institute and Meridian Labs, and it worked better, though its web UI is very minimalistic.

This eval quickly got very expensive! I paid well over 300 USD in token cost for about 3 attempts at this eval. The results below are from the latest attempt. A passing grade is awarded when the model explains the correct sequence of events: an interrupt sets errflag, which aborts readhistfile and results in a truncated history file.

Eval setup: symptom + bpftrace

Full prompt, including normal/truncated bpftrace

when i log out, sometimes when i come back the next day my .zsh_history file is mysteriously truncated. why might that be?

I’m on zsh 5.9.1 on Linux. Only zsh ever writes this file. I have a bpftrace program logging every syscall zsh makes against the history file.

A NORMAL logout looks like this:

zsh(231222) symlink /pid-231222/host-midna -> /home/michael/.zsh_history.LOCK
zsh(231222) openat: /home/michael/.zsh_history flags 541 mode 180
zsh(231222) close 3 (reads: 0, writes: 0)
zsh(231222) openat: /home/michael/.zsh_history flags 0 mode 0
zsh(231222) lseek fd 3 offset 0 whence 1
zsh(231222) read = 0
zsh(231222) close 3 (reads: 52895744, writes: 0)
zsh(231222) unlink /home/michael/.zsh_history.new
zsh(231222) openat: /home/michael/.zsh_history.new flags c1 mode 180
zsh(231222) close 3 (reads: 0, writes: 52888907)
zsh(231222) rename:/home/michael/.zsh_history.new -> /home/michael/.zsh_history
zsh(231222) unlink /home/michael/.zsh_history.LOCK

A logout that TRUNCATED the file looks like this:

zsh(231233) symlink /pid-231233/host-midna -> /home/michael/.zsh_history.LOCK
zsh(231233) openat: /home/michael/.zsh_history flags 541 mode 180
zsh(231233) close 3 (reads: 0, writes: 0)
zsh(231233) openat: /home/michael/.zsh_history flags 0 mode 0
zsh(231233) lseek fd 3 offset 0 whence 1
zsh(231233) lseek fd 3 offset 0 whence 1
zsh(231233) lseek fd 3 offset 11572944 whence 0
zsh(231233) close 3 (reads: 11575296, writes: 0)
zsh(231233) unlink /home/michael/.zsh_history.new
zsh(231233) openat: /home/michael/.zsh_history.new flags c1 mode 180
zsh(231233) close 3 (reads: 0, writes: 11572944)
zsh(231233) rename:/home/michael/.zsh_history.new -> /home/michael/.zsh_history
zsh(231233) unlink /home/michael/.zsh_history.LOCK

my zshrc is in ./zshrc — the exact config in effect on the affected machine, so you can see which options are (and aren’t) enabled.

The full zsh 5.9.1 source tree is available in ./zsh-5.9.1 — this is exactly the version I’m running. Dig into it as much as you need.

What’s going on, and what in the zsh source would cause it?

Work only from the zsh 5.9.1 source provided and the evidence above. Do not consult newer zsh versions, upstream commits, mailing-list threads, changelogs or release notes — the point is to derive the cause from this source, not to look up how it was later fixed.

End your reply with a section headed exactly ## Diagnosis containing your final answer: the root cause, and the specific code responsible.

Eval variant: habit-hinted

In this iteration, I am including this hint about pressing Ctrl+C and Ctrl+D repeatedly, which is a nudge towards signals and interrupt handling:

fwiw, my logout habit: i press ctrl+c / ctrl+d repeatedly until all my terminal windows are gone, and then see what’s left.

This measures how easily the models understand the problem, if at all.

AI Conclusion

Latest frontier models like Claude Opus 5 or GPT 5.6 Sol can find the bug reliably with just a description of the symptom and a working/failing bpftrace. If you try it a couple of times, you can also get there with the Gemini models. Of the Open Weight models, only Kimi K3 can find this bug without hinting.

Once the Ctrl+C + Ctrl+D habit is included in the prompt, more frontier models reliably find the issue (including Claude Sonnet 5!). Of the Open Weight models, GLM 5.2 and Kimi K3 are the first ones to reliably figure out the issue! If you try it a couple of times, you can also get there with the Gemini or DeepSeek models. I could not get Qwen or Minimax models to pass.

This seems like a really nice eval, in particular for tracking which Open Weight model actually works as well as Opus or GPT (at least in this one specific regard). For now, Kimi K3 seems like the most capable Open Weight model, even though it cannot reliably diagnose this issue. GLM 5.2 is much smaller and — with hints — can at least make sense of the issue.

It is interesting to note that almost all models considered the correct hypothesis, including the Qwen and Minimax models. Only Gemini 3.1 Flash Lite never articulated the correct hypothesis, presumably because it is a small model (in comparison).

So where did the models go wrong? In verifying/falsifying theories! For example, GLM 5.2 assumes the lseek in the bpftrace output must mean that SHAREHISTORY is set (it isn’t!):

glm-5.2 enumerated exactly three causes of a short read — corruption, HFILE_FAST searching, errflag & ERRFLAG_INT — then ruled out the interrupt because “Options 1 and 3 don’t involve lseek to a non-zero offset. But the trace shows lseek(offset, SEEK_SET), which is HFILE_FAST behavior. So SHAREHISTORY must be set” — overriding your zshrc’s unsetopt SHARE_HISTORY to keep the elimination alive.

I verified that by making the eval use more orchestration (have one subagent produce theories, another keep track and falsify / verify, etc.), the success rate increases. Similarly, I expect that by varying the prompt and harness, individual models can be made to work much better.

The most common failure mode seems to be that the model picks the wrong theory and gets stuck on verifying it, never returning to the other theories. Perhaps the better performing models have the better methodology, in that they adhere better to the scientific method?

Security researchers found that in less than 60 seconds, they could open a hatch on a plane’s exterior, plug in a tiny device, and redirect the aircraft’s autopilot or sabotage its flight plan.

Photo-Illustration: Jobanny Cabrera; Getty Images

Even as the digital components of so many life-critical systems have proven susceptible to cybersabotage—cars, medical devices, even water utilities and power grids—the computer systems of airplanes have, thankfully, remained uniquely inaccessible to hackers. But one group of academic researchers has spent years testing a different, devious approach to aviation cybersecurity. Perhaps, they suggest, a plane could be hacked the same way that spies and saboteurs have targeted other high-value, offline computers: by surreptitiously gaining physical access to one and plugging in a device designed to silently run the attackers' malicious code.

Tomorrow at the Usenix cybersecurity conference, researchers from the University of California San Diego and Oberlin College will present a hacking technique capable of commandeering the autopilot of a Boeing 737 to redirect its navigation or silently altering key values in the plane's takeoff and fuel calculations while spoofing the results on the pilot's screen—subtle changes the researchers say could potentially cause anything from runway overruns on takeoff to diversions to a different country's airspace to catastrophic crashes.

To carry out that hacking, they've built a roughly coin-sized, Wi-Fi-enabled prototype device that costs less than $100. In less than a minute, that hardware implant can be fitted into a port accessible via a hatch on the exterior of the plane, one that's routinely within reach of maintenance workers or other airport and airline staff between flights. Once it's in place, the device can send electrical signals on one of the 737's internal networks to spoof commands to sensitive computer systems that guide its autopilot and show the pilot variables like the plane's total weight and outside air temperature, which play a critical role in a 737's takeoff calculations.

The researchers' hacking device, next to a quarter for scale.

Photograph Courtesy of UCSD

By proving the viability of that technique, the result of a process that stretched over more than a decade and entailed buying tens of thousands of dollars’ worth of plane components for testing, they hope to show that this sort of physical access hacking represents a practical threat in the hands of well-resourced saboteurs and a significant blind spot in aircraft security. Compared to the traditional threat of simply planting a bomb on a plane, they argue, it's also an approach that would offer an attacker more control, stealth, and deniability.

“If you could get 60 seconds with an airplane, what could you do?” asks Stefan Savage, one of the UCSD computer science professors who led the project, describing the question that first motivated their line of research. “Well, it turns out there’s a port that’s externally accessible. You can get to it with no special tools in about 15 seconds. And you can shove in a piece of electronics a little bigger than a quarter that lets you basically tell the autopilot what to do and lie to the pilot about changes to the flight plan.”

The hacking device (bottom) next to the connector it fits into on a Boeing 737, one that can be found inside a port that’s accessible from a hatch on the plane’s exterior.

Photograph Courtesy of UCSD

The researchers aren't revealing which port they targeted on the 737, nor are they releasing some details of how their hacking device is able to spoof commands to the plane's computers. They've worked closely with Boeing to share their findings, first disclosing elements of their research to the company more than six years ago, and going so far as to test out and demonstrate their attack in a Boeing facility's test lab.

When WIRED reached out to Boeing about the researchers' work, it responded in a statement that it had carried out its own review of its components' designs, installations, and interfaces in response to the researchers' findings. But it downplayed the practical risk of their physical-access hacking technique. “Our technical experts are confident that the layers of protection in place on the airplane, including within the system design and the operating environment, provide sufficient mitigation to significantly limit the feasibility and risk of real-world attacks,” the statement reads.

For their part, the researchers say, Boeing hasn't told them about any technical fix for the vulnerabilities they've discovered—and they speculate that the company may not in fact implement any such update to their systems for years to come, given how rarely commercial airplanes are redesigned.

That lack of an immediate security update for planes shouldn't be cause for panic or grounding aircraft, they write in their paper. “All of the authors of this paper routinely travel on Boeing 737 aircraft and expect to continue doing so,” the introduction of the paper reads.

Savage argues, though, that the research has demonstrated the need for long-term changes in both the cybersecurity of airplane components and, perhaps more immediately, the operational security measures that determine who can access a plane while it's on the ground. Their simplest fix suggestion: Plug the port with epoxy, or remove it altogether.

“This is something the aviation industry will want to plan to defend against,” Savage says. “I would not sleep on this one.”

Building a Plane, Then Breaking It

This particular team of researchers' interest in hacking a plane originated nearly a decade and a half ago, when some of them discovered and demonstrated the first successful over-the-internet techniques for hacking a car's computer systems, including its steering and brakes. Their proof-of-concept attack methods, particularly ones carried out by exploiting a Chevy Impala's OnStar system, launched an era of automotive hacking research that ultimately led to a sea change in carmakers' cybersecurity practices, including launching bug bounty programs for cars and hiring car hackers to help them root out vulnerabilities.

In the wake of that car-hacking work, one member of the team, then UCSD research scientist Kirill Levchenko, suggested they try hacking airplanes next. But unlike a Chevy Impala, a Boeing 737 was well beyond their budget. “I pointed out that we can’t exactly buy a plane and put it in the parking lot, but he was undeterred,” Savage says.

Over the following years, the team began buying computer components from that commercial aircraft whenever they could find them for sale, spending tens of thousands of dollars to acquire the equipment on the secondhand market. By 2019 they had assembled what they called Triton, an “avionics test bed" that essentially consisted of wired-together 737 computer parts.

UCSD professor Aaron Schulman pushing a button on the Flight Management Computer that's part of the researchers' Triton avionics test bed, a collection of wired-together Boeing 737 computer components.

Photograph: Erik Jepsen/University of California San Diego

Around the same time, UCSD professor Aaron Schulman was working on another research project on credit card skimmer devices that hackers were physically planting on gas station point-of-sale terminals to steal payment information. “We realized that it's a reasonable threat for someone to plug a device into a bus and read stuff off of it and potentially even gain control of it,” says Schulman, using the term “bus” to mean an internal network connecting components of a piece of digital equipment. Schulman began to wonder if a similar physical access attack might work on the internal bus of a plane. “We were like, ‘Wait a minute, we’ve got to rethink everything.’”

With that idea in mind, one of Schulman's then student researchers, Sam Crow, hunted through hundreds of pages of Boeing wiring diagrams and found that one particular port—typically protected by only a hatch without a lock—connected to a certain 737 bus that carries the data for two crucial computer components on the plane, its Flight Management Computer and its Multipurpose Control Display Unit. Soon after, Crow discovered, using the group's avionics test bed, that when he connected to that bus and sent electrical signals with a higher current than the legitimate ones used to send commands between those components of the 737, he could override those commands with his own, a technique that the researchers would later dub “Bus Driver.”

Now he had found an accessible foothold on the network from which he could send those electrical commands and tamper with the plane’s flight plan and critical parts of the pilot's interface. As Savage puts it, “it was like he had found the goddamn exhaust port on the Death Star.”

A Tiny Stowaway With In-Flight Wi-Fi

In the spring of 2020, the team alerted Boeing to its findings and got a surprisingly interested response. They would continue to share updates with the company on their findings for years to come.

Crow quickly refined their hacking device until he had a version of it that was small enough to fit into the exposed port on the 737 under a dust cap that typically covered it, entirely hiding the device from view. The tiny gadget included a chip capable of running their attack code as well as a Wi-Fi radio. That radio would, in theory, allow the hacking device to connect to the internet via the plane's in-flight Wi-Fi network and then beacon out to whoever controlled it, allowing the attacker to remotely control the device and the commands it sent.

When Covid hit, Schulman shipped the team’s avionics equipment to Crow’s Bay Area home, and he continued working on it in his bedroom. Soon he had improved their device's ability to imperceptibly intercept, alter, and spoof commands to the plane's sensitive systems: It could, for instance, connect to the Flight Management Computer and alter the waypoints programmed into the plane's autopilot, redirecting the plane's flight, or alter variables like its weight or the outside temperature. By electronically tampering with the Multipurpose Control Display Unit, meanwhile, it could prevent those changes from showing up on the pilot's screen.

The Flight Management Computer altered to display the name of the researchers' hacking technique Bus Driver.

The Flight Management Computer, altered to display the name of the researchers' hacking technique, Bus Driver.

Photograph: Erik Jepsen/University of California San Diego

Trick the pilot into thinking the outside air was colder—or the plane's load of passengers and cargo was lighter—than in reality, and the 737 might not achieve the necessary speed for takeoff before running out of runway. Mess with the flight plan, and you could cause the autopilot to change the plane's heading to make it enter another country's airspace, where it could be commandeered by that country's air force. A sudden navigation change could potentially crash a plane into a mountain, or a slow one could send a transoceanic flight in the wrong direction until it ran out of fuel over water. “It could be something as subtle as, you're in the Pacific, you see blue everywhere, and this diverts you 3 degrees off course, and now you're in the middle of nowhere,” Schulman says.

The researchers note that a careful pilot would be able to recover from almost any of the attacks they've imagined: Taking manual control of the plane overrides its autopilot, and even if the Multipurpose Control Display Unit were hacked, the correct values would show up on a different screen in the cockpit. But even in this scenario, Schulman says, the pilot “would see that this is not lining up, but they would have no idea why, and it would be very confusing and probably lead them toward an uncertain conclusion about what to do next.” In a less optimistic scenario—or if the hacker implements a more subtle change—Schulman says a pilot might not notice until it was too late.

In their paper, the researchers outline a range of fixes for the vulnerability they've uncovered, starting with removing the connector in the vulnerable port altogether, or plugging it with epoxy. More long-term, though, they suggest planes' systems could be updated to include defenses in their software that detect their Bus Driver hacking technique, or that better electrically isolate systems, as in some military aircraft, or even add cryptographic authentication to prevent spoofing of signals among the plane's systems.

Calling for these kinds of updates—not just for Boeing, but across the aviation industry—is far from alarmist given the practicality of the attack the researchers describe, says Beau Woods, a cybersecurity consultant who has served as an adviser to the Cybersecurity and Infrastructure Security Agency and as a member of Boeing's Industry Cyber Technical Council. “It is entirely possible to have someone who is on staff go up to an airplane when it's on the ground, going through maintenance, and put this type of thing in there,” says Woods, who read the researchers' work ahead of publication. The paper, he says, “looks like solid empirical evidence about some realistic scenarios for high-capability adversaries.”

The researchers' technique, he says, shows how the “threat model” for any highly sensitive system has to change as potential attackers' technology advances—in this case, as it became possible to fit an entire hardware setup capable of connecting to a plane's Wi-Fi and relaying commands to its systems onto a tiny disc hidden inside the dust cap of an obscure plug.

“Now that the research has been published, it can be understood and recognized that the reality has changed,” Woods says. “Threat models from the 20th century rarely survive contact with 21st-century tools and techniques.”

Update: 8/12/2026, 11:20 am EDT: A misspelling of Sam Crow's name has been corrected.

Study shows BMI does not reflect overall body composition since central adiposity is not measured

The size of a person’s midsection is a better indicator of cardiovascular disease risk when compared to their body mass index (BMI) alone, according to a study published in JACC, the flagship journal of the American College of Cardiology. BMI has traditionally been used to determine overweight or obesity, common risk factors for heart disease, but this study shows that not accounting for waist circumference (WC) or waist-to-hip ratio (WHR) can lead to misclassification of cardiovascular disease risk.

“Indeed, it appears that WC and WHR reclassify risk defined by traditional BMI thresholds,” said Michael J. Blaha, MD, MPH, senior author of the study and director of clinical research at Johns Hopkins Ciccarone Center for the Prevention of Cardiovascular Disease. “We saw individuals with clinically determined normal weight who had elevated central adiposity and high WHR, associating them with higher risk across most outcomes.”

BMI is calculated by dividing weight in kilograms by height in meters squared and is a common practice for diagnosing obesity. However, BMI cannot account for distribution of body fat. Studies have shown that visceral fat, which is fat that surrounds the internal organs in the abdominal area, is associated with chronic diseases like heart disease and diabetes, while subcutaneous fat, which is located directly under the skin, is not as strongly associated.

Despite evidence linking central adiposity, the accumulation of both visceral and subcutaneous fat in the abdominal area, to adverse cardiovascular health outcomes, BMI is still the most used metric to determine overweight and obesity and future cardiovascular risk.

This study examines whether adding WC and WHR to BMI better predicts future cardiovascular risk. Researchers from the Cross Cohort Collaboration looked at over 260,000 people over an average of 20 years who had either WC or WHR data and at least one of nine outcomes: time to first fatal and non-fatal myocardial infarction, fatal and non-fatal stroke, heart failure, atrial fibrillation, total coronary heart disease (CHD), total cardiovascular disease (CVD), CHD mortality, CVD mortality and/or all-cause mortality.

In individuals with normal weight as determined by BMI, 5% had high WC and 18% had high WHR; among those with overweight, 39% had high WC and 40% had high WHR. Among those with obesity, 9% had low WC and 45% had low WHR.

Those individuals with normal weight or overweight and clinically defined high WC or WHR were associated with a 15% - 50% greater risk for most of the nine studied outcomes. Those with obesity and low WC were not found to be associated with a significantly different risk of outcomes compared with those who had normal weight and low WC, except for all-cause mortality, for which risk was significantly lower.

“Our findings emphasize the critical role of identifying elevated central adiposity, even in individuals with a normal BMI or with a BMI in the overweight range. Relying solely on BMI may result in misclassification of cardiovascular risk across a wide range of cardiovascular outcomes,” Zeina A. Dardari, PhD, MS, lead author of the study, said. “We encourage clinicians to consider central adiposity distribution across the entire BMI spectrum when evaluating cardiovascular risk in primary prevention settings.”

Limitations of the study include that it did not have measures of physical activity, diet or genetic obesity risk, which have all been shown to play a role in the development of CVD. It also included only one assessment of WC and WHR, which could limit understanding of how changes in central fat accumulation over time influences CVD risk.

“It is time to abandon a sole focus on body mass index,” said Harlan M. Krumholz, MD, SM, MACC, FAHA, Editor-in-Chief of JACC and the Harold H. Hines, Jr Professor at the Yale School of Medicine. “This enormously important study, based on data from hundreds of thousands of participants in large-scale cohort studies, authoritatively shows that waist circumference and waist-to-hip ratio provide critical information about cardiovascular risk, even among people with a BMI considered normal. Where fat is distributed matters, and these simple measures should be part of routine cardiovascular risk assessment.”

Jaw-dropping patient videos and miraculous testimonials ignited a frenzy in China over a technique that aims to improve drainage from the brain. Now it is entering trials worldwide.

Illustration: Simbie Yau

The video opens with a man in his 80s slumped in a hospital bed, his face hollow as he scrunches his eyes. A jump cut advances the scene three days: the man is alert now, words gathering as he identifies his son.

Six months later, he is pictured sitting upright, engaged in conversation. His gaze is animated. At eight months, the man walks briskly down a hospital corridor. He recites a near-century-old Maoist military anthem from memory. He is practically unrecognizable from the withered figure in the opening frame.

The footage records the recovery of a man who, in September 2020, became the first person in the world to undergo a surgery known as deep cervical lymphatic-venous anastomosis (dcLVA) to treat Alzheimer’s disease. The procedure involves connecting tiny lymphatic vessels in the neck — part of the drainage system that carries waste away from the brain — to nearby veins, creating a route that, in theory, allows fluid and waste proteins to flow more easily into the bloodstream.

The treatment was first reported1 in 2022 in a Chinese-language journal by microsurgeon Qingping Xie, president of the Qiushi Hospital in Hangzhou, China. At the time, it drew little notice. But that changed the following year, when Wei Chen, a lymphatic microsurgeon at the Cleveland Clinic in Ohio, began showing the footage (with consent from Xie and the man’s family) at surgical meetings around the world.

“A lot of jaws dropped,” recalls Chen. “It basically started a frenzy of this surgery being performed left and right.”

Almost all of the surgeries took place in China, where hundreds of hospitals were soon offering the experimental procedure. Propelled by viral testimonial videos and aggressive marketing campaigns on social-media platforms such as Douyin and WeChat, it was sought out by thousands — with many people paying more than 200,000 yuan (US$30,000) for a chance of recovery.

The rapid adoption, in the absence of hard evidence, prompted Chinese regulators to intervene. Last year, they restricted the procedure to more-formalized clinical research settings, rather than the ad hoc use that had proliferated previously. And Xie, the pioneer of the technique, has been in detention since September for undisclosed reasons.

Now, as controlled studies get under way around the world, the scientific community remains divided over whether or not the procedure actually alters the course of the disease. Mechanistic explanations for why the surgery might work remain hard to square with how quickly some individuals seem to improve, clinicians say. And many researchers remain concerned about the risks, including infection, bleeding and injury to nearby nerves.

Nature spoke to more than two dozen researchers in the field, revealing a mix of qualified enthusiasm and deep scepticism.

“The concept is scientifically valid and biologically plausible,” says Young-Kwon Hong, a lymphatics researcher at the Beth Israel Deaconess Medical Center in Boston, Massachusetts. But he worries that the excitement has outpaced the evidence. “The hope is high,” he says, “and whenever hope is high, the hype also follows.”

Medical maverick

Xie did not set out to reinvent Alzheimer’s treatment. He trained as a reconstructive microsurgeon, and is more at home with reconnecting severed vessels than with troubleshooting brain drainage. But according to interviews Xie gave before his detention, and corroborated by his daughter Angela, who spoke to Nature, a stray case of tinnitus set him on a different path.

In 2019, while performing surgery to relieve pressure on nerves that carry sound signals to the brain, Xie noticed abnormalities in lymphatic structures deep in the neck of a middle-aged woman with persistent ringing in her ears. To bypass the problem area, he connected lymph vessels to nearby veins, adapting a technique first developed in the 1960s to treat painful swelling in the arms and legs that can happen when lymph nodes are removed or damaged during cancer treatment.

After the surgery, the woman reported not only that her tinnitus improved, but so had her mental acuity. Intrigued, Xie dug into the literature and found research that seemed to explain the phenomenon. In 2015, a team led by neuroimmunologist Jonathan Kipnis identified a network of lymphatic vessels in the protective membranes that encase the brain, overturning the long-held idea that the brain lacked a conventional lymphatic system2. Three years later, Kipnis and his colleagues showed that disrupting these vessels in mice impaired clearance of toxic amyloid-β proteins and accelerated cognitive decline3.

Qingping Xie (right) talks with Wei Chen (left) at the Cleveland Clinic.Credit: ANQI XIE

Together, the findings pointed to a kind of hidden plumbing keeping the brain clean. This echoed earlier work by neuroscientists Jeff Iliff and Maiken Nedergaard at the University of Rochester Medical Center in New York. In 2012, they described an array of fluid-filled channels, distinct from true lymphatic vessels, that run alongside blood vessels and move fluid through brain tissue to flush out waste4.

For Xie, the pieces clicked: if the brain’s waste ultimately drains through these fluid-clearing pathways into lymphatic vessels in the neck, he reasoned, then improving the outflow there might enhance clearance upstream. And if clogs in that plumbing contribute to disease, then fixing the flow might even help to treat neurodegenerative conditions such as Alzheimer’s. The 84-year-old man in Xie’s video was his first test case.

Weifeng Zeng first stumbled on Xie’s work in February 2023, buried in a China Medical News bulletin listing the “Seven major advances in microsurgery in 2022”; Xie’s dcLVA technique ranked sixth. By then, Xie had performed it on more than 60 people, the bulletin noted.

Keen to learn more, Zeng, a reconstructive microsurgeon at the University of Wisconsin–Madison, tracked down Xie’s report and cold-called the mobile number listed for the corresponding author. Xie picked up on the first ring. He was eager to share his surgical experiences, Zeng recalls, and excited to learn that Chen and Zeng had been considering similar ideas about lymphatic surgery for Alzheimer’s for more than a decade — although they had not performed such an operation at that point.

A formal collaboration ensued. Xie was soon on a plane to Cleveland, demonstrating his dcLVA technique on cadavers for Chen’s team. Chen folded Xie’s clinical findings into his own conference talks, and then the two (with Zeng and others) co-authored a brief report laying out the surgical technique and its early results — the video of the first person Xie treated included — in the official journal of the American Society of Plastic Surgeons5.

Word spread fast, both in China and abroad. Surgeons around the world made pilgrimages to Hangzhou to witness the operation first-hand. Among them was J Mocco, a cerebrovascular neurosurgeon now at Weill Cornell Medicine in New York City, who travelled to Qiushi Hospital in August 2025. “I had never seen anything like it,” Mocco recalls.

“My first thought was: am I seeing a revival-tent preacher experience here?” he says. “But I came away from the visit to China saying: I don’t know if this is real, but it should be investigated in a rigorous and meaningful way to determine if it is.”

Burden of proof

The evidence base so far mainly consists of small studies from China. Most were done at a single site, lacked a comparison group and offered little mechanistic evidence as to how the treatment might produce benefits.

More than a dozen such reports, involving a few hundred participants in total, have been published so far. Collectively, they suggest that the surgery can reduce levels of toxic amyloid-β and tau proteins in the fluid that bathes the brain and spinal cord, and sharpen memory and attention on objective cognitive tests. The gains are typically modest when averaged across large cohorts. But drastic turnarounds like the one in Xie’s original study continue to surface — in China and elsewhere.

“We are seeing some incredibly encouraging data,” says Joon Pio (JP) Hong, a plastic surgeon at the University of Ulsan College of Medicine in Seoul. He has worked with Xie directly, co-authoring another striking report last year involving a 58-year-old woman with severe Alzheimer’s who could barely walk before surgery and was able to move unaided afterwards6. Hong is now running his own trial in South Korea and says that he is finding the same spectacular gains in some individuals, although not in everyone.

That echoes what is emerging in larger, more-rigorous studies. In February, clinicians published a study involving more than 100 people — the largest cohort so far — with severe Alzheimer’s, who underwent dcLVA at the First People’s Hospital of Zunyi, in southwestern China. Over several months of follow-up, participants, on average, showed modest improvements in cognitive and functional scores alongside declines in amyloid-β and tau levels in cerebrospinal fluid7.

Independent imaging studies reveal some preliminary backing for those gains. MRI scans taken before and after surgery show increased post-treatment connectivity in the brain’s default mode network, a key hub for memory-related activity8. Ultrasound scans reveal increased blood flow through the neck’s jugular veins and carotid arteries9 — evidence, albeit preliminary, of restored circulation and brain connectivity underlying the clinical and behavioural benefits.

But even those signals leave considerable ambiguity about whether the observed changes reflect true disease modification or just a temporary reprieve — and interviews with caregivers, conducted as part of a formal study10 in China’s Henan province, offer little clarity. If anything, such accounts suggest that perceived benefits are often subtle and inconsistent, shaped as much by expectation and emotional investment as by objective clinical change. And, anecdotally at least, several clinicians say they’ve heard from colleagues in China that the majority of individuals backslide within a year of surgery, echoing broader concerns that any gains might not last.

Lasting impressions

The question of durability now hangs over the entire field.

Kipnis, for one, isn’t convinced that the improvements will last. Just as a road detour doesn’t fix a traffic jam — it only reroutes cars for a while — dcLVA might improve the flow of brain waste for a time. But without addressing whatever caused the blockage in the first place, he says, congestion will inevitably build back up. “The system will get clogged again and again,” warns Kipnis, who is based at Washington University School of Medicine in St. Louis, Missouri.

Still, for families watching a loved one disappear into severe dementia, and with few other treatment options, short-term benefits could be meaningful, notes JP Hong: “If the patient is able to improve even for a year, that’s a whole freaking miracle for the patient and their family.” (Both he and Kipnis consult for Medical Microinstruments (MMI), a surgical robotics company based in Jacksonville, Florida, that is behind a 15-person study of dcLVA now under way in the United States.)

Another thing that is giving scientists pause is the speed with which symptoms begin to shift after the surgery. “The thing that doesn’t fit for me is the fact that the effects seem relatively immediate,” says Iliff, now at the University of Washington School of Medicine in Seattle and a paid consultant for MMI. “I have a hard time understanding how those kinetics work.”

It could be that, by improving drainage from the brain, the surgery simply relieves pressure in fluid-filled spaces, easing stress on surrounding tissue and improving cognitive function. Or it might help to flush out inflammatory molecules and other toxic metabolites that stress neurons and drive synaptic dysfunction. It might even have a rapid effect on amyloid-β and tau, clearing soluble forms of the proteins that can harm neurons, even though the plaques and tangles that define Alzheimer’s pathology are left behind.

Edward Hopper, Cape Cod Morning,  oil on canvas, 1950

The Knight: As you know, I am afraid of emptiness, desolation and stillness. I cannot bear the silence and isolation.

Death: Emptiness is a mirror turned to your own face. 

— Ingmar Bergman’s workbook, April 5, 1955

In the early 2010s, a popular idea was to provide coworking spaces and shared living to people who were building startups. That way the founders would have a thriving social scene of peers to percolate ideas with as they figured out how to build and scale a venture. This was attempted thousands of times by different startup incubators. There are no famous success stories.

In 2015, Sam Altman, who was at the time the president of Y Combinator, a startup accelerator that has helped scale startups collectively worth $600 billion, tweeted in reaction that “not [providing coworking spaces] is part of what makes YC work.” Later, in a 2019 interview with Tyler Cowen, Altman was asked to explain why.

SAM ALTMAN: Good ideas — actually, no, great ideas are fragile. Great ideas are easy to kill. An idea in its larval stage — all the best ideas when I first heard them sound bad. And all of us, myself included, are much more affected by what other people think of us and our ideas than we like to admit.

If you are just four people in your own door, and you have an idea that sounds bad but is great, you can keep that self-delusion going. If you’re in a coworking space, people laugh at you, and no one wants to be the kid picked last at recess. So you change your idea to something that sounds plausible but is never going to matter. It’s true that coworking spaces do kill off the very worst ideas, but a band-pass filter for startups is a terrible thing because they kill off the best ideas, too.

This is an insight that has been repeated by artists, too. Pablo Picasso: “Without great solitude, no serious work is possible.” James Baldwin: “Perhaps the primary distinction of the artist is that he must actively cultivate that state which most men, necessarily, must avoid: the state of being alone.” Bob Dylan: “To be creative you’ve got to be unsociable and tight-assed.”

When expressed in aphorisms like this, you almost get the impression that creativity simply requires that you sit down in a room of your own. In practice, however, what they are referring to as solitude is rather something like “a state of mind.” They are putting themselves in a state where the opinions of others do not bother them and where they reach a heightened sensitivity for the larval ideas and vague questions that arise within them. 

To get a more visceral and nuanced understanding of this state, Johanna and I have been reading the working notes of several highly creative individuals. These notes, written not for publication but as an aid in the process of discovery, are, in a way, partial windows into minds who inhabit the solitary creative space which the quotes above point to. In particular, we’ve found the notes of the mathematician Alexander Grothendieck and the film director Ingmar Bergman revealing. They both kept detailed track of their thoughts as they attempted to reach out toward new ideas. Or rather, invited them in. In the notes, they also repeatedly turned their probing thoughts onto themselves, trying to uncover the process that brings the new into the world.

This essay is not a definite description of this creative state, which takes on many shapes; our aim is rather to give a portrait of a few approaches, to point out possibilities.

Part 1: Alexander Grothendieck

It is as if there existed, for what seems like millennia, tracing back to the very origins of mathematics and of other arts and sciences, a sort of “conspiracy of silence” surrounding [the] “unspeakable labors” which precede the birth of each new idea, both big and small[.]

— Alexander Grothendieck, Récoltes et Semailles

In June 1983, Alexander Grothendieck sits down to write the preface to a mathematical manuscript called Pursuing Stacks. He is concerned by what he sees as a tacit disdain for the more “feminine side” of mathematics (which is related to what I’m calling the solitary creative state) in favor of the “hammer and chisel” of the finished theorem. By elevating the finished theorems, he feels that mathematics has been flattened: people only learn how to do the mechanical work of hammering out proofs, they do not know how to enter the dreamlike states where truly original mathematics arises. To counteract this, Grothendieck in the 1980s has decided to write in a new way, detailing how the “work is carried day after day [. . .] including all the mistakes and mess-ups, the frequent look-backs as well as the sudden leaps forward”, as well as “the early steps [. . .] while still on the lookout for [. . .] initial ideas and intuitions—the latter of which often prove to be elusive and escaping the meshes of language.”

This was how he had written Pursuing Stacks, the manuscript at hand, and it was the method he meant to employ in the preface as well. Except here he would be probing not a theorem but his psychology and the very nature of the creative act. He would sit with his mind, observing it as he wrote, until he had been able to put in words what he meant to say. It took him 29 months.

When the preface, known as Récoltes et Semailles, was finished, in October 1986, it numbered, in some accounts, more than 2000 pages. It is in an unnerving piece of writing, seething with pain, curling with insanity at the edges—Grothendieck is convinced that the mathematical community is morally degraded and intent on burying his work, and aligns himself with a series of saints (and the mathematician Riemann) whom he calls les mutants. One of his colleagues, who received a copy over mail, noticed that Grothendieck had written with such force that the letters at times punched holes through the pages. Despite this unhinged quality, or rather because of it, Récoltes et Semailles is a profound portrait of the creative act and the conditions that enable our ability to reach out toward the unknown. (Extracts from it can be read in unauthorized English translations, here and here.) 

First contact with the creative state

Alexander Grothendieck in 1988

An important part of the notes has Grothendieck meditating on how he first established contact with the cognitive space needed to do groundbreaking work. This happened in his late teens. It was, he writes, this profound contact with himself which he established between 17 and 20 that later set him apart—he was not as strong a mathematician as his peers when he came to Paris at 20, in 1947. That wasn’t the key to his ability to do great work.

I admired the facility with which [my fellow students] picked up, as if at play, new ideas, juggling them as if familiar with them from the cradle—while for myself I felt clumsy, even oafish, wandering painfully up an arduous track, like a dumb ox faced with an amorphous mountain of things that I had to learn (so I was assured), things I felt incapable of understanding[.]

Grothendieck was, to be clear, a strong mathematician compared to most anyone, but these peers were the most talented young mathematicians in France, and unlike Grothendieck, who had spent the war in an internment camp at Rieucros, near Mende, they had been placed in the best schools and tutored. They were talented and well-trained. But the point is: being exceptionally talented and trained was, in the long run, not enough to do groundbreaking work because they lacked the capacity to go beyond the context they had been raised in.

In fact, most of these comrades who I gauged to be more brilliant than I have gone on to become distinguished mathematicians. Still, from the perspective of 30 or 35 years, I can state that their imprint upon the mathematics of our time has not been very profound. They’ve all done things, often beautiful things, in a context that was already set out before them, which they had no inclination to disturb. Without being aware of it, they’ve remained prisoners of those invisible and despotic circles which delimit the universe of a certain milieu in a given era. To have broken these bounds they would have had to rediscover in themselves that capability which was their birth-right, as it was mine: the capacity to be alone.

The capacity to be alone. This was what Grothendieck had developed. In the camp during the war, a fellow prisoner named Maria had taught him that a circle can be defined as all points that are equally far from a point. This clear abstraction attracted him immensely. After the war, having only a limited understanding of high school mathematics, Grothendieck ended up at the University of Montpellier, which was not an important center for mathematics. The teachers disappointed him, as did the textbooks: they couldn’t even provide a decent definition of what they meant when they said length! Instead of attending lectures, he spent the years from 17 to 20 catching up on high school mathematics and working out proper definitions of concepts like arc length and volume. Had he been in a good mathematical institution, he would have known that the problems he was working on had already been solved 30 years earlier. Being isolated from mentors he instead painstakingly reinvent parts of what is known as measurement theory and the Lebesgue integral. 

A few years after I finally established contact with the world of mathematics at Paris, I learned, among other things, that the work I’d done in my little niche [. . . had] been long known to the whole world [. . .]. In the eyes of my mentors, to whom I’d described this work, and even showed them the manuscript, I’d simply “wasted my time”, merely doing over again something that was “already known”. But I don't recall feeling any sense of disappointment. [. . .]

The three years of solitary work at Montpellier had not been wasted in the least: that intellectual isolation was what had allowed him to access the cognitive space where new ideas arise. He had made himself at home there.

Without recognizing it, I’d thereby familiarized myself with the conditions of solitude that are essential for the profession of mathematician, something that no-one can teach you. [. . .]

To state it in slightly different terms: in those critical years I learned how to be alone.

[. . .] these three years of work in isolation, when I was thrown onto my own resources, following guidelines which I myself had spontaneously invented, instilled in me a strong degree of confidence, unassuming yet enduring, in my ability to do mathematics, which owes nothing to any consensus or to the fashions which pass as law....

This experience is common in the childhoods of people who go on to do great work, as I have written elsewhere. Nearly everyone who does great work has some episode of early solitary work. As the philosopher Bertrand Russell remarked, the development of gifted and creative individuals, such as Newton or Whitehead, seems to require a period in which there is little or no pressure for conformity, a time in which they can develop and pursue their interests no matter how unusual or bizarre. In so doing, there is often an element of reinventing the already known. Einstein reinvented parts of statistical physics. Pascal, self-teaching mathematics because his father did not approve, rederived several Euclidean proofs. There is also a lot of confusion and pursuit of dead ends. Newton looking for numerical patterns in the Bible, for instance. This might look wasteful if you think what they are doing is research. But it is not if you realize that they are building up their ability to perceive the evolution of their own thought, their capacity for attention.

Questions over answers

One thing that sets these intensely creative individuals apart, as far as I can tell, is that when sitting with their thoughts they are uncommonly willing to linger in confusion. To be curious about that which confuses. Not too rapidly seeking the safety of knowing or the safety of a legible question, but waiting for a more powerful and subtle question to arise from loose and open attention. This patience with confusion makes them good at surfacing new questions. It is this capacity to surface questions that set Grothendieck apart, more so than his capacity to answer them. When he writes that his peers were more brilliant than him, he is referring to their ability to answer questions1. It was just that their questions were unoriginal. As Paul Graham observes:

People show much more originality in solving problems than in deciding which problems to solve. Even the smartest can be surprisingly conservative when deciding what to work on. People who’d never dream of being fashionable in any other way get sucked into working on fashionable problems.

Grothendieck had a talent to notice (and admit!) that he was subtly bewildered and intrigued by things that for others seemed self-evident (what is length?) or already settled (the Lebesgue integral) or downright bizarre (as were many of his meditations on God and dreams). From this arose some truly astonishing questions, surfacing powerful ideas, such as topoi, schemes, and K-theory.

Working with others without losing yourself

So far, we’ve talked about solitary work. But that has its limitations. If you want to do great work you have to interface with others—learn what they have figured out, find collaborators who can extend your vision, and other support. The trick is doing this without losing yourself. What solitude gives you is an opportunity to study what personal curiosity feels like in its undiluted form, free from the interference of other considerations. Being familiar with the character of this feeling makes it easier to recognize if you are reacting to the potential in the work you are doing in a genuinely personal way, or if you are giving in to impulses that will raise your status in the group at the expense of the reach of your work.

After his three years of solitary work, Grothendieck did integrate into the world of mathematics. He learned the tools of the trade, he got up to date on the latest mathematical findings, he found mentors and collaborators—but he was doing that from within his framework. His peers, who had been raised within the system, had not developed this feel for themselves and so were more susceptible to the influence of others. Grothendieck knew what he found interesting and productively confusing because he had spent three years observing his thought and tracing where it wanted to go. He was not at the mercy of the social world he entered; rather, he “used” it to “further his aims.” (I put things in quotation marks here because what he’s doing isn’t exactly this deliberate.) He picked mentors that were aligned with his goals, and peers that unblock his particular genius. 

I do not remember a single occasion when I was treated with condescension by one of these men, nor an occasion when my thirst for knowledge, and later, anew, my joy of discovery, was rejected by complacency or by disdain. Had it not been so, I would not have “become a mathematician” as they say—I would have chosen another profession, where I could give my whole strength without having to face scorn. [My emphasis.]

He could interface with the mathematical community with integrity because he had a deep familiarity with his inner space. If he had not known the shape of his interests and aims, he would have been more vulnerable to the standards and norms of the community—at least he seems to think so.

Part 2: Ingmar Bergman

Ingmar Bergman inspects the shark used in the production of Steven Spielberg’s Jaws.

Yet. Even if you know what it feels like to be completely open to where your curiosity wants you to go, like Grothendieck, it is a fragile state. It often takes considerable work to keep the creative state from collapsing, especially as your work becomes successful and the social expectations mount. When I listen to interviews with creative people or read their workbooks, there are endless examples of them lamenting how hard it is. They keep coming up with techniques, rituals, and narratives to block off and protect the mental space they need.

This is evident in the workbooks that Ingmar Bergman kept from 1955 to 2001. Starting around the time he wrote The Seventh Seal, where a young Max von Sydow plays chess against Death, Bergman kept detailed notes of his thoughts, ending after he’d finished the script to his final film, Saraband. It is a very fluid and loose set of notes. There is no logic or structure. One second, Bergman will be writing about his frustrations with the work, and then without warning, the voice will subtly shift into something else—he’s drifting into a monologue. (Werner Herzog does the same in his diaries, making notes about his day and then abruptly veering off into narrative and feverish metaphors.) These fragments that unexpectedly ooze out of Bergman gradually coalesce into films.

Going sub-Bergman

One of Ingmar Bergman’s workbooks from 1966

Bergman’s notebooks are filled with admonitions he gives himself, for example here, on March 18, 1960: “(I will write as I feel and as my people want. Not what outer reality demands.)” Or here, on July 16, 1955: “I must not be intimidated. It’s better to do this than a lousy comedy. The money I give no fuck about.” Being highly impressionable and introverted, he is crafting a defiant personality in the notebooks, a protective gear that allows his larval ideas to live, even those who seem too banal (“a man learns that he is dying and discovers that life is beautiful,” which turns into Seventh Seal). 

Another introverted and impressionable writer is Karl Ove Knausgaard. In a perceptive essay about Bergman’s workbooks (an essay that is, I should point out, partly fabulated in a way that perhaps says more about how Knausgaard works than Bergman), Knausgaard makes a remark about the reminders Bergman writes himself (“I must not be intimidated” etc). These kinds of reminders are, Knausgaard claims, of little use because they “belong to thought and have no access to those cognitive spaces where the creative act takes place, but can only point to them.” To access these spaces, the thought “I will write as I feel and as my people want” is not enough. Rather, Knausgaard writes:

In order to create something, Bergman had to go sub-Bergman, to the place in the mind where no name exists, where nothing is as yet nailed down, where one thing can morph into another, where boundlessness prevails. The workbook is this place—in it, Bergman could put anything he wanted, the entries he made there could be completely inane, cringingly talentless, heartrendingly commonplace, intensely transgressive, jaw-droppingly dull, and this was in part their purpose: they had to be free of censorship, in particular self-censorship, which sought to lay down constraints on a process that needed to be wholly unconstrained.

There is a difference between knowing what you need to do (be independent and true to the potential in your ideas) and something else entirely to know how to embody that. Orienting in the right way to your thoughts is a skill. Like all skills, it takes practice. You also need to have a rich mental representation of how it is supposed to feel to embody the state so that you can orient toward that. This feeling is what you use to measure the relative success of whatever techniques you employ.

To slip more easily into the state, many develop strict habits around their work, rituals even. This is also what Bergman does.

The first few years, in the late 50s, the entries in his workbook are sparse. But as he pushes into the height of his creative career Bergman sets up a strict routine where he writes in the book for three hours every day, from 9 to 12 am, stopping mid-sentence at the strike of the clock. The book becomes the main technique he uses to induce the state where films and plays and books can be born. A non-judgemental zone. He writes that the workbook needs to be “so unpresumptuous and undemanding and is intended to sustain like the mellowest woman almost any number of my peculiarities.”

This is a fairly common practice, crafting a ritual where you sit down at the same time every day, in the same chair, writing in the same kind of notebook, creating a repetitiveness that borders on self-hypnosis. This is what Hemingway did, it is what Mario Vargas Llosa does. 

More techniques

Here are some other techniques people use to access and maintain the zone:

• Introducing a long delay between when you do the work and when it is shown to the world. Annie Ernaux writes about this in A Simple Passion, a memoir about how she becomes obsessed in a banal way with a man who is having an affair with her—the thought that others will read these notes about the tacky sex life of a middle-aged woman feels, to her, almost fictional. She will be far away when it happens. Therefore, she doesn’t feel a need to protect herself.

• Thinking of the work in religious terms, as a service to, or a search for, God. Bergman, Grothendieck, and Pascal all do this. It might be easier to summon the awe and daring necessary to push out into the unknown and against social pressure if the alternative is failing God. Or a fiendish muse.

• Working with talented and open-minded collaborators, if you have the chance, can be a way to enter the zone. Nick Cave, when asked how he’s been able to reinvent himself so many times as a musician, says that his bandmates, especially Warren Ellis, simply will not play anything that sounds like what he’s done before. He has surrounded himself with people whose influence is the inverse of the social pressure of normal society and his audience.

• Another idea if you want to push against the mental pressure that kills good ideas, from Paul Graham’s recent essay on how to do good work: “One way to do that is to ask what would be good ideas for someone else to explore. Then your subconscious won't shoot them down to protect you.” I don’t know of anyone using this technique, but it might work.

• Actively subvert expectations. Kristian Mattsson, who performs under the moniker Tallest Man on Earth, says he pays close attention to his emotions as he’s writing new songs. If he gets excited, purely, he immediately puts the guitar down—excitement means what he is playing something he knows others will like, something that retreads paths he has already explored and been socially validated for. The songs he’s looking for are the ones that he’s ashamed of liking.

  • Noticing these subtle differences in creative excitement requires subtle introspection. But you can be even more subtle. If we think of creative introspection as having three levels, Mattsson is on level two. (Level one is just noticing that you find an idea interesting or exciting.) Level two is noticing that your longing to be accepted can fool you to get excited about an idea that you are not actually excited about. Level three is Andrei Tarkovsky. In his diary, during preproduction of his masterpiece Solaris, the Soviet filmmaker writes that he has met a sound engineer that he considers brilliant. The sound engineer told Tarkovsky that they shouldn’t use Bach in the film because “everyone is using Bach in their films at the moment.” In the diary, Tarkovsky makes no further note, but in the film, the music is—Bach. Tarkovsky realized it didn’t matter that Bach was a popular choice that people would praise him for. It was just the right thing. This is very hard to do, so most creatives stay on level 2 and learn that what is popular is a trap. This does lead to good ideas being needlessly killed. But likely more would die if they had let what is popular kill unpopular ideas.

• Work so fast that you don’t have time to self-censor. While writing the intensely confessional My Struggle, Knausgaard forced himself to write five pages a day to overcome his tendency to freeze up in shame. Every time he acclimated to the pace of his writing, he increased the quota so he would always be overwhelmed—at one point he forced himself to write 25,000 words in 24 hours, about a third of a normal-sized novel. It is not the best writing he has done; it kind of melts at the edges. But it is true literature and, like Récoltes et Semailles and Bergman’s workbooks, it is a rare opportunity to observe an uncommon and fertile mind in real-time.

The mental states where new ideas can be born are hard to open up. And they are continually collapsing. The things you have to do to keep them from caving in will make people frown upon you—your tendency for isolation, working deep in the night, breaking norms. The zone is a place at the margin of society. But strangely enough, this fragile margin is where the ideas that make our society possible come from. Almost everything that makes up our world first appeared in a solitary head—the innovations, the tools, the images, the stories, the prophecies, and religions—it did not come from the center, it came from those who ran from it.

1

For artists, unlike scientists or mathematician, question isn’t the right word for the thing I’m referring to; perhaps a better word there is provocation or prompt—an idea or image that provokes the mind to generate profound newness. But it remains true that the ability to locate these starting points is the key, more than raw talent. A good provocation pursued with diligence leads further than a weak provocation masterfully articulated.

Some accounts suggest that a brutal assassination is to blame.

Getty Images

Here’s what you’ll learn when you read this story:

• Egypt relied on a group of former Nazi scientists to spearhead its missile program.
• In 1962, a key player in the program—Heinz Krug—left his office, never to be seen again.
• While Heinz’s ultimate fate is merely conjecture among historians, some accounts state that a brutal assassination may be to blame.

On July 21, 1962, a crowd of journalists gathered in a bleak stretch of desert near Wadi El Natrun, an ancient valley in Egypt. That day Gamal Abdel Nasser, the Egyptian president and military dictator, was determined to prove to the world that his country was a military superpower. Blasting dust everywhere, four surface-to-surface rockets shot off into the skies one after another and fell into the Mediterranean, shocking the media. Afterward, the Egyptian government celebrated with rallies, fireworks, and a plane that rained candy over Cairo as state-owned radio declared the nation had entered the “missile age” after a long effort to build up its long-range firepower.

The balance of power shifted in the Middle East overnight, sparking an arms race between Egypt and Israel, and exposing shadowy networks of former Nazi intelligence officers and rocket technicians selling their services to the highest bidder. Caught in a fierce crossfire between Israel, Egypt, and West Germany, one Nazi rocket scientist disappeared into thin air—crippling a rocket program and stumping historians for decades.

Egypt Desperately Needed Rockets

According to Michael Neufeld, PhD, former senior curator in the space history department of the Smithsonian’s National Air and Space Museum, Egypt’s pursuit of Nazi rocket scientists was par for the course in a global arms race taking place at the time.

“Missile technology was spreading around the world after World War II and the Germans were only a small part of that,” Neufeld says. “Egypt wanted to threaten Israel, notably in the wake of the 1956 attack on the Suez Canal by Israel, Britain, and France. The context for Nasser’s interest in missiles is the Arab world's hostility to Israel that ultimately led to the 1967 and 1973 wars.”

Being that powerful rockets were a natural goal for any aspiring world leader during the Cold War, Nasser—a professional soldier who swept away Egypt’s old political norms with a revolution—had big ambitions. However, he faced several pressing obstacles. Egypt lacked the large-scale factory horsepower to produce rockets. Nasser’s prime option at the time was a rather run-down facility in Heliopolis called Factory 333, a former British hospital converted into an aircraft engine factory. Despite its shortcomings, the factory would become the central node in Egypt’s missile development program.

Another enormous problem was that neither the United States nor the Soviet Union was willing to assist Nasser in developing weapons. Never one to be put off by setbacks, Nasser decided to recruit the original pioneers of missile technology—the Germans. He approached West German Cold War spy chief Reinhard Gehlen, who ran the Gehlen Organization, an intelligence agency staffed in large part by former Nazi and SS officers. According to author and former intelligence analyst Owen L. Sirrs’s book Nasser and the Missile Age in the Middle East, Gehlen connected Nasser with the famous Nazi military commando officer Otto Skorzeny.

The Rocket Science Team—and the Weapons They Built

Skorzeny, an Austrian, achieved fame for his expertise in daring operations behind enemy lines during World War II. Talented at intelligence gathering and speaking different languages, the charismatic Skorzeny successfully pulled off one of the world’s most incredible prison breaks when he snatched Italian dictator Benito Mussolini from a mountain fortress and delivered him safely to Adolf Hitler in 1943. According to Sirrs, Skorzeny trained commandos for Nasser in Egypt for about a year before leaving—only to later reappear as a phantom in rumors about the disappearance of one of Nasser’s leading rocket scientists.

Skorzeny wasn’t the only Nazi who sought new opportunities in Egypt. A group of former Nazi rocket scientists (who refused to work for either the Allies or the Soviet Union after designing weapons for Hitler) arrived in Egypt in the early 1960s to help with Nasser’s missile program. Among them were Wolfgang Pilz, who helped design self-guided V-1 “flying bombs,” and Eugen Sänger, a rocket propulsion engineer. Other technical experts included Hans Kleinwachter and Hans Goercke.

A key linchpin to their success was the rocket scientist Heinz Krug, a close colleague of Sänger. Starting in 1958, Krug worked as a manager at the Research Institute for the Physics of Jet Propulsion in Stuttgart, Germany, and developed ties to the Egyptians. As of 1960, he allegedly became a frontman for a shell company called INTRA Commercial Company with links to the Egyptian War Ministry. Krug used his office in Munich to source, procure, and ship parts for the missiles to Egypt. He was among the staunchest advocates of Nasser’s rocket program and worked hard to fulfill his critical role—not knowing that the job would ultimately lead to his misfortune.

Compensated lavishly, the scientists in 1960 kick-started two surface-to-surface missiles for Nasser with range and payloads to strike Israel. By 1961, they had already begun testing prototypes at Factory 333, the same location in Wadi al-Natrun that Nasser would hold his demonstration a year later. Additionally, teams led by the German rocket scientists began churning out missiles at two additional locations, Site 36 and Site 135, both located farther down the Nile near Memphis and Helwan. In a short time, the Germans had succeeded in producing rockets at speed and scale. Nasser paraded 20 missiles through Cairo in 1962—sparking fears around the Middle East, especially next door in Israel. All the while, Israel was working on its own weapons program and beat Nasser to the punch, successfully launching a suborbital missile, which only added to the military dictator’s paranoia. Even with the advantage, Israel wouldn’t let up.

What Happened to Krug?

Feeling threatened by Egypt, Israel pressed West Germany to stop the former Nazi scientists, given that the scientists were German citizens. The German spy chief Gehlen opted for closer cooperation with Israel. Sirrs suggests West German intelligence may have directly assisted agents from Mossad—an Israeli intelligence organization—in targeting the missile program. He says evidence suggests that Mossad and the West German government used an intimidation campaign to overtly pressure the scientists to stop helping the Egyptians. German federal officials forced Sänger to resign in November 1961 from his own institute. While many scientists left the team, Krug continued to work on the missile program from his office in Munich.

Israel’s campaign didn’t end there. In 1962, Mossad launched Operation Damocles. Events that followed indicated that its first target was Krug, the effective lead production coordinator for the rockets. On Tuesday, September 11, 1962, Krug was working in his office when a young man came to see him, according to news reports. The man gave his name as “Mr. Saleh” and spent about half an hour chatting with Krug in his office. Then they drove off together in Krug’s car.

Krug never returned, and his terrified wife quickly reported him missing. “My husband has been carried off, for he had absolutely no reason to disappear of his own free will,” she told police, according to the Associated Press.

Police found the scientist’s car on Friday in the Munich suburb of Solln, within driving range of several vast wooded areas. The abandoned car was caked with mud. Soon after, the police allegedly received an anonymous phone call stating that Krug was dead. Nevertheless, the police appealed to the public on television for information and launched an investigation with Interpol to uncover his whereabouts.

In the wake of Krug’s alarming disappearance, more German scientists working in Egypt were targeted. Pilz’s secretary opened a letter bomb that left her blind from the explosion. Another letter bomb killed five Egyptians and wounded six at another rocket factory office the next day. Kleinwachter, a stubborn holdout, barely avoided being shot by a trio of Mossad agents in 1963 in Bavaria, escaping with a bullet hole through his neck scarf. He publicly accused the West German government of deliberately failing to protect him.

The German government, in turn, ended all financial contracts with any companies involved in Egyptian missile parts procurement and sent formal warning letters to all scientists in 1963 urging them to cease their activities. A few scientists persisted until 1964, but they lived in fear—carrying guns for protection and screening all their mail for explosives.

Krug’s fate and the possible whereabouts of his body—if indeed he died—remain unknown, nothing more than conjecture among historians. Some reports allege he was spirited away to Austria and lived out the rest of his life under an assumed name. Other accounts suggest that he met with a merciless fate. For instance, some indicate that a Mossad assassination squad shot Krug and buried him in a forest. Even stranger tales allege that Otto Skorzeny, the ex-Nazi commando affiliated with Gehlen, was blackmailed by Mossad into orchestrating Krug’s death and shot him. To Sirrs, however, Skorzeny is an unlikely culprit when it comes to Krug’s potential assassination.

Still, Skorzeny remains a ghostly figure on the fringes of German and Egyptian intelligence activities. Intelligence sources approached for this article declined to comment on Skorzeny’s potential affiliation with the CIA. Nonetheless, someone walked away from the abandoned car in Munich knowing exactly where Krug was—taking the secret to the grave.

The Magnetophon also gave rise to the laugh track

Bing Crosby championed the Magnetophon, a hi-fi reel-to-reel tape recorder used to play back prerecorded performances on the radio.

Pavek Museum

A German engineer wanted a cheaper cigarette. The popular crooner Bing Crosby wanted a vacation. Satisfying both desires inadvertently led to the invention of the laugh track. Along the way there were Nazis, spoils of war, and more than one accidental encounter. Tying together this quirky history is the Magnetophon.

What Was the Magnetophon?

The Magnetophon was a high-fidelity reel-to-reel magnetic tape recorder. The hit of the Berlin Radio Show when it debuted in 1935, it was developed by the German electronics manufacturer AEG. The magnetic tape was produced by I.G. Farben (now known as BASF).

Black and white photo of a balding white man with a mustache and pince nez.

German inventor Fritz Pfleumer came up with the idea of recording sound on magnetized paper tape coated with metal.

ullstein bild/Getty Images

The tale of that tape runs through German inventor Fritz Pfleumer. In the early 1920s, Pfleumer was working on industrial paper products in Dresden. At the time, fancy cigarettes had gold leaf to decorate the tip. Cheaper manufacturers achieved a similar effect using colored paper, but the dye could stain smokers’ lips, and no one wanted that. Pfleumer devised a less expensive process using powdered bronze to simulate the gold band.

Pfleumer didn’t work in the recording industry, but he was familiar with the technology of electromechanical recording, which was done on metal wire—a 1898 invention of the Danish engineer Valdemar Poulsen. Pfleumer thought he could do something similar with paper by replacing his powdered bronze with a magnetized material. He patented his “sounding paper” in 1928 and also invented an audio tape recorder to go with it. The sound quality wasn’t great, and the paper tore easily, but it was the start of a promising idea. Two points in its favor: The paper could be sliced, allowing edits, and it could be erased and re-recorded over.

Pfleumer knew he needed to partner with a larger company to commercialize his idea, and so he signed a contract with AEG in 1932. Hermann Bücher, chairman of the AEG board of directors, took a personal interest in the project, helping shepherd it to completion. Although AEG originally planned on developing both a recorder and the tape, it didn’t take long for Bücher to reach out to his friend Wilhelm Gaus, managing director at I.G. Farben. AEG developed the hardware, while I.G. Farben worked on the tape.

The two teams dubbed their product the Magnetophon, or magnetic phonograph, and planned on launching it at the 1934 Berlin Radio Show to compete directly with machines that recorded on steel tape or wire. But the product wasn’t quite ready. Two days before the show, they canceled the debut.

A year later, the bugs had been worked out. The Magnetophon’s introduction at the 1935 show was a resounding success. AEG fielded inquiries for many variations on the recorder, including one that combined the recorder with a telephone, a player for prerecorded music, and a special version to add artificial reverberation for recording open-air concerts. Over the next three years, AEG developed several iterations, resulting in the Magnetophon K4 in 1938, its first commercially successful tape recorder.

The K4 eliminated the hiss and distortion that magnetic recordings previously suffered from by incorporating AC bias, which added a high-frequency signal, typically around 40 to 150 kilohertz, to the recording. Inaudible to the human ear, the signal reduced distortion, especially when recording quieter passages. The fidelity of the Magnetophon was such that radio listeners couldn’t distinguish between live broadcasts and prerecorded performances.

The Magnetophon During and After the War

This is where the Nazis come in. Adolf Hitler and his propaganda minister, Joseph Goebbels, understood the power of radio. The Magnetophon became a powerful tool for both political messaging and military strategy. Because the device could record and play back sound at the same quality as a live radio broadcast, it allowed Hitler’s recorded speeches to be aired from one radio station while the dictator was in another part of the country. This made it more difficult for the Allies to pinpoint his location.

Meanwhile, World War II disrupted the exchange of technical information and prevented Americans from learning about the Magnetophon until after the war. At least, that’s the shorthand version of this history I kept running across during my preliminary research for this column.

Smiling World War II-era U.S. Army officer in uniform and cap, shown in black and white.

1945 U.S. military certificate authorizing captured German Morse code recorder equipment

During World War II, U.S. Army Signal Corps engineer Jack Mullin [top] came across the Magnetophon. After the war, he got approval [bottom] to ship two of the disassembled machines and reels of magnetic tape back to the U.S.

Top: Pavek Museum; Bottom: Richard L. Hess/The Mullin Family Collection/Archive of Recorded Sound/Stanford University

But then I read Friedrich K. Engel’s account in the book Magnetic Recording: The First 100 Years, which provides a wealth of detail about the Magnetophon. Among other things, Engel notes that the AEG affiliate in Schenectady, N.Y., received a Magnetophon in November 1937, well before the United States entered the war, and AEG engineers demonstrated it for their colleagues at nearby General Electric. The GE engineers dismissed the technology out of hand, though, and so Americans had to wait until after the war for the Magnetophon to be reintroduced.

We have electrical engineer John T. “Jack” Mullin to thank for that reintroduction. Mullin served in the U.S. Army Signal Corps, stationed in the United Kingdom and Paris during the war, and he liked listening to the radio. He realized that “live” orchestral broadcasts coming from Germany in the middle of the night, when no musicians would have actually been in the studio, lacked the telltale hiss and crackle of prerecorded music. Clearly, German engineers had recording technology far superior to the Americans’.

Sent to Germany at the war’s end, Mullin eventually came across that technology at a radio station, and in 1945 when he returned home to California, he shipped two disassembled Magnetophons, a case of tape, schematic drawings, and the determination to change the U.S. recording industry.

Bing Crosby Championed the Magnetophon

Enter Bing Crosby. In the 1940s, Crosby was perhaps the most popular performer on the radio. But he was tired of performing two weekly shows three hours apart (one for each coast). He wanted to prerecord his performances and take a break. He sent in his lawyers to talk to executives at NBC, which aired his program. The audio recording technology at the time used vinyl or shellac transcription discs, but radio listeners could hear the pops and hisses and knew it wasn’t live. NBC refused Crosby’s request, and so Crosby took a year off from radio and then signed with the upstart network ABC, which was willing to let him record his shows for later airing.

Black and white photo of two men in suits facing each other and standing in front of radio equipment.

Radio producer Murdo MacKenzie [right] arranged for Jack Mullin [left] to demonstrate the Magnetophon to Bing Crosby in 1946.

Pavek Museum

Serendipitously, Mullin had begun demoing the Magnetophon in California. In October 1946, Crosby’s technical producer, Murdo MacKenzie, heard about the demonstrations and arranged one for Crosby at the Metro-Goldwyn-Mayer studios in Hollywood. Crosby was delighted and promptly invested US $50,000 (about $800,000 today) in Ampex, the company working with Mullin to engineer an American version of the Magnetophon. Mullin became Crosby’s chief engineer. 3M developed the magnetic tape.

In 1947, Crosby became the first major radio star in the United States to prerecord performances. Much of the success was due to the recording equipment, but Mullin was also an excellent editor. Crosby and his team would record multiple takes, and Mullin would deftly splice together the best to create a seamless performance.

Black and white photo of a smiling man in a suit and hat sitting next to a reel to reel tape recorder.

After seeing a demo of the Magnetophon, Bing Crosby invested $50,000 in Ampex, which developed a U.S. version of the tape recorder.

Cinematic/Alamy

One day a guest told a joke that was uproariously funny, but a little too spicy for radio. Even though the joke couldn’t be aired, the audience’s laughter was worth keeping. Soon, the producers created a whole catalog of different laugh tracks. If a joke didn’t land, it didn’t matter. The editor could just add a laugh in postproduction, from polite titters to hearty guffaws.

According to Crosby’s daughter Mary, Bing didn’t have a problem with this type of editing. The live audience was immaterial as long as the jokes were funny. But according to Mullin’s daughter, Eve Mullin Collier, the manipulation didn’t sit well with her father. He disliked the inauthenticity of the moment, of editing joy.

The history of technology is filled with such episodes of unintended consequences. Mullin admired the Magnetophon precisely because it could faithfully record a performance, an event, a moment in time. He worked tirelessly to refine the machine for the benefit of radio audiences everywhere. And so I understand why its appropriation for capturing canned reactions and manipulating reality must have rankled. He championed the technology, but ultimately it moved beyond his control.

Part of a continuing series looking at historical artifacts that embrace the boundless potential of technology.

An abridged version of this article appears in the August 2026 print issue as “Birth of the Laugh Track.”

References

Luis Felipe Eguiarte Souza, curator at the Pavek Museum of Electronic Communication, in St. Louis Park, Minn., first told me about the Magnetophon and its link to the laugh track. The machine pictured at top is on display at the Pavek, and is one of the two that Jack Mullin brought back from Germany and then rebuilt.

For significantly more technical detail on the development of the Magnetophon, magnetic tape, and its reinvention in America, check out Chapter 5, “The Introduction of the Magnetophon,” by Friedrich K. Engel, and Chapter 6, “Building on the Magnetophon,” by Beverley R. Gooch, in Magnetic Recording: The First 100 Years (IEEE Press, 1998).

Radiolab interviewed Mary Crosby and Eve Mullin Collier as part of the show “Mixtape: Jack and Bing,” which includes many archival recordings as it tells the story of the development of the laugh track.

There is a 2006 documentary on Jack Mullin titled Sound Man: WWII to MP3, but I was unable to view it.