TheDaily Front

Singapore’s National Library Board offers micropayments via a five-year pilot to build daily reading habits among a phone-first population

Image: ReadSG

Key Takeaways

• Singapore’s ReadSG rewards 15 minutes of daily reading with 20 virtual coins redeemable for cash.
• Reach 7.5 million collective reading minutes to unlock a S$150,000 charitable donation via Read for Good.
• Governments worldwide watch ReadSG as a test of gamification shifting screen habits at population scale.

Singapore’s National Library Board launched ReadSG on 6 September 2026, a five-year national reading campaign that converts logged reading time into virtual coins redeemable for cash. The initiative has drawn attention from governments and educators worldwide who are watching to see whether a small financial reward can reshape phone-centric habits.

How the Scheme Actually Works

The program uses a simple coin mechanic: log reading time on a government platform and earn rewards redeemable for cash.

Log at least 15 minutes of reading on GovTech’s CrowdTaskSG platform and you earn 20 virtual coins. The conversion rate is fixed: 1,000 coins equal S$1, putting one daily session’s value at roughly S$0.02.

Only one session per day counts toward your coin balance. Fifty consecutive days of reading earns you a single Singapore dollar, according to The Times and Indulge Express.

The National Library Board has characterized the payout as deliberately modest, designed as a behavioral nudge rather than an income stream, according to coverage in multiple outlets. This is not a side hustle.

A parallel track called Read for Good adds a communal dimension to the campaign. The program targets 7.5 million cumulative reading minutes across all participants, a collective goal that would unlock a charitable donation of up to S$150,000, according to Indulge Express.

Your 15 minutes of daily reading is no longer just a personal habit. It contributes to a shared number with a concrete social outcome attached.

A Decade of Trying to Get Singapore Reading

ReadSG builds on nearly a decade of national literacy initiatives, but arrives with a new gamified mechanic and a pilot-phase designation.

ReadSG follows Singapore’s National Reading Movement, launched in 2016, and a series of earlier literacy initiatives documented by The Financial Coconut. The National Library Board has described the current campaign as still in a pilot phase, according to the Straits Times, with plans to refine the program based on user feedback.

Critics note that the financial reward is too small to attract anyone who was not already inclined to read. Supporters point to behavioral economics research, which broadly suggests that even trivial incentives can help establish a new habit when participation friction stays low.

The coin mechanic borrows from the familiar language of fitness apps and loyalty programs. Those platforms have spent years conditioning users to close their rings or collect their points.

What Singapore’s Experiment Means Beyond Its Borders

Governments and educators are watching ReadSG as an early test of whether gamification and micropayments can shift public behavior at population scale.

Gamification and micropayments applied to public-interest behavior represent a relatively new frontier in civic design. Libraries and education agencies globally are hunting for tools that can compete with a notification feed.

If ReadSG generates strong participation data over its five-year run, it may become a case study for other cities to examine. The behavioral data collected could prove valuable to policymakers trying to understand what genuinely pulls people away from their screens, according to analysts following the campaign.

It says something specific about 2026 that a government created a financial incentive to make reading competitive with a scroll. Whether 20 virtual coins is a meaningful answer depends on one thing: how many people open a book tomorrow and remember to log it.

This article tells the last part of the story of Lemmings.

For a while there in the early 1990s, there seemed to be good reason to believe that Lemmings was on its way to becoming one of gaming’s perennial franchises. Developed by DMA Design of Dundee, Scotland, the original Lemmings game took the world by storm upon its release in 1991. From its first home on the Commodore Amiga, its Liverpool-based publisher Psygnosis brought it to no fewer than 23 other platforms, from natural habitats like MS-DOS and the Apple Macintosh to such refugees of the 8-bit era as the Sinclair Spectrum and the Amstrad CPC, from cutting-edge multimedia appliances like the Philips CD-i and the 3DO to handheld gadgets like the Atari Lynx and the Sega Game Gear. In all of its incarnations, it outsold any other game ever to have passed through the hands of Psygnosis by a veritable order of magnitude.

The combination of simplicity, juiciness, addictiveness, and cuteness was truly a lethal one. Psygnosis had to set up an entire call center in the United States to deal with the avalanche of frustrated players begging for hints on this or that devious level. The little creatures made appearances in newspapers, glossy magazines, and mainstream news broadcasts all over the world, even as playing the annual holiday Lemmings mini-games that could be found stacked as stocking stuffers next to cashier stands threatened to become a new Christmas-morning tradition for families. Lemmings joined Tetris and SimCity as one of the computer games that the zeitgeist deemed acceptable for those who did not self-identify as gamers to indulge in. Like those two others, it was an early harbinger of the casual revolution in gaming that would begin to arrive in earnest a decade later.

But whereas Tetris and SimCity would be grandfathered into that revolution at the turn of the millennium — SimCity spawning The Sims, the 800-pound gorilla in the new space — Lemmings would have flamed out by then, being remembered, to whatever extent it was remembered at all, more as a kitschy fad than as an enduring entertainment staple. I quite like the early Lemmings games, for reasons that I’ve explained in detail in the past. Therefore I came to this article with a simple question: just how did the franchise manage to squander all of that early momentum so quickly and completely? And as usual, I found that the answer comes down less to the choosing of a single ill-advised fork in the road than to a series of smaller decisions that doubtless seemed like a good idea at the time, but were exposed as less than good in retrospect.

Lemmings 2: The Tribes, the big sequel to the original game — it was actually the third full-sized game in the series, coming after the less inspired Oh No! More Lemmings, which was largely filled with levels that had been rejected for its predecessor — was released in 1993 amidst huge anticipation and expectation. In many ways, it lived up to its advance press. In place of the suite of eight special abilities with which a player of the earlier games could invest her lemmings, it provided no fewer than 60 of them. The subtitle came from the fact that the lemmings were now divided into twelve tribes, each with their own abilities, appearance, and environments to traverse. The whole endeavor also boasted a new thread of narrative, about helping the twelve tribes escape from captivity. (The Biblical overtones were apparently intentional.)

As a player who had greatly enjoyed the first Lemmings, I found it to be a brilliant sequel. It was, however, every inch a sequel, being built strictly for the gratification of someone like me: someone who had a lot of experience with what had come before. This was not the best place to start your Lemmings journey, as the kids like to say today — not with all of the additional complexity, not with the disappearance of the gentle tutorial levels that eased you into the first game. Inspired as it was on its own terms, *The Tribes‘*s approach was questionable for a mass-market series of more or less casual games, which the customer should ideally be able to hop on and off of with the same aplomb as a subway rider. Then, too, the heavier weight of the game meant that it could be ported to only eight instead of 24 platforms. Lemmings 2 was a solid hit by all of the normal standards of the games industry, yet it failed to do much to further raise its franchise’s profile as a budding pop-culture staple. The original game, which was now available at a greatly reduced price, in all likelihood outsold it considerably even in the year of its release.

One of the problems, if we want to call it that, was a growing mismatch between the expectations of the market and the instincts and desires of DMA Design. The Scottish gang were gamer’s gamers of the old school, whose two creations before Lemmings had been the straightforward shoot-em-ups Menace and Blood Money. Lemmings had been something of an aberration for them; the cuteness had come about almost by accident, a case of mordant laddish humor ramming slightly askew. If DMA were to continue with the series, they wanted to make it more complicated, more difficult, more ambitious… in short, more hardcore. This was not a recipe for success in the space the first Lemmings had opened up for itself. After The Tribes failed to set the world on fire, DMA felt pressure from above to revert to the mean. This bred an atmosphere of resentment in Dundee that was hardly conducive to productive game development.

As was typical of such contracts at the time, the Lemmings intellectual property actually resided with the publisher rather than the developer. And the former was undergoing big changes. Just before the release of The Tribes, Psygnosis and its in-house studio — the same one responsible for most of the Lemmings ports — were acquired by the Japanese electronics giant Sony, which was looking for a Western partner to make games for its upcoming PlayStation console. This event, combined with the success of Lemmings, raised Psygnosis’s profile enormously, from a niche purveyor of graphically dazzling but often gameplay-deficient Amiga action games to a real force to be reckoned with on the modern entertainment landscape. The Children’s Television Workshop, the maker of Sesame Street, entered into talks with Psygnosis about a Lemmings television show.

It was a tempting prospect on the face of it, but it became problematic as soon as you started to dig into it. Television shows, whether aimed at adults or children, are built around memorable characters. DMA’s lemmings were not quite this. They were a generic mass, all of them identical, the very definition of a faceless horde. Each blobby little creature filled a space no more than ten pixels square on the screen. The lemmings were many things, but they were not Super Mario or Sonic the Hedgehog, much less Big Bird or Oscar the Grouch.

So, an edict went down from Psygnosis to DMA to find a way to give the lemmings more individualized personalities. For their next trick, DMA had been toying with a continuation of the story of The Tribes — or rather a whole set of continuations. The next four Lemmings games would each deal with the fate of three of the twelve tribes after they left the Ark the player had helped them reach last time around. The whole series would be called All New World of Lemmings. (In North America, it would take the name of The Lemmings Chronicles; unlike the vast majority of such re-christenings, this was actually a better name in my opinion.) To satisfy Psygnosis and the Children’s Television Workshop, DMA now promised to make the lemmings bigger, and to fill the menu and victory and failure screens with cute faces that looked different from one another and could play well on television.

If matters had transpired differently, All New World of Lemmings would have been the banner title of a new sub-series, with different subtitles for each of its entries. As it was, Psygnosis just dumped the would-be first entry of the sub-series out there on its own, sans subtitle, and abandoned the other nine tribes to their fate.

“I couldn’t think of any new way to take Lemmings at this point,” admits David Jones, the series’s mastermind from the beginning. Desperate for some sort of gameplay distinction from what had come before, DMA decided to combine the newly television-friendly presentation with yet more complications to the formula — complications that proved not to be much fun. You could no longer invest your lemmings with specific skills from a master pool of same just by clicking on them; now, you had to have them walk over loot boxes scattered around the levels, an incredibly fiddly and annoying mechanic. Meanwhile the increased size of the lemmings had unexpected knock-on effects. When the creatures were tiny, watching them die was funny even for people who reacted to the violence found in other videogames with horror and outrage. But when they became bigger, that changed; their deaths became grislier, possibly amusing for a teenage boy but less so for his mother. In addition to its gameplay issues, the tone of World of Lemmings was badly off in relation to the franchise’s established personality.

You have to walk your lemmings into crates like this one to give them special abilities now. Everything about the gameplay and aesthetics of World of Lemmings hits wrong, sometimes subtly so, sometimes blatantly. This was not promising television fodder.

Not long into development, a feeling took hold in Dundee that they were all just done with Lemmings, ready to come out from under the shadow of the casual monster they had accidentally created and return to making the games they wanted to make. As it happened, this was to be the last game of a six-game contract they had signed with Psygnosis back in the 1980s. Just get ‘er done became the watchword, so that they could take their Lemmings loot and do something more exciting with it. The new passion project around the office was a little something called Race ‘n’ Chase, which would evolve into a bigger something called Grand Theft Auto. “Lemmings 3 was a bit crap,” says DMA’s Mike Dailly, the artist who drew the very first eight-pixel-high versions of the creatures. It was done “more to end our commitment to Psygnosis than to actually do a good game.”

Unsurprisingly, then, it did not end up being a good game. Released in 1994, it made the concept that had seemed so fresh barely three years earlier feel trite and stale and kind of tasteless; even the soundtrack was annoying, the same two-bar fragment of circus music pounded into your head ad nauseam. DMA’s ennui oozed from its every slapdash pixel. This was a genuinely new thing under the sun: an outright bad Lemmings game. Evidently knowing it had a turkey on its hands, Psygnosis did virtually nothing to promote it, a huge contrast to the extensive outreach and advertising that had been done for The Tribes. It appeared on only two platforms, MS-DOS and the by now fast-fading Amiga. Those magazines that bothered to review it at all generally weren’t kind.

This first bad Lemmings game combined with a distracted Psygnosis, now busying itself making fresh games to order for the Sony PlayStation, to let all of the air out of the Lemmings balloon, practically all at once. The negotiation with the Children’s Television Workshop died on the vine. Lemmings was in acute danger of being exposed as a one-hit wonder, fodder for nostalgic “Where Are They Now?” retrospectives, rather than the enduring entertainment icon it had so recently seemed destined to become.

The following year the PlayStation dropped; it would go on to become the most successful single games console of the entire twentieth century, redefining the broader culture’s view of digital games forever with its edgy advertising and its deliberate courtship of the booming rave scene. Psygnosis was a big part of all that, through such early PlayStation hits as WipEout and Destruction Derby. Betwixt and between, it tried to get some more mileage out of Lemmings, efforts which came across rather like a child halfheartedly poking a dead dog with a stick. Having now parted ways with DMA Design, Psygnosis had to give its latest Lemmings projects to other developers with no previous connection to the franchise.

In 1995, we got Lemmings 3D from an outfit called Clockwork Games. The episodic approach of All New World of Lemmings was abandoned, in favor of the technological gimmick of the title. In a sense, the franchise became a pioneer one last time, albeit of a more dubious sort than before: Lemmings 3D was an early manifestation of an industry mania for stuffing 3D graphics into absolutely everything, whether they made sense there or not. “The fundamental core of the game of digging or building across the landscape to rescue the lemmings didn’t need an extra dimension,” says Gary Timmons of DMA Design, who could now only look on from afar as Psygnosis subjected their concept to ever more unfortunate contortions. Mike Dailly is blunter: “It was badly thought-out and just plain rubbish.” And truly, Lemmings 3D is agony to play; the newly mobile camera always seems to be pointing just where you don’t want it to be, forcing you to spend more time trying to get the viewing angle right than manipulating your lemmings. Nobody ever asked for this.

3D Lemmings Winterland marked the last gasp of the tradition of Christmas-themed Lemmings mini-games. The spirit was willing, but the 3D engine was lacking.

With the semi-traditionalist (but in 3D!) approach having failed, Psygnosis swung and missed in 1996 with Visual Science’s Lemmings Paintball — yes, really — and the in-house-developed The Adventures of Lomax. Here we can see the publisher still trying to do what the Children’s Television Workshop had once requested, turning the lemmings into characters to be moved in and out of different gameplay paradigms, something Nintendo had long done for its stable of trademarks with remarkable fluidity. But it was far too late in this case, and far too poorly done. The Adventures of Lomax gave a lemming a name of his own for the first and only time — for, of all types of game, a Super Mario Bros.-style platformer. The kindest thing to be said about it is that some of the art — pixel art again this time — is quite lovely, even as the gameplay is more or less acceptable in a workmanlike sort of way. But it never feels like it has much of anything to do with Lemmings.

After this, Psygnosis seemed to have decided to let the dead dog lie — until, that is, another, final Lemmings game popped up out of the blue almost four years later. It did not succeed in rescuing the franchise from oblivion; it came and went from the budget bins barely noticed by the gaming press. But it did succeed, against long odds indeed, in being a really, really good Lemmings game, by far the best since The Tribes. It’s hard to imagine any anticlimactic swansong acquitting itself much better.

Lemmings Revolution is not without a gimmick, but for once it’s a reasonably interesting one. Instead of stretching out from left to right, each of its 102 levels is wrapped around a giant pillar that you can rotate at will. The gimmick is not what I would call revolutionary in the non-physical sense, but it mixes the usual formula up in enjoyable ways when it’s at its best, and is never actively irritating even at its worst.

But more important to this game’s success than the gimmick are the cleverest, most inspired set of levels since The Tribes. The team who made it, who were housed at Psygnosis’s satellite studio in Leeds, brought a passionate whimsy to the project that the series had been sorely lacking in its last several installments. Co-designer Mat Thomas describes a creative atmosphere that resembles the one from back in the day at DMA Design, when the idea of Lemmings was still fresh and everyone was eager to get an oar in with a level or two.

The entire development team were delighted to work on such a famous series, and we took the challenge. We had a fabulous editor developed by a programmer called Ben Dixon. It was very easy to mock up and create levels. The key to level design with Lemmings is trying to think of ways to use the skills presented to the player in different ways. Inspiration came from each other, and other mediums such as the earlier Lemmings games and, dare I say it, films! Goonies — one of my levels — was inspired by the named film. I tried to put in many gadgets to give that feeling of technology driving the level. Our team created nearly 200 levels for Lemmings Revolution, which allowed us to pick the cream of the crop.

The “Goonies” level mentioned by Mat Thomas.

The finished game goes back to the roots in many ways. The suite of abilities with which you can equip your lemmings is the exact same collection as the original game: the familiar climbers, parachutists, bombers, and bridge builders, plus the three kinds of diggers. Seeing them again feels like meeting old friends.

Yet by no means is Lemmings Revolution a carbon copy of the first game in the series. In addition to the revolution mechanic itself, there are any number of new obstacles and affordances within the levels: switches, saws, lasers, instantaneous transporters, and, most ingeniously, gates that reverse the force of gravity to make your lemmings walk on the ceiling. All of this contribute to a slightly more mechanized feel this time around. There are now weasels to contend with as well; these consider a lemming to be a delightful snack, and must be either avoided or — more satisfyingly — done away with in one way or another. Another new wrinkle comes in the form of special lemmings who are impervious to either drowning in water or being scorched by acid, the better to handle the pools of same that are occasionally found in their way. Exits are now hot-air balloons that must be reached, and sometimes there are more than one of these, with each only able to accept a limited number of riders. The most important point is that all of the additions feel organic to the experience rather than tacked on by some focus group somewhere. Thankfully, no attempt is made to individualize the lemmings and turn them into fodder for television. We’re back to the juxtaposition of generic cuteness and morbidity that made the series stand out during its glory days.

For the first time since The Tribes, these lemmings look like they should.

All told, then, playing this Lemmings game is like going home again. Early training levels yield to more fiendish ones that will have you scratching your head by the middle stages, pulling your hair out by the last ones, as you have to make use of every single affordance at your disposal to succeed. As many of you know by now, I adore games that build up gradually in just this way to challenge their players. And I’m proud to say that my wife and I rose (gradually) to the challenge; we made it all the way to the end.

That Lemmings Revolution succeeds as well as it does in so many ways begins to seem still more remarkable when one considers the rest of its development history. After first agreeing to pay for this last kick at the can for the franchise, Sony abruptly pulled its funding just as it was nearing completion. Through some last-minute scrambling, Psygnosis’s management was able to place the game with the rival publisher Take Two Interactive and see it released for Microsoft Windows if not for the PlayStation.

Granted, some scars from the trauma were left behind. Several levels were bugged badly enough in the initial release to be impossible to complete, until Take Two came out with a patch. Even in the final version of the game, one level — the one called “Lock In” — can be completed only by exploiting a glitch in the game engine. It’s so at odds with the rest of the level designs, which are sometimes diabolical but always scrupulously fair, that I can’t believe this was intentional. I even fancy I can see how the level was supposed to be solved. At any rate, if you decide to play the game, I recommend that you save yourself a lot of frustration and just turn to a walkthrough when you get to this level.

One nice touch in Lemmings Revolution is the ability to choose to some extent what level you tackle next. The green dots represent solved levels, the yellow those you haven’t yet completed. Once you beat a level, you open up the two that are connected to it by arrows. It’s a little hard to see here because I’m boring and just tackle the levels in order, top to bottom and left to right. But if you’re less stubborn and methodical than me, you can take a break from a level that’s giving you trouble and work on another one for a while. Like so much else here, this is just really good, player-friendly game design.

It’s obvious that Lemmings Revolution didn’t have an overly lavish budget even before Sony pulled the plug on it. A thin glaze of story — something or other to do with those weasels who can sometimes be found lurking in the levels — is presented via an opening movie that probably absorbed about half the budget on its own. It’s never mentioned again afterward — not that it really matters. We are given only as many bells and whistles as are necessary to make the levels go and remind us that we’re playing a Lemmings game. The closing movie lasts all of seven seconds; the operative philosophy was apparently that it’s better to tempt the punters in the shops who might buy the game than it is to reward the already captured players who have put many hours of their life into it.

All of which is fine really. By the turn of the millennium, so-called “mid-priced releases” like this one — Lemmings Revolution sold for just $20 from its very first day on store shelves — were often more interesting and innovative than the expensive AAA opuses, whose publishers felt more of a need to protect their investment via micromanagement and risk aversion. Certainly the production values of this Lemmings game are no worse than those of the original. We never came to Lemmings to participate in epic stories; we come to face puzzles and to curse and stomp and almost throw our mice across the room, until a light bulb goes off somewhere in the old noggin and the fingers do what they’re supposed to and we solve that level. At which point it’s on to the next level, to repeat the procedure. We are strange creatures, we humans, aren’t we? Even stranger than lemmings, one might want to say.

Anyway, this humble game called Lemmings Revolution is one of my favorites of all the ones I’ve played from the year 2000 for these histories. I’m happy to give it a place in my personal Hall of Fame and to recommend it to all of you — especially any of you who might have enjoyed the more famous and popular Lemmings games, and thought the later ones had nothing comparable to offer. You were mostly right — but only mostly.

For better or for worse, there isn’t much else to say about the series after Lemmings Revolution came and went with so little fanfare (or, one has to suspect, sales). There have been occasional remakes over the past quarter-century, mostly for mobile platforms, but nothing that displays much in the way of design or commercial ambition. As late as 2010, the franchise was frequently mentioned as being “ripe for revival” for a new era where unabashedly simple and cartoony casual games were by some reckonings out-earning the hardcore ones. But for whatever reason, that revival was never seriously attempted, and the cultural window for the franchise is probably closed by now, what with most of us who were there when it was so huge having entered into our fifth or sixth decade of life by this point.

This is no tragedy. Lemmings came to the zeitgeist and then it went; such is the fate of most pop culture. Good game design, however, transcends trends and fads. If you’re a fan of puzzle games and you haven’t played these ones, know that Lemmings, Lemmings 2: The Tribes, and Lemmings Revolution, the three high points of the series, can still be very vexing and satisfying indeed. This is as true today as it was a quarter-century ago, as true as it will still be a quarter-century on. Even if everyone stopped making games tomorrow, we would still live in an era of unprecedented ludic riches.

Sources: The books Grand Thieves & Tomb Raiders: How British Video Games Conquered the World by Magnus Anderson and Rebecca Levene and Jacked: The Outlaw Story of Grand Theft Auto by David Kushner. Retro Gamer 39; Computer Gaming World of September 2000.

Online sources include “The Making of Lemmings” by Rich Stanton for Read-Only Memory, “An Ode to the Owl: The Inside Story of Psygnosis” by Damien McFerran for Time Extension, a Lemmings Universe interview with Mat Thomas, and Damian Katz’s gamebook page.

Where to Get Them: Oddly, none of the vintage Lemmings games are currently available for purchase. You can find downloadable versions of the original Lemmings and Lemmings 2: The Tribes in my earlier articles associated with those games. As for Lemmings Revolution, the third of the three Lemmings games that are well worth revisiting today: you can find it on a certain well-known archiving site if you only search for it. After you install it, you will need to install a patch to make it run properly on modern versions of Windows (and Linux systems under WINE).

Danny Bate on the Linguistic History of Our Alphabet

“The letter W is a child of the fall of Rome. In the fifth century CE, the western half of the Roman Empire disintegrated into a patchwork of new kingdoms and new rulers. The reasons behind this collapse of imperial power are complex, but a large role was played by various peoples who had formerly lived outside its borders. The Romans might have looked down on these migrants as ‘barbarians’, but they also increasingly came to rely on them for military support. They were foederati—peoples bound by treaty to fight Rome’s enemies in return for land and food. It was only with the help of the foederati (a Latin word related to English federation) that the Romans were able to see off the threat of Attila the Hun in 451. Yet the more power these regional leaders had, the less authority the emperor and the central state could wield. This culminated in the overthrowing of the last emperor in the west in 476.

Language would have been a part of the divide between Roman and barbarian. By the fourth and fifth centuries, the western empire had become overwhelmingly Latin-speaking. By contrast, the newcomers spoke their own languages, perhaps with a passing knowledge of Latin too. From what we can tell, a great many of these migrants spoke Germanic languages. One of these incoming tongues was the ancestor of the language you are reading right now, English, which arrived in the remains of Roman Britain during this era. Germanic-speaking elites could now be found from southern Spain to the coasts of the North Sea. These new rulers were keen to sell themselves as legitimate successors to the emperors, and there was considerable continuity during this turbulent period.

By keeping up appearances and styling themselves as good Romans, they could dampen the jealousy of the old aristocracy and gain popular support. The new kings did not insist that scribes ought to write official documents in their own Germanic tongue, but eagerly adopted the more prestigious Latin language. This worked fine most of the time, but might occasionally hit a snag. Latin-writing lands were now ruled by men whose names contained un-Latin sounds. A new king might want his scribes to draw up a charter for some great display of generosity, but how were the scribes to spell that king’s name?

One of the troublesome sounds for writers was /w/. This is the common consonant in English water and want, and it would have been present in kingly Germanic names like Clovis, Vitiges and Odoacer. The trouble was, the Latin alphabet now had no letter for this sound.

In ancient times, you would’ve heard the sound /w/ all around the Mediterranean Sea. Both Latin and Ancient Greek once used the sound, and both the Romans and the Greeks had letters to spell it. This was a sound that they, just like English, had inherited as part of their common Indo-European ancestry. Yet, as we saw in Chapters F and U, it was now foreign to them. In Greek, the consonant and its letter Ϝ had faded away, while in Latin, V had come to stand for the fricative /v/ instead.91 Time and time again, we find the ancient sound /w/ being lost or altered across the Indo-European family of languages. It would later happen in Continental Germanic languages too; in German today, W stands for /v/. The English consonant /w/ is actually a rare survivor, rescued from potential change by its migration to the island of Britain.

Out of the meeting of languages and writing in the new post-classical world, a letter was born to spell the alien /w/. From the sixth century onwards, likely starting in the powerful kingdom of Francia, innovative scribes doubled U. Within Latin texts, we find Germanically-named individuals like the abbot UUandeberctus and King UUaldemarus. The two letters were increasingly written as -one, and at least by the 11th century, they had fused into the letter W as we know it. Note that this was long before the split of V and U into two separate letters, hence some modern disagreement over their offspring’s name. In the English alphabet, it’s called double U. For the French, it’s double vé.

From its origins in Francia, W was exported to nearby lands that also needed it. W appears in early English texts, although not without competition. One alternative, seen in Cædmon’s Hymn in Chapter U, was a single U. Scribes would switch to one U when the following vowel was an /u/. This would avoid awkward-looking sequences of three Us in a row.

This dislike of ‘triple U’ in medieval texts is in fact still active in English spelling today. In the later Middle Ages, scribes would swap a U for an O if it came after W. This was done for the sake of clarity when reading. Even when words had a short /u/ vowel, spellings like wulf, wud and wunder would have been too confusing in the era of manuscript writing, what with its rows of upright quill strokes. This avoidance tactic can explain the modern mismatch between sounds and spelling in wolf, wood and wonder. Nonetheless, W established itself as a standard way to spell the Germanic sound /w/, including in Latin texts produced in England.

For example, the Life of Saint Æthelwold is a tenth-century biography that narrates the holy life of an English bishop. Being based in southern England, its Latin language is crammed full of English place-names containing the sound /w/, like Winchester, Worcester and Wallingford. The saint’s own name is spelled Aðeluuoldus.

Yet, during the same pre-Norman period, a specifically English written culture was also emerging alongside Latin. Its writers clearly had a sense that this was a separate language from Latin, and therefore could have its own spelling practices. While writing in Latin ought to use only Latin letters, they felt that they had more freedom when spelling Old English. Just as they had done with the letter Þ, English writers looked for an alternative to a lengthy W or an ambiguous single U. They reached into the world of runes, and employed Ƿ.

Known as wynn, the letter Ƿ is extremely common in our Old English sources. See on p.301 how it appears twice in the first line in our only surviving copy of the poem Beowulf, in the words hƿæt ‘what’ and ƿe ‘we’.

It was standard spelling in the Wessex tradition, which would have written two and word as tƿa and ƿord. Examples of Ƿ outside the parchment pages of manuscripts show that the letter enjoyed popular use for centuries. A decorated dagger, found in Kent and dated to the ninth or tenth century, informs its viewers:

Biorhtelm me ƿorte
‘Biorhtelm wrought me’

S[i]gebereht me ah
‘S[i]gebereht owns me’

Yet, as you might have noticed, wynn is no longer a part of the English alphabet. It did survive the Norman Conquest, but gradually fizzled out during the Middle English era. It faced considerable opposition from the spelling of French and Latin, which had continued to use W since the sixth century. The pressure to match them meant that it was by W that Ƿ was eventually replaced.

Ever since the reapplication of W to English, the language has put the letter to a great many uses. Some instances of W are more recent in origin. Some even developed out of an original G.

In Old English, the letter G stood for one of a couple of similar sounds, depending on where in the word it came. In the middle of a word, a G represented a velar and fricative sound that was like a weaker /g/. During the Middle English period, this sound shifted into /w/, which also has a velar quality as a sound. This is how an Old English word like fugol ‘bird’ has become fowl, or how the sagu tool is now a saw. The Norse concept of lǫg, the facts of life laid down by fate or society, is behind English law.

These changes of G to W reflect a changed consonant, but elsewhere in spelling, W is used to tell us something about a vowel. It is especially common in words that have undergone the Great Vowel Shift, like town, cow and owl. These go back to tun, cu and ule in Old English, none of which had a G. Instead, W has been brought in to tell the reader that the OW in town is a greatly shifted diphthong, no longer a single long vowel as it had once been.

OW shares this role with OU. The second option for the same vowel appears instead in words like hour, shout and found. There has been a half-hearted rule in English spelling to use OW at the end of a word or syllable, and OU everywhere else. This rule would explain why we don’t write ‘nou’, ‘eyebrou’ and ‘allou’, but rather now, eyebrow and allow. At the end of a word like now, there is an audible /w/ sound, especially if the next word begins with a vowel (e.g. now I think …). However, this rule hasn’t been rigorously applied; we ought to write ‘broun’ and ‘croun’, not brown and crown.

Both OU and OW had good reasons to become the standard spelling for this post-shift vowel, but English failed to make a firm decision in favour of one or the other. It has even exploited the optionality to distinguish different words with a common origin. We spell flower with OW, while we use OU for the best quality or the ‘flower’ of ground grain—that is, flour.

Before we can leave W, there’s a mischievous effect of the letter to be acknowledged.

Consider three words: as, has and was. The third word, I think you will agree, does not rhyme with the previous two, despite their common spelling. Likewise, consider: and, hand and wand. The same lack of rhyme occurs, as it does in the trio arm, harm and warm. Notice the odd one out in ash, bash, cash, dash, gash and wash. If we also compare fan with swan, far with war, or fat with what, then their common denominator becomes clear: there’s something disruptive about the letter W.

To understand this effect, we have to concentrate on a particular quality that sounds in our languages can have. Vowels have featured often in this book, especially with regard to how far forwards, backwards, high or low our tongue is when we pronounce them. These features of tongue position are accompanied by the additional factor of lip rounding—whether or not we purse our lips at the same time.

The key thing to note here is that the consonant /w/ is pronounced with the lips and the back of the tongue. In the case of was, wand, wash and the rest, what has happened is that the /w/ rounded the following vowel, and also dragged it backwards in the mouth. The consonant has shared its rounded lips with the formerly unrounded vowel that comes immediately after it.

Consequently, in many varieties of English today, was, wand and ward have rounded vowels, while unrounded vowels can still be heard in their W-less counterparts, has, hand and hard. The cot-caught merger in North American English (see Chapter O) may be shifting and unrounding the particular vowel in the W-words, but nonetheless, hand still doesn’t rhyme with wand. The fact that this is an effect of adjacent sounds explains why the same changes and divergent vowels have also occurred in quality and quartz. They are not spelled with a W, but they still contain the influential consonant. Quartz doesn’t rhyme with parts, but rather shorts.

We still spell wash and warm as if they rhyme with ash and arm, because until fairly recently, they did. Their rounding is quite modern. It may have started sometime in the 15th century, but for the following four centuries, it remained limited to certain words and contexts. The first instances of W-rounding were likely in very common and unstressed words, like was. When said frequently and quickly, it’s more efficient to progress from a rounded-lipped consonant to a rounded vowel, than to switch off that rounding between the two. The effect was probably not present in the English of Chaucer, nor standard in the later speech of Shakespeare, on the basis of the words that these poets think are rhymes. In his sonnets, Shakespeare pairs was with glass, and warmed with disarmed.

Then were not summer’s distillation left,
A liquid prisoner pent in walls of glass,
Beauty’s effect with beauty were bereft,
Nor it, nor no remembrance what it was.

–Shakespeare, Sonnet 5

The fairest votary took up that fire
Which many legions of true hearts had warmed;
And so the general of hot desire
Was, sleeping, by a virgin hand disarmed.

–Shakespeare, Sonnet 154

Even Lord Byron, composing his narrative poem Childe Harold’s Pilgrimage in the early 19th century, rhymes three words that together sound awkward today.

I stood in Venice, on the Bridge of Sighs,
A palace and a prison on each hand:
I saw from out the wave her structures rise
As from the stroke of the enchanter’s wand:
A thousand years their cloudy wings expand …

–Lord Byron, Childe Harold’s Pilgrimage, Canto IV

Yet again, we have an instance of a reasonable change in sounds, and spellings that have not caught up. We could of course start to write wond instead of wand, or wor instead of war, or even woz for was. Maybe we will one day. For the moment at least, English readers and writers know to be cautious around the English letter W.” (297–306)

This article has been adapted from Why Q Needs U (Blink/Bonnier, U.S. June 2, 2026) by Danny Bate. It is provided courtesy of the publisher.

Unlimited UTF-8! ASCII ⊆ UTF-8 ⊆ UTF-8000.

No special cases introduced. All properties preserved.

Try out the reference implementation with $ pipx install UTF-8000.

UTF-8000 is in no way endorsed by or representative of the Unicode Consortium.
This is a fun standalone project / proposal.

TLDR / Examples

ASCII 1 0xxxxxxx UTF-8 2 110xxxxx 10xxxxxx 3 1110xxxx 10xxxxxx 10xxxxxx 4 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx UTF-8000 5 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 6 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 7 11111110 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 8 11111111 100xxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 9 11111111 1010xxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10 11111111 10110xxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx ... 22 11111111 10111111 10111111 10110xxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx ... 10xxxxxx ...

There is nothing special-case-y about the example 22-byte code unit here. It is just a good prototypical example, demonstrating the power of UTF-8000 with multiple start bytes.

There are only two special cases, both of which are inherited from UTF-8: ASCII as is, and 2-byte UTF-8 having 4 mandatory content bits to check against overlong encoding as opposed to 5 for all longer length code units.

Anatomy

Here is anatomical diagram of the example 22-byte code unit from the tldr.

See the glossary for more information on the definitions of the terms.

Byte number four is exciting! It is a continuation byte, a start byte, the final start byte, has content bits, and has only some of the mandatory content bits, which are straddled across the final start byte and first non-start byte.

The main contribution of UTF-8000's specification is clarity on splitting the highest bits of the first byte of UTF-8 code units into self-synchronization bits and start bits, and then making it clear how to stripe the start bits across the continuation bytes if needed, to achieve arbitrarily large code units.

Glossary

These terms are ordered somewhat by chronology of first requirement, rather than alphabetically, for convenience.

Codepoint

A non-negative integer, aka an unsigned integer.

Code Unit

A sequence of UTF-8000 bytes that encode a single codepoint.

First Byte

The first, one and only, byte that begins a UTF-8000 code unit.

The self-synchronization prefix of a first byte is either 0 for ASCII or 11 for multi-byte code units.

This term is not synonymous with start byte. A first byte is necessarily a start byte, but not the other way around. It is for this reason that first byte is sometimes also known as first start byte.

Fun observation: because of the self-synchronization prefix 0 the upper hex nibble of ASCII bytes can only be one of 0, 1, 2, 3, 4, 5, 6, 7.

This term is mutually exclusive with continuation byte due to self-synchronization.

Continuation Byte

A byte beyond the first byte of a multi-byte UTF-8000 code unit.

The self-synchronization prefix of a continuation byte is 10, which is also known as the continuation prefix bits.

Fun observation: because of the self-synchronization prefix 10 the upper hex nibble of continuation bytes can only be one of 8, 9, A, B.

This term is mutually exclusive with first byte due to self-synchronization.

Self-Synchronization Prefix

The highest bits of every UTF-8000 byte that indicate whether it is a first byte or a continuation byte.

The possible self-synchronization prefixes form a prefix-free tree:

  .----0              First byte for ASCII
  `----1---0 Continuation byte for multi-byte UTF-8000
        `----1        First byte for multi-byte UTF-8000

This piece of the clever architecture of UTF-8, which UTF-8000 inherits, provides the property of self-synchronization at a byte level: we can instantaneously tell what kind of byte we are looking at, and where it should belong in a code unit, just by looking at these highest bits.

This is most useful when decoding part of a file encoded in UTF-8000. If we randomly seek through the file to an arbitrary byte, we can unambiguously tell whether we are at a first byte whence we can begin decoding a new code unit immediately, or that we are at a continuation byte whence we need to seek a little further on in order to find the next first byte in order to begin decoding. Nor do we have to process any bytes prior to our seek position in order to discover some global state or the context of the byte we have seek-ed to; a first byte is always unambiguously a first byte wherever it appears, which we can deduce by its self-synchronization prefix being either 0 or 11.

This is useful not only for random access, but also for error recovery. Suppose that we are decoding an error-prone stream of UTF-8000 bytes and that whenever when we encounter an error (e.g. a rogue 0xC0 byte) we wish to keep calm and carry on instead of immediately exiting. We can yield Unicode replacement characters U+FFFD � and then await the next first byte, discarding anything in the interim.

See the Wikipedia article for self-synchronizing code for more general info.

These bits are highlighted in bright cyan.

Start Byte

A byte containing one or more start bits. The start bytes exist contiguously at the beginning of a UTF-8000 code unit. The power of UTF-8000 is that we can have multiple start bytes, to achieve arbitrary code unit lengths, to encode arbitrarily large codepoints.

Sometimes it is sensible to colloquially also include ASCII as a start byte when we are talking about the bytes towards the start of a code unit, even though ASCII bytes have no start bits.

Every non-ASCII code unit has at least one start byte. The first start byte is the first byte, and it is followed by zero or more continuation bytes that are also start bytes. Therefore because a UTF-8000 code unit can have multiple start bytes, this term is not synonymous with first byte.

In restricting to only UTF-8 without UTF-8000, this term is synonymous with first byte. This is because UTF-8-length code units only require one start byte, whether using up to 4 bytes in the current UTF-8 standard (RFC 3629 (2003)), or using up to 6 bytes in former standards (RFC 2044 (1996) and RFC 2279 (1998).

Start Bits

The unary-code sequence of bits contained in the start bytes of a multi-byte UTF-8000 code unit that tells us the length of the code unit in bytes.

For a code unit made of n bytes the start bits are n-2 1 bits followed by a terminating 0 bit. To be clear, the start bits include this terminating zero bit. Thus the start bits sequence is of length n-1 and looks like 111...10.

The possible start bits sequences form a prefix-free tree:

  .----0                    Two byte UTF-8
  `----1---0            Three byte UTF-8
        `----1---0       Four byte UTF-8
              `----1---0 Five byte UTF-8000
                    `----...    n byte UTF-8000

For an n byte code unit where n < 8 the start bits all fit together snugly in the first byte. Otherwise they are striped across as many of the first few bytes as they need, filling the free bits that are not occupied by continuation prefix bits.

This is another piece of the clever architecture of UTF-8, which UTF-8000 inherits, that provides the property of self-punctuation also known as a prefix code or a prefix-free code: when decoding a multi-byte code unit, once we have read to the end of the start bytes, that is we have encountered the terminating 0 bit, we know exactly how many bytes we expect in that code unit. Notwithstanding errors we can therefore succeed in decoding the code unit by reading exactly that many bytes, and no more.

This avoids a problem of dumber variable-length encodings whose code units do not intrinsically indicate their length: one has to read beyond the last byte of a code unit, that is one reads the first byte of the next code unit, in order to know that the current code unit has finished. For very dumb encodings which have neither self-synchronization nor self-punctuation, to make random access possible one would have to put dedicated auxiliary bytes, punctuation like a comma byte, between code units to be able to tell where one ends and another begins.

See the Wikipedia articles for prefix code and unary coding for more general info.

This term is mutually exclusive with content bits.

These bits are highlighted in bright magenta.

Content Byte

A byte containing one or more content bits.

A byte being a content byte does not imply that it is a continuation byte. For example a 3-byte code unit begins with 1110xxxx, which contains 4 content bits and is not a continuation byte.

A byte being a continuation byte does not imply that it is a content byte. For example a 22-byte code unit contains 10111111 as its second byte, which is a continuation byte and has no content bits.

Content Bits

The sequence of bits in a code unit beyond the start bits and to the end of the code unit, in which the codepoint's binary bits are stored. For example a 3-byte code unit, which has the form 1110xxxx 10xxxxxx 10xxxxxx, has 16 content bits.

For ASCII there are 7 content bits. These seven bits xxxxxxx combined with a byte's highest bit being set to the self-synchronization prefix 0 means that ASCII is perfectly included into UTF-8 without being altered. Thus ASCII code units take the form 0xxxxxxx.

Otherwise for an n byte code unit, where n > 1, there are 5n+1 content bits. This is how we arrive at that formula: We start with n blank bytes, each of which has 8 bits. For each byte 2 bits are taken by the self-synchronization prefix. Then an additional n-1 bits are taken by the start bits. Thus there are 8n - 2n - (n-1) = 5n+1 bits left for content bits. Another way to think about the 5 in this formula is by extending from n-1 bytes to n bytes by appending another continuation byte. By doing this we gain 6 free bits in the continuation byte, but we lose 1 bit to the longer start bits sequence, thus overall we gain 6-1 = 5 bits for content bits.

This term is mutually exclusive with start bits.

These bits are highlighted in lime.

Mandatory Content Byte

A byte containing one or more mandatory content bits.

These are the bytes we check for overlong encoding when decoding a code unit.

Mandatory Content Bits

The first 0, 4, or 5 content bits of a code unit in which there must be at least one 1 bit, lest the bytes form an overlong encoding, which is forbidden.

For ASCII there are 0 mandatory content bits, and thus no anti-overlong checking is required. This is because ASCII is the smallest possible code unit.

For 2-byte UTF-8000 there are 4 mandatory content bits. This is because in the jump from 1-byte ASCII to 2-byte UTF-8 we jump from 7 content bits to 11 content bits. Thus the number of content bits we gain is 11 minus 7 which is 4.

Otherwise for n byte UTF-8000, where n > 2, there are 5 mandatory content bits. This is because in the jump from n-1 byte UTF-8000 to n byte UTF-8000 we add on an extra continuation byte, which has 6 free bits, but we lose 1 bit to the longer start bits sequence. Thus overall the number of content bits we gain is 6 minus 1 which is 5.

Read about overlong encoding for why mandatory content bits are of interest.

These bits are highlighted in bright lime.

Overlong Encoding

Forbidden encodings of codepoints that could be encoded correctly in UTF-8000 using a shorter code unit.

For example one could incorrectly try to encode the codepoint 0x41, 65, ASCII capital A, using 2-byte UTF-8 as 11000001 10000001. Observe that all the mandatory content bits are 0 which is the definition an overlong encoding. This indicates that we could have encoded 0x41 in a shorter code unit, in this case as ASCII 01000001.

Security is one main reason why we forbid overlong encoding. For example we ensure that 11100000 10000000 10000000 cannot be decoded as codepoint 0, the null byte, lest one speciously pass such an overlong byte (code unit) to C functions like strcpy(3) and friends. strcpy would not interpret this code unit as a null byte, leading to a segfault at best, and serious vulnerabilities at least-worst.

Uniqueness of encoding is another reason why we forbid overlong encoding. Every codepoint has one unique valid representation as a UTF-8000 code unit, which is easy to encode and decode using bitshifting.

Fun observation: because all 4 of 2-byte UTF-8's mandatory content bits lie in the first-and-final start byte, we can explicitly rule out 11000000 (0xC0) and 11000001 (0xC1) as permanently invalid bytes. They will never ever appear anywhere in a valid UTF-8000 code unit!

Properties

Many of these properties of UTF-8000 are explained in detail in an appropriate section of the glossary and hyperlinks to the glossary are provided.

Bit Counts

The number of content bits and mandatory content bits are very predictable as a function of n, the length of a code unit.

code unit length	number of content bits	number of mandatory content bits
n = 1	7	0
n = 2	5n+1 ( = 11)	4
n > 2	5n+1	5

Why the Special Cases?

As stated in the tldr, there are only two special cases, both of which are inherited from UTF-8:

1-byte UTF-8 (ASCII) which has two points of interest:

• It has 7 content bits which does not fit the pattern of 5n+1. See the glossary section for content bits for an explanation, and see the rejected alternative ASCVI code for a version of UTF-8 if ASCII were 6 bit instead of 7 bit which eliminates this special case.
• ASCII has 0 mandatory content bits because it cannot possibly be overlong since it is the smallest possible code unit. This is fine.

2-byte UTF-8 which has one point of interest:

• It has 4 mandatory content bits, as opposed to 5 for all longer code units. See the glossary section for mandatory content bits for an explanation.

The remarkable fact that UTF-8000 does not introduce any new special cases in extending UTF-8 is confirmation to me that this is the canonical, correct way to extend UTF-8. In other words UTF-8 in its current restricted 4 byte form is UTF-8000, but only a small part of it.

The fact that we are even able to extend in the first place is also testament to the clever planning and care that Ken Thompson and Rob Pike put into the architecture of UTF-8, which we ensure to maintain as we extend to UTF-8000. Unary code codewords for the start bits sequences, which form a self-similar tree, were a great choice being simple and extensible. In the earliest draft of UTF-8, the six-byte start-byte looked like 111111xx. This was changed a few days later to 1111110x. That way the number of content bits is not a special case, and the start bits don't saturate the unary code binary tree, leaving the door open for our future expansion.

This is why I think of UTF-8 as the capstone of the Unix Philosophy.

Information Rate

What proportion of a code unit is content bits?

For ASCII this is 7/8 = 87.5%.

Otherwise for an n byte code unit this is (5n+1) / 8n, that is 5n+1 content bits out of a total of 8n bits from n bytes. We can rewrite this as (5/8) + 1/(8n) which moderately quickly approaches 5/8 = 62.5%. It is nice that this limit is nonzero and does not depend on n.

Self-Synchronization

Inherited from UTF-8 and maintained in UTF-8000.

See the glossary section for self-synchronization prefix for an explanation of self-synchronization.

Here's a bit of history: Self-synchronization is one of the reasons why Ken Thompson and Rob Pike decided to design UTF-8, to supersede the earlier FSS-UTF draft by Dave Prosser et al. FSS-UTF proposed a design like eg 110xxxxx 1xxxxxxx 1xxxxxxx for three-byte code units. The problem with it is that one cannot distinguish between first bytes (110xxxxx) and continuation bytes (110xxxxx) without knowing the prior history of a stream. The UTF-8 fix is to make first byte and continuation byte values disjoint from each other, as one can witness in the byte map below. I have not put Prosser's draft into the rejected ideas section as it has already been formally addressed and superseded by UTF-8.

Self-Punctuation

Inherited from UTF-8 and maintained in UTF-8000.

See the glossary section for start bits for an explanation of self-punctuation.

Byte Map

Extended from UTF-8, making use of the higher value bytes. Based off Wikipedia's UTF-8 Byte Map.

All bytes except 0xC0 and 0xC1, colored in tomato red, can appear in a valid UTF-8000 stream. See the glossary section for overlong encoding for an explanation of why those two bytes never appear.

ASCII, colored in gold yellow, occupies the first half of the table, being 7-bit. Continuation bytes occupy the region colored in sandybrown orange. All other bytes are first bytes for multi-byte code units, whose lengths are indicated in the table.

strcmp(3) Ordering

Inherited from UTF-8 and maintained in UTF-8000.

The self-synchronization prefixes of first bytes are monotonically-increasing-ly ordered with respect to code unit length. In other words ASCII is of length 1 and multi-byte is of length greater than 1, and 0 < 11 occupying the highest bits of UTF-8000 bytes.

The start bit sequences are also monotonically-increasing-ly ordered with respect to code unit length. In other words if n < m then 111...[n]...10 < 111...[m]...10 as an integer value, occupying the heads of the code unit bytes beyond the self-synchronization prefixes. This would not have been the case had UTF-8 been designed to use the alternative form of unary codewords given by 000...01.

The content bits of code units are also monotonically-increasing-ly ordered with respect to codepoint value.

Combining these three things together means that strcmp(3), the C stdlib string comparing function, works the same way on UTF-8000 bytes as it does on UTF-8, as it does on ASCII, effectively comparing the encoded codepoint values against each other without having to actually decode the code units. Nice!

No Endianness

The quantum of ASCII, UTF-8, and UTF-8000 is a single byte. This makes life a breeze! There is no need for a concept of endianness for UTF-8000.

UTF-16 however has a quantum of two bytes, 16-bit units. When writing the codewords in bytes, 8-bit units, should the byte containing the most significant digits or least significant digits be written first? Big-endian, or little-endian? This choice gives UTF-16 two variants, UTF-16-BE and UTF-16-LE. If one cannot predetermine the endianness of a stream, one may wish to use a BOM which is discussed below.

BOM Support

A Byte Order Mark (BOM) is used at the start of an encoded text stream to indicate what encoding is used. I have never actively used BOMs myself so I've only put a bit of thought into this section.

As far as I'm aware we don't break BOM support for UTF-8, though we may wish to have a different BOM to strictly distinguish UTF-8 from UTF-8000. Maybe UTF-8000 could have multiple BOMs, one for each integer N greater than or equal to four, to indicate to a decoder the maximum expected code unit length.

One of the reasons why U+FFFE is not a valid Unicode Scalar Value is because 0xFE 0xFF is the BOM for UTF-16. Since UTF-16 code units are two bytes wide, one may read either 0xFE 0xFF or 0xFF 0xFE depending on endianness. To make it clear that 0xFF 0xFE implies correct for endianness and cannot be mistaken for a legitimate character, U+FFFE is designated as <noncharacter-FFFE>. We are relieved in that neither 11111111 11111110 nor 11111110 11111111 are valid UTF-8000 sequence extracts, ie UTF-8000 does not introduce incompatibilities with UTF-16.

Arbitrary Lengths, Sensible Limits

I think we've made it clear by now that UTF-8000 code units can be arbitrarily large. In practice however one may wish to set a sensible limit on code unit lengths when decoding. Here we'll discuss a method of finding some nice code unit lengths whose code units store 5n+1 = 2^N bits, as we are often interested in powers of 2 in computer science.

It is a common observation that 3-byte UTF-8 stores 5 * 3 + 1 = 16 bits, meaning the Basic Multilingual Plane of Unicode can be encoded in one two and three byte UTF-8. We see that 2^4 mod5 = 16 mod5 = 1 mod5; if 5n+1 is to be 2^N for some n then certainly 2^N = 1 mod5. If we enumerate powers of two modulo five then there is a very predictable repeating pattern of 1, 2, 4, 3. Formally you might say that 2 is a generator of 𝔽5* if you want impress a mathematician! The takeaway is that when N=4K for K≥1 we can find a corresponding n such that 5n+1 = 2^N. We can rewrite 2^N as 2^(4K) = 16^K.

In other words any power of 16 has a UTF-8000 code unit length containing that many bits. Here are a few of these for reference.

K	N=4K	number of content bits = 2^N	code unit length = (2^N - 1) / 5
1	4	16	3
2	8	256	51
3	12	4096	819
4	16	65536	13107
...	...	...	...

Do remember that strictly speaking one shouldn't allow overlong encodings, if one were for example thinking of storing a small uint256_t key in a 51 byte code unit! UTF-8000's variable width nature helps out leading to smaller code units for smaller integers.

Intuitive Derivation

There are a few ways that one could arrive at the design for UTF-8000 and the bit counts above.

One may think to start with UTF-8, notice that the start byte of an n byte code unit is prefixed with the unary codeword of length n+1, that is n 1 bits followed by a 0, and then figure out how to extend those bits and roll them over into the continuation bytes without losing any important properties. This is what I originally did.

Writing this document over a couple of weeks made me introspect the code unit anatomy further, whence I figured out that separating the leading bits into a self-synchronization part and self-punctuation part further illuminates and simplifies the thought process. We shall thus proceed with this perspective.

Blank Slate

We set out to derive the design of an n byte code unit, starting out with n blank bytes, all of whose bits could possibly be content bits.

00000000 00000000 00000000 ... 00000000

To achieve self-synchronization we need to distinguish the first byte of the code unit from the continuation bytes that follow. We could do that by setting the highest bit of first bytes to a 0 and to 1 for continuation bytes. Doing it this way round maintains compatibility with ASCII's highest bit being 0.

00000000 10000000 10000000 ... 10000000

With the design so far, all code units begin with an ASCII byte. When decoding a code unit, we have no idea whether this first byte actually is ASCII, or it is the first byte of a multi-byte code unit. We want self-punctuation, where a code unit intrinsically tells us how long it is.

To achieve self-punctuation we create a prefix-free code binary tree, whose leaf node codewords correspond to code unit lengths. These are the start bits sequences. The codeword for n shall be embedded inside the code unit towards the start. It must therefore be short enough to fit into the n bytes, and reasonably computationally predictable. We try:

  .----0                          One byte UTF-8 (ASCII)
  `----1---0                    Two byte UTF-8
        `----1---0            Three byte UTF-8
              `----1---0       Four byte UTF-8
                    `----1---0 Five byte UTF-8000
                          `----...    n byte UTF-8000

This seems reasonably simple so far. We stripe the start bits into the available bits not taken by the self-synchronization prefix. All other bits shall be content bits.

1 00xxxxxx 2 010xxxxx 1xxxxxxx 3 0110xxxx 1xxxxxxx 1xxxxxxx ... 17 01111111 11111111 1110xxxx 1xxxxxxx ... 1xxxxxxx ...

But wait we've broken the distinction of ASCII! We cannot tell the difference between eg 0110xxxx and 0110xxxx, or 01111111 and 01111111. This code would only work if ASCII were six-bit instead of seven-bit. Out of curiosity we investigate this code in the rejected alternatives section ASCVI.

To maintain compatibility with ASCII we must treat it as a special case, whereby the self-synchronization prefix 0 is alone sufficient to characterize ASCII. This highlights that the architecting of UTF-8 was not purely a mathematics problem, but was also an engineering problem, working around what already exists.

Seeing the ASCII-characterizing prefix 0 and the erstwhile continuation prefix 1 as forming a prefix-free tree, albeit only of size two, we must repurpose the the latter codeword as the beginning of the self-synchronization prefixes for first bytes and continuation bytes of multi-byte code units. We choose our new self-synchronization prefixes as 11 for start bytes and 10 for continuation bytes. This produces the following tree:

  .----0              First byte for ASCII
  `----1---0 Continuation byte for multi-byte UTF-8000
        `----1        First byte for multi-byte UTF-8000

Accordingly adjusting the self-punctuation codewords to apply only to multi-byte code units produces the following tree:

  .----0                    Two byte UTF-8
  `----1---0            Three byte UTF-8
        `----1---0       Four byte UTF-8
              `----1---0 Five byte UTF-8000
                    `----...    n byte UTF-8000

Putting these mechanisms together yields UTF-8000 and we're done!

ASCII 1 0xxxxxxx UTF-8 2 110xxxxx 10xxxxxx 3 1110xxxx 10xxxxxx 10xxxxxx 4 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx UTF-8000 5 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 6 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 7 11111110 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 8 11111111 100xxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 9 11111111 1010xxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10 11111111 10110xxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx ... 22 11111111 10111111 10111111 10110xxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx ... 10xxxxxx ...

It is a trivial* result of coding theory that the product of two prefix-free codes is also a prefix-free code. The product of our trees looks like:

  .----0                                                  ASCII byte
  `----1---0                               UTF-8 continuation byte
        `----1---0                    Two byte UTF-8    start byte
              `----1---0            Three byte UTF-8    start byte
                    `----1---0       Four byte UTF-8    start byte
                          `----1---0 Five byte UTF-8000 start byte
                                `----...    n byte UTF-8000 start byte

Without the color highlighting this is how most people think about UTF-8: a start byte whose prefix of n 1 bits and a terminating 0 bit provides both self-synchronization and self-punctuation, and continuation bytes using a short prefix of 10 for space-efficient encoding. This makes sense for a small number of bytes, but the trick to unlock a perspective of infinite extensibility is to split this tree into the self-synchronization part and self-punctuation part; ie we un-product those prefix-free codes. Failing to do this leads to the rejected alternative UTF-Infinity.

Encoding

This section is based off the reference implementation which is written in Python. It is well documented, and is more specific on how to use bitwise operations. This is an abridged HTML version.

Suppose that we have an unsigned integer n that we want to encode in UTF-8000. Initialize an empty dynamic array of bytes ret_ints that will store the UTF-8000 code unit.

If n < 0x80, eg n = 0x41, then insert n at the head of ret_ints and we are done. This is the ASCII byte for n, which in our example of n = 0x41 is a capital letter a, 'A'.

Otherwise for n ≥ 0x80, eg n = 0x0321C0FFEE8086, we use UTF-8000. Initialize an integer counter n_bits_content_occupied to zero.

Our example n's content bits look like 11 001000 011100 000011 111111 111011 101000 000010 000110 as a big raw number, with spaces added for visual ease.

While n has more than 6 content bits, aka n > 63 = 00111111, extract the least-significant 6 bits of n and insert them at the head of ret_ints, incrementing n_bits_content_occupied by 6 and downwards bitshifting n by 6.

n_bits_content_occupied = 48, n = 0b11,

ret_ints: 00001000 00011100 00000011 00111111 00111011 00101000 00000010 00000110

Now insert the rest of n at the head of ret_ints. Count the number of bits left in n by downwards bitshifting n one bit at a time while it is non-zero. This is between 1 and 6 (inclusive), which we also add to n_bits_content_occupied.

n_bits_content_occupied = 50, n = 0,

ret_ints: 00000011 00001000 00011100 00000011 00111111 00111011 00101000 00000010 00000110

Now we calculate how many bytes our UTF-8000 code unit requires, n_utf_8000_bytes_needed. We know that a k-byte code unit has capacity for 5k+1 content bits. Therefore ⌈(n_bits_content_occupied-1) / 5⌉ is the sufficient and minimal answer. Any larger code unit size would lead to an overlong encoding! For our example n_utf_8000_bytes_needed = ⌈(50-1) / 5⌉ = 10.

Leftwards pad ret_ints with empty bytes to the length n_utf_8000_bytes_needed.

ret_ints: 00000000 00000011 00001000 00011100 00000011 00111111 00111011 00101000 00000010 00000110

Now we add the start bits. The number of 1 start bits is equal to two less than the number of bytes in the code unit, which we just calculated. We therefore calculate q, r = divmod(n_utf_8000_bytes_needed-2, 6), which tells us we need q hextets full of 1 start bits, and a final hextet of zero to five 1 bits, which also has space to contain the terminating 0 bit. In our example (q = 1, r = 2) = divmod(10-2, 6).

Apply the start bits across ret_ints using bitwise-or. The final start bits hextet can be given by ((1 << r) - 1) << (6 - r).

ret_ints: 00111111 00110011 00001000 00011100 00000011 00111111 00111011 00101000 00000010 00000110

Any of the lowest six bits of each byte that are not set by this point, unoccupied by content bits and untouched by start bits, are really content bits that the k-byte capacity provides but that we didn't need. Our example's n_bits_content_occupied = 50 is one less than 5k+1 = 5*10+1 = 51. We can color highlight it green as a content bit for completion's sake.

ret_ints: 00111111 00110011 00001000 00011100 00000011 00111111 00111011 00101000 00000010 00000110

Now we crown the bytes with their self-synchronization prefixes, which delivers us from hextets to UTF-8000 octets. The first byte's self-synchronization prefix is 11, and continuation bytes have 10.

ret_ints: 11111111 10110011 10001000 10011100 10000011 10111111 10111011 10101000 10000010 10000110

And we're done!

Decoding

As with the encoding section, this section is based off the reference implementation which is written in Python and well documented.

Suppose that we are receiving a stream of UTF-8000 bytes (possibly with errors!), and we wish to extract and taxonomically annotate the incoming code units. There are a few ways that we could approach this, such as using the byte map as a state machine, which I want to try in the future, or the classic way of using bitwise masks. We are going to use the latter approach in this section, as we describe how to decode a single code unit. But first, a look at error handling.

Error Recovery

The errors that can occur when decoding a UTF-8000 stream are:

1. Reading a continuation byte (10) when we are expecting the first byte of a code unit (0 or 11).

2. Reading a first byte (0 or 11) when we are expecting a continuation byte (10).

3. Early EOF midway through a code unit.

4. Encountering an overlong encoding

   1. For 2-byte code units this is bytes 0xC0 (11000000) and 0xC1 (11000001).
   2. For n-byte code units in general, with n > 2, eg 11100000 10010111 10010000.

5. Encountering an encoded surrogate codepoint in the range U+D800 to U+DFFF, which is forbidden for compatibility with UTF-16.

For standard UTF-8 one would also have to be concerned with codepoints beyond the range U+10FFFF whence bytes 0xF5 to 0xFF go unused.

For any of these errors a parser could raise an exception and refuse to continue. Alternatively it could take advantage of UTF-8000's self-synchronization property, and keep calm and carry on, yielding Unicode replacement characters U+FFFD � until we reach the first byte of the next code unit. Let us investigate the latter course.

To handle error 1. the parser should return one � and get ready to parse the next code unit. When handling error 2. the parser should make sure to unpop the byte encountered, as it is the first byte of the next code unit. When handling errors 2. through to 5. there are a couple of mainstream approaches for yielding � characters:

Maximal Subpart

The Unicode Consortium recommends, but does not enforce, a maximal subpart approach, in which the longest well-formed part of a code unit should return a single � character, rather than one for each byte involved. For example the three bytes in error 4.2. above should return one � as it is well-formed with respect to self-synchronization and self-punctuation, and only invalid at an overlong level, being an overlong encoding of 11010111 10010000 U+05D0, a Hebrew letter Aleph 'א'.

I dislike this approach. Waiting for maximal subparts has the problem that the rest of an invalid code unit may never arrive. If we receive the bytes 11100000 10010111 from a socket, then the remote end may be waiting for us to chastise their overlong opening bytes with a response, because we can already tell that these bytes form part of an invalid code unit. Using the maximal subpart approach we also would be waiting, for the remote end to send a continuation byte eg 10010000 to form an overlong but otherwise complete 3-byte code unit. This is uncooperative, and not what I want.

One � For Each Byte Read

We are going to do what Python, my terminal KDE Konsole, and others do, and simply return a � character for each invalid byte. In Python b'\xE0\x97\x90'.decode(errors='replace') returns '���'.

This approach is easier and more versatile. The end user can see how many invalid bytes occurred by counting the number of � characters. There are no deadlock waiting events that can occur with the maximal subpart approach.

The Main Decode Loop

Initialize an empty dynamic array of bytes parsed_bytes that will store the bytes as we parse them.

Read a byte, store it as start_byte. Use bitwise masks to find the index, idx_0, of the most-significant zero bit in the byte. If there are no zeros in this byte, idx_0 should be set to -1.

If idx_0 == 7 (0xxxxxxx) then start_byte is an ASCII byte, which has seven content bits. Append start_byte to parsed_bytes and we are done.

If idx_0 == 6 (10xxxxxx) then start_byte is a continuation byte, which is an invalid start byte. Go to error 1.

If idx_0 == 5 (110xxxxx) then this is the first byte of a 2-byte code unit. We treat this as a special case because there are only 4 mandatory content bits, not 5. As they are all contained in start_byte we can check them immediately for overlong encoding, to see if we need to handle error 4.1. If start_byte passes this check then append it to parsed_bytes and await a continuation byte. Handle error 2 if necessary, else append the continuation byte to parsed_bytes and we're done.

We could (should really) make idx_0 == 4 a special case too, to check for and forbid the surrogate ranges. I have omitted this for the time being and we drop through to the generic case below.

Otherwise we enter the generic case (111[1...]). Initialize an integer counter n_bytes_expected to 2. Increment n_bytes_expected by 5 - idx_0, as idx_0 now serves the purpose being the index of the terminating zero of the start bits, 0.

If idx_0 == -1 then our code unit has multiple start bytes, exciting! Append start_byte to parsed_bytes, and while(1):

Read a byte, and make sure it is a continuation byte lest we go to error 2. Use bitwise masks to find idx_0, the index of the most-significant zero bit in the lowest six bits of the byte, setting idx_0 to -1 if there is none. This is to continue trying to find the 0 start bit. Increment n_bytes_expected by 5 - idx_0. If idx_0 == -1 then append start_byte to parsed_bytes and continue again through this loop, until we find the 0 bit, at which point we break this loop.

At this stage, whether our code unit has multiple start bytes or just one, start_byte is the final start byte of the code unit, idx_0 is between 0 and 5 (inclusive), and we move towards checking for overlong encoding. Just as ordinals count the number of things less than themselves, idx_0 counts the number of content bits contained start_byte, occupying the least significant bits.

There are six cases for anti-overlong checking, which correspond to idx_0's value. That may sound like a lot, but the looping gif below that I made should relax you. It demonstrates periodic behavior. Even though it shows deep code unit sections with multiple start bytes, this animation still applies for all code units of length n > 2. The colored bars are based off the anatomy section image.

If idx_0 == 5 then all the mandatory content bits are contained together in the final start byte. Thus we should immediately check start_byte using the mask 00011111. We then read the first non-start byte, a continuation byte which does not need overlong checking (10xxxxxx).

Otherwise we read another continuation byte, the first non-start byte. If idx_0 == 0 then all the mandatory content bits are contained together in this first non-start byte (10xxxxxx), and we use the mask 00111110 to check for overlong encoding. Else idx_0 is between 1 and 4 (inclusive) and the mandatory content bits are straddled across the final start byte and first non-start byte. In these cases we use two masks to check for overlong encoding, which one can see in the gif above.

Perhaps the case of idx_0 == 0 could be grouped in with idx_0 being between 1 and 4, by using an empty mask to check the final start byte, in order to make the algorithm less branch-y, but this walkthrough isolates which bytes are responsible for potential overlong encoding.

Given that the final start byte and first non-start byte have passed the overlong check, append them to parsed_bytes. Finally while the length of parsed_bytes is less than n_bytes_expected, read plain-old continuation bytes (10xxxxxx) and append them to parsed_bytes.

And we're done!

For the first few years of my career, I didn’t know how to tell my computer to do things. I could kick off a few commands from the terminal – run those tests, install that dependency, start that container, ssh to that machine – but I was limited to running simple commands one at a time. My terminal was the world’s worst GUI, and I thought that the shell was the way to start programs that didn’t have application wrappers.

At a fundamental level, I wasn’t able to tell my computer to accomplish anything that involved logic or stitching together multiple programs like:

• Do one thing and then do another
• If a command fails, log out an error message
• Kick off a program when the computer starts
• Run two commands at the same time
• Loop through all of the files in a directory and take an action on each one
• Use the output of one command as the input for another

In theory, I could have used NodeJS to write those sorts of programs. In practice, I never did. This was partially mindset: I wasn’t used to thinking about the programs I used on the command line as things that I could control. And the rest of it was a lack of skill: I didn’t know the programs that I was using well enough to integrate them into a script that I’d written.

I kept trying to learn the shell though, and I slowly got to the point where I could muddle my way through scripts like this one that had basic logic:

set +e
npm install
status_code=$?
if [[ "$status_code" != "0" ]]; then
    echo "Something went wrong with your npm install. Check your ~/.npmrc to make sure it's authed to our registry."
    exit 1
fi
set -e

Depending on your familiarity with shell scripting, You might be gibbering right now. Sorry. (If you’re not bleeding from the eyes yet, here’s why you should be: 1)

Even though the commands and scripts I wrote had problems, it was transformational for me; I had the sudden ability to tell my computer to stitch together existing programs to accomplish my goals. It felt a little bit like the change that came from learning to program in the first place.

Over the course of years, I slowly started to learn more shell tools and figure out shell syntax. I’d often learn a new tool or pattern and then have a moment of pain when I realized how much easier a past problem would have been to solve if I hadn’t used a hammer to solve a problem that needed a drill.

Over that time, I’ve worked with a ton of (incredibly strong!) engineers who didn’t spend as much time learning the shell, and instead relied on GUIs to do things like run tests, manage git, talk to databases, and do day-to-day tasks. Relying on a GUI works well until you want to accomplish something that the GUI wasn’t set up to handle, and I think it’s easy to slip into a mindset where you’re limited to what the GUI is capable of. I’ve seen skilled engineers spend a ton of effort because they didn’t know how to do things like use while to keep running a command or write a for loop to do the same operation on every file in a directory.

I think that same GUI-focus can be a problem when it comes to editing and maintaining shell scripts. I’d wager most companies have a decent amount of essential logic to build, deploy, validate code, and test in languages like Bash or Zsh. If you’re not comfortable with the language your tooling is built in, then you won’t be able to easily read or improve it. You might be able to tell the remote servers that your code runs on how to behave but not be able to tell the computer that you work on how to do things like run linting and tests in parallel – that’s a bummer!

Let’s pause to take a quick detour: Why the heck are so many of these scripts end up written in languages that aren’t the main ones the team uses? I think it’s often more ergonomic to write a script that stitches together commands in a language that’s been designed to be easy to stitch together commands. Let’s take a super simple example of running a test until it fails: while pnpm exec mocha ./pathToFile.test.ts; do true; done. There are obvious things to critique about this syntax, but let’s take a look at what it looks like in NodeJS:

const { execSync } = require("child_process");
while (true) {
    try {
        execSync(`pnpm exec mocha ./pathToFile.test.ts`,  { stdio: "inherit" });
    } catch (err) {
        console.error("failed", err);
        break;
    }
}

There are a lot of rough edges and gotchas here, and I personally think the shell is easier! I don’t need to worry about creating a file, requiring child_process, or setting { stdio: "inherit" } to see output. And this is a pretty simple example that doesn’t even stitch together multiple programs with a pipe, capture any output, or use a temporary file!

This doesn’t mean that you need to resign yourself to writing in Bash or Zsh or any similar language! For teams that know JavaScript well, one tool I’ve enjoyed is zx. I think it can make these scripts pretty ergonomic to write and maintain. Ruby and Python are both easier than NodeJS is, but I think there are plenty of languages out there that require even more ceremony to write a quick little script like this.

I’m certainly not arguing that shell scripts will always be easier for build scripts! When problems are simple enough that you’re just stitching two programs together, a tool like Bash or Zsh feels pretty ergonomic. As soon as you want more sophisticated logic and data types, you’ll want to choose a language that makes it easy to represent (and test!) more sophisticated logic and data types.

I suspect that many engineers who gripe about build scripts being written in a shell language haven’t actually tried converting them to a different language. Aside from the syntax (potentially) being more complicated, a huge part of "learning to write shell scripts" isn’t actually syntactical. If you convert a script that stitches together commands but don’t actually know how the commands you’re stitching together behave, the resulting script is likely to be similarly impenetrable.

Knowing the shell – being able to tell a computer to do things – depends a ton on knowledge of the programs that do the things that you want to accomplish! I’d argue that knowing the shell is 20% syntax and 80% having a good toolbox:

• If you know fzf, you can build quick utilities with interactive fuzzy-searching. (Example: git checkout $(git branch --sort=-committerdate | fzf) will let you fuzzy-choose a branch.)
• If you know tldr or eg, you can pull up usage examples for any other command in this list
• If you know rsync, you can copy changed files on to a faster remote server to run something heavy and slow
• If you know xargs, you can build up commands incrementally and parallelize work
• If you know sed -i or ast-grep, you can quickly rewrite complicated patterns across a bunch of files
• If you know direnv, you can make sure the right environment variables are set for everyone who runs commands in a codebase.
• If you know duckdb, you can write SQL to query CSV and JSON files locally as part of a larger script.
• If you know gh, you can build scripts to check on your PRs and open up new PRs from the cli.
• If you know ngrok, you can quickly serve a local port to test something out on a different machine.

Each additional program you learn expands your capabilities more because each new tool can be used with every other tool you already know.

I can’t stress enough that I’m the furthest thing in the world from a shell scripting expert, and I’m terrible compared to people who know it deeply,2 but I’ve still gotten a lot of value out of knowing enough shell syntax to stitch together programs and knowing enough programs that I actually want to stick together.

Telling your computer to do things is great!

---

1. This script was written with good intentions, but it was overly complicated because I didn’t know that if's main mode is to take commands. [[ (or test) is just a special command. This means that our if check can just be ! npm install.

   • With this approach, there’s no need to set +e to allow commands to fail. (set +e and then setting it back with set -e is to avoid the shell-script as a whole failing because a command failed – Unofficial Bash Strict Mode)
   • It also means we don’t need to capture the exit code with $?

   if ! npm install; then
       echo "Something went wrong with your npm install. Check your ~/.npmrc to make sure it's authed to our registry."
       exit 1
   fi
   

   ↩︎

2. From an HN comment on a previous blog post: "This post […] means I no longer wonder why 99% of shell scripts I come across look inept." ↩︎

Origins of life

Demonstrate the emergence of life from chemical precursors in a laboratory setting.

Specifically, demonstrate the unassisted emergence of self replicating RNA- and protein-based cells from a plausible primordial soup with a plausible energy source. A “cell” may be any compartment with a defined boundary. To be considered successful, the following conditions must be met. Firstly, it must be shown that the emergent cells can increase their abundance by at least a factor of 10⁶ (roughly 20 generations), when provided with sufficient primordial soup and energy. Secondly, it must be plausible that division could continue indefinitely given sufficient energy and primordial soup. For example, solutions that involve the cells monotonically decreasing in size over successive divisions would not be accepted. Finally, the cells must have a clear way of encoding heritable genetic information, i.e., the molecular composition of the cells must be causally determined at least in part by information stored within the cell. Solutions that involve storing the information in the form of nucleic acids, polypeptides, or similar polymers are strongly preferred. Solutions in which the existence of heritable genetic information is ambiguous or controversial will be rejected by default.

Cryopreservation

Demonstrate the ability to cryopreserve and recover live wild-type mice with high viability.

Specifically, demonstrate the reversible cryopreservation of live, intact, wild-type adult mice in a whole-body frozen or vitrified state. The mice must remain frozen or vitrified for at least 24 hours, must be recovered with >99% viability, and must not suffer any permanent organ damage or bodily harm. Somatic genetic engineering is discouraged but permitted. All experiments must be conducted with ethics approval.

The reverse translatase

Create an enzyme that can “reverse translate” an arbitrary peptide sequence into RNA or DNA.

Specifically, create a purified protein catalyst or fixed protein complex that processively reads an untagged polypeptide and synthesizes a covalent nucleic acid strand encoding its residue sequence under a preregistered codon convention, without a nucleic-acid template, preattached sequence barcode, residue-specific operator cycle, or database lookup. The resulting nucleic acid strand must be compatible with ordinary polymerases, ligases, and other similar enzymes, i.e., if nucleic acids other than RNA or DNA are used, they must be compatible with downstream amplification or sequencing reactions. For the challenge to be considered complete, at least 100 random peptide sequences of at least 50 amino acids each must be preregistered, synthesized, and pooled. It must then be shown that the sequences of these peptides can be inferred, without reference to a dictionary, by reverse translation and sequencing with at least 90% sequence accuracy. Moreover, the average read length must be at least 25 residues, and the average read quality score should be at least Q10.

Improve Rubisco

Produce a Rubisco enzyme with specificity and enzymatic turnover beyond the naturally occurring pareto frontier.

Specifically, produce an enzyme that catalyzes the carboxylation of ribulose-1,5-bisphosphate with a specificity for carbon dioxide over oxygen (Sc/o) at least as high as that of Galdieria Partita Rubisco, and with an enzymatic turnover (kcat) at least as high as that of maize Rubisco. To be considered successful, the specificity and enzymatic turnovers of the candidate enzyme must be measured in paired enzyme assays using G. Partita Rubisco and maize Rubisco as controls, respectively. The candidate enzyme may be designed de novo, discovered in nature, or engineered or evolved from naturally occurring starting points.

The quadruplet cell

Produce a living cell that uses a four-base codon code.

Specifically, produce a living and replicating cell in which every protein-coding sequence, including the translation machinery itself, is encoded as uninterrupted nonoverlapping quadruplet codons, without detectable triplet decoding. The encoding scheme must be a bona fide quadruplet encoding, i.e., in the quadruplet encoding, the probability that a mutation is non-synonymous must be similar regardless of the index of the mutation in the codon. For example, quadruplet encodings in which the first three codon positions are always or almost always sufficient to specify the encoded amino acid will not be accepted.

Somatic limb regeneration

Demonstrate the ability to regenerate lost limbs in adult wild-type mice.

Specifically, demonstrate, in an adult wild type mouse, the reproducible ability to regrow limbs following amputation. Following regeneration, the mouse must perform indistinguishably from controls in a standard battery of motor function tests, must demonstrate indistinguishable sensory perception in the regrown limb, and blinded observers must not be capable of distinguishing which limb was regrown based on non-invasive observational data. All experiments must be conducted with ethics approval.

Bacterial production of gene therapies

Demonstrate the ability to produce gene therapies in a bacterial host.

Specifically, produce infectious replication-incompetent AAV and lentivirus in bacteria. The particles must contain a pre-specified viral genome; the ratio of physical capsids to viral genomes and the ratio of infectious units to viral genomes must be similar to the ratios obtained when purifying viruses from mammalian cell culture; and the viral genomes must be nuclease-resistant. It is anticipated that producing lentivirus in bacteria may be much more challenging than producing AAV, and thus demonstrating the ability to produce AAV on its own will be considered a partial success.

Programmable proteases

Demonstrate the ability to produce enzymes on demand that will specifically and efficiently cut a specific protein sequence.

Specifically, given a blinded, accessible site in an endogenous folded protein, demonstrate the ability to prospectively design a protease that cleaves that site efficiently in living cells. The resulting enzyme must have catalytic efficiency and proteome-wide off-target cleavage similar to or greater than other widely-used site-specific proteases. The challenge will be considered complete when the design can be demonstrated against 20 preregistered sites with a success rate greater than 80%. Once the target sites are preregistered, the designs of the resulting proteins must be produced within 24 hours, and no wet lab work is allowed prior to evaluation except for the purpose of producing the designed proteins for assay. (Hence, for example, screening and target-specific evolution are not permitted once the target sites are provided.)

Note that a weaker form of this challenge involves demonstrating the ability to produce enzymes that specifically and efficiently cleave specific preregistered peptide sequences, when those sequences are provided in solution, along with off-target sequences. Demonstration of that ability will be considered a partial success.

Cell-penetrating protein binders

Demonstrate the ability to produce protein binders against intracellular targets.

Specifically, demonstrate the ability to design zero-shot protein binders that, without further evolution or optimization, will reliably engage preregistered intracellular protein targets in living cells when administered extracellularly to those cells at pharmacologically supported concentrations. The cell entry mechanism must be plausible in a therapeutic context, i.e., transfection, intrabody expression, electroporation, membrane disruption, or similar methods are not permitted. The challenge will be considered complete when the design can be demonstrated against 20 preregistered targets with an 80% success rate. Once the targets are preregistered, the designs of the resulting proteins must be produced within 24 hours, and no wet lab work is allowed prior to evaluation except for the purpose of producing the designed proteins for assay. (Hence, for example, screening and target-specific evolution are not permitted once the target sites are provided.)

The original intention of this problem was specifically to design antibodies against intracellular targets. However, it is anticipated that modifications to the antibody scaffold will be required in order for the problem to be solvable. Since we cannot put an upper bound on the magnitude of the modifications required, we have broadened the problem to encompass any protein binders. However, solutions that involve binders resembling humanized monoclonal antibodies will be greatly preferred. The problem would likely be even more impactful if solved in general for small molecule binders, rather than protein binders or antibodies. However, with small molecule binders, synthesis is a major bottleneck that would limit validation, and thus we have chosen to restrict the scope to protein binders.

Protein amplification chain reaction

Demonstrate exponential amplification of arbitrary peptide substrates.

Specifically, demonstrate input-protein-dependent synthesis of new, full-length, sequence-faithful covalent polypeptide copies from amino-acid monomers without a nucleic-acid template or preformed cognate scaffold, in a single pot reaction. For the challenge to be considered complete, at least 100 random peptide sequences of at least 50 amino acids each must be preregistered, synthesized, and pooled. It must then be shown that the abundance of these peptides in solution can be amplified at least 1000x with at least 90% sequence accuracy on a per-residue basis. Reasonable modifications may be added to the peptide sequences to facilitate post-amplification analysis if necessary, provided they are not active in the amplification. Methods that rely on explicit sequencing of the peptide are not permitted. Methods that rely on reverse translation to generate a nucleic acid intermediate are not permitted, because they are duplicative with a separate Millennium Problem.

5′ polymerases

Produce a full set of polymerases that act in the 5′ direction.

Specifically, produce a complete set of 3′>5′ polymerases comparable to commonly used 5′>3′ polymerases, including a 3′>5′ DNA polymerase, a 3′>5′ RNA polymerase, a 3′>5′ reverse transcriptase, and a 3′>5′ RDRP. The proteins should have processivity and error characteristics that are similar to or superior to those of Taq, T7 RNA pol, M-MLV RT, and Phi 6 RDRP respectively. These proteins may be designed de novo, discovered in nature, or engineered or evolved from naturally occurring starting points.

New nitrogenases

Create a new nitrogenase that does not bear sequence or structural homology to the natural family.

Specifically, the protein must convert N₂ to ammonia at rates that are at least of a similar order of magnitude to the rates of naturally occurring proteins, and must fall well below the sequence- and structure-similarity thresholds relative to all known nitrogenase and nitrogenase-like proteins. The protein may be designed de novo, discovered in nature, or engineered or evolved from naturally occurring starting points.

Some money is better than others. This is foundational to how money and banking works. I explore this idea and its many consequences, without any prerequisites or jargon.

Intermediation

Money. The villagers are tired of bartering. The dairy farmer wants to buy corn, even when he does not have milk to trade, and the corn farmer wants to buy meat, even when the butcher does not want corn. So they decide that special gray stones that they can collect from a nearby riverbed will represent an abstract unit of value, called money. They reason that if everyone uses stones to represent value, then people can transact when they would like, rather than when both parties are willing and able to barter. The villagers have abstracted value.

Supply. The villagers picked special gray stones to be money because the stones were portable, durable, and most importantly hard to collect. The only way to get them was to walk an hour outside of town and spend all day sifting through the riverbed. Sometimes, a villager would do this and only find one or two special stones. And so like any other job—winemaking, farming, cobbling—the job of collecting stones was self-regulated by the value of the activity. If the villagers collected too many stones, like they did after a flood cut open a new seam of special stones in the riverbed, then the cost of goods would go up and the relative value of stones, and thus collecting them, would go down. Or vice versa. So the villagers decided that anyone could collect stones, just as anyone could forage for berries or dye cloth. More or fewer people would do it as demand changed.

Debt. The rancher has a problem with money. He raises cows, but this takes a long time, much longer than it takes the dairy farmer to gather fresh eggs. He must go long periods of time without earning more stones. So the villagers decide that some people can simply pay for goods later. The two parties just record the details of the trade on a piece of paper and settle up later. The person who owes money is said to have debt, while the person who is owed money is said to have credit. For example, the woman who owns the general store in town is happy to let the rancher buy on credit, since she has known him since they were both children. However, she does not sell on credit to strangers or to people who do not pay their debts.

Interest. While the general store owner is happy for the rancher to buy on credit, the shoemaker is not. He too trusts the rancher, but he wants money now to expand his business. Since the shoemaker would not be paid in stones for a year—it takes a long time to raise a cow—, the shoemaker cannot use that money to buy new tools or hire an assistant in the meantime. Having stones today is better than having stones in a year. So the shoemaker makes a deal with the rancher: the rancher can have boots today but pay for them in a year; however, rather than paying one hundred stones for the new boots, the rancher must pay one hundred and five stones. The extra five stones are for the lost value of not having money sooner. The villagers like this idea and adopt it. Soon, all debt is repaid with excess stones, which the villagers call interest. The villagers have created the time-value of money.

Bank. The rancher still has a problem. He can buy on credit from the general store and from the shoemaker, but most stores in town will not lend to him, since they do not know or trust him. One entrepreneur in the village wonders about this problem. He notices that the rancher needs to buy on credit, but none of the stores he needs to buy from will lend, while the widow across town keeps a hundred stones in a jar in her cupboard, but has no friends who need the money. The entrepreneur has a clever idea. First, he borrows the stones from the widow, and he promises to return them in one year with an interest of three stones. And then he lends these stones to the rancher, on the condition that the rancher pays him five stones of interest in a year. The business plan is to make the spread, two stones, in a year’s time. This works because the entrepreneur knows both the widow and the rancher. Over time, word spreads, and many villagers who want to borrow or lend come to him. The entrepreneur calls his business a bank. The bank is very profitable, and over time, many banks pop up in the village.

Balance. Eventually, the entrepreneur is borrowing and lending from so many people that there is no correspondance of one person’s lent stones to another person’s debt. At the end of the year, when the widow asks for her money back, the entrepreneur goes into his storehouse to fetch some stones he hasn’t yet lent and gives them to her. He does not even know if they are the stones repaid by the rancher or not, but it does not matter. He even starts letting customers ask for their stones back whenever they would like, to encourage more people to deposit stones. However, this creates a problem: the number of stones in the banker’s storehouse tells him very little. If someone lends him five hundred stones, and then he lends four hundred of those, he will have one hundred stones in his storehouse. But this is a very different situation than the one in which someone simply deposits a hundred stones. So the banker begins to track two lists. On one list, he records everything the bank owns or is owed: the stones in the storehouse and the debt owed by borrowers. He calls these his assets. On the other list, he records everything the bank owes to others, namely deposits. He calls these liabilities. When a villager deposits fifty stones, the banker records fifty stones in liabilities and fifty stones in assets. He calls these two lists his balance sheet, since the bank’s assets must equal its liabilities. Counting his stones in his storehouse only tells him what he has now; his balance sheet tells him what he is owed and what he has promised.

Illiquidity. One morning, the teacher walks by his bank and notices a queue. The bank isn’t even open yet. He asks around, and the people in line say that they heard a rumor that this bank had been lending aggressively and even made some bad loans. Those in line didn’t want their stones to go missing, so they were about to pull their money out. The teacher thinks about it, and decides to wait in line too. By the time the bank opens, there is a very large line. The banker panics. He dutifully gives out all the stones that he can, but eventually he runs out of stones in his storehouse, and there is still a line of people demanding their stones. The banker is frustrated. He knows that his balance sheet balances! He is owed many stones from various villagers. But he does not have the stones now. He does everything he can. For example, the winemaker is late to repay a debt, but the banker and the winemaker are friends, so the banker has allowed the debt to persist. Now the banker forces the winemaker to sell her wine early, at a discount, in order to be repaid today. By nightfall, he asks the remaining villagers to come back the next morning. Then he goes to to another banker in town, the owner of a much larger bank with more stones, and he sells them his balance sheet at a discount. For example, one villager owes the banker two hundred stones in one year’s time. The banker is only able to sell this loan for one hundred and fifty stones, because the larger bank knows he is in trouble. And thus, the smaller bank is forced to close, and the bigger bank assumes his assets and his liabilities. The next morning, the larger bank starts giving money to any depositer that wants their money back, but people stop panicking once they realize the larger bank is the backstop. However, because of this panic, wealth in the village is destroyed. The winemaker was forced to sell good wine at a discount, and the small banker was forced to sell his good debt at a discount.

Speculation. The bankers realize that their business model is inherently fragile due to this timing mismatch: villagers can ask for their deposited stones back before the bank earns back its loans plus interest. If all the depositers were to do this at once, the bank would simply run out of stones. So different bankers experiment with different banking models. For example, one banker does not make money by collecting a spread. Rather, she safekeeps peoples money and charges them interest to do so. Another banker only allows people to withdraw their deposited stones at fixed times, giving him time to ensure he has had some of his loans repaid in order to match the outflowing stones. However, the original banker’s business model is the most popular, because people get paid to store their money and can withdraw it as they wish. Most villagers are happy to accept the risk of a bank running out of money in exchange for being paid interest while still being able to withdraw their money at any time. Much like planting corn is a speculative investment—one could pay money for seed and yield no crop—the villagers realize that depositing money at the bank is a kind of speculative investment. But they are happy to take this risk because they expect to get paid interest.

Creation

Payment. At first, the banking business model was to collect a spread between the interest banks paid on deposited stones and the interest banks collected on lent stones. However, over time, the banks became trusted intermediaries for day-to-day payments. For example, imagine that the carpenter wants to buy goods from various merchants. He does not want to cart his stones around all day. This is heavy and dangerous. So instead, he goes to the bank, hands over some stones, and the bank gives him a paper note indicating that the bank is good for those stones. The bankers called these banknotes. Various shops in town were originally skeptical of this scheme; they thought that banknotes were not money but only the promise of money. But over time, they liked the system too, because they did not have to keep as many stones in the back rooms of shops. Everyone could transact with banknotes, and simply exchange them for stones when needed.

Settlement. This new payment system worked extremely well, because now villagers can buy things when they need them, rather than when they have stones, and they can buy at nearly every shop in the village using debt or banknotes, because the debtor is a trusted third-party, a bank. However, the banks realized something odd: they often become each other’s creditors without trying. For example, imagine that the architect banks at Athena Bank and the zoologist banks at Zeus Bank. When the architect buys from the zoologist, she gives the zoologist a banknote from Athena. The zoologist then goes to exchange this banknote for stones at Athena Bank. But this is a hassle. Now the zoologist has to walk his stones from Athena to Zeus. The zoologist would rather have Athena just deposit the stones directly at Zeus, but Athena cannot do this, as it would require manipulating Zeus’s balance sheet. So instead, the banks decide that the zoologist can deposit the architect’s banknote directly at the zoologist’s own bank, and then Zeus will collect the debt from Athena. The banks call this scheme gross settlement. However, for a brief moment, Zeus is inadvertently a creditor to Athena, because it creates a deposit for the zoologist before it has the architect’s stones from Athena. Zeus is loaning Athena stones, as an artifact of who pays who in the village. So the banks hire the fastest kids in town to run stones between banks. They settle these incidental, transient debts as fast as possible.

Residual. Gross settlement is appealing because it is simple. Athena Bank knows the architect, and Zeus Bank knows the zoologist. Every banknote is settled immediately after the transaction, by stone runners. Neither bank is touching the other bank’s balance sheet, and the zoologist himself does nothing. His stones stay within the banking system. But the banks have problems with this system. First, it is costly, time-consuming, and dangerous to transport stones constantly. And second, it is terribly inefficient. In one day, Athena might transfer ten thousand stones to Zeus, while Zeus transfers eight thousand stones to Athena. It would be better if they netted, if Athena simply transferred two thousand stones. So the banks agree: at the end of each day, the bankers will convene and settle all debts by netting their transactions. They call this nightly meeting scheme net debt settlement and the net payment the residual. At the end of the day, Athena might transfer only five stones to Zeus, but this residual payment says nothing about the day’s transactions. It could mask hundreds of transactions between its customers.

Deferral. One night, the bank leaders convene to settle their debts, and Poseidon Bank asks a question: rather than settle with Athena Bank tonight, could it possibly settle with Athena tomorrow night and pay one night of interest? The bankers thought about this and decided that it was not only acceptable, it was desirable. The ability to pay one’s debts, which the bankers called solvency, is different from liquidity. When the small bank was forced to sell its balance sheet at a discount, it was solvent but not liquid, and the inflexibility of the system caused real value to be destroyed. Or take the fishmonger, who pays his suppliers with banknotes in the morning before going out to fish but isn’t able to sell his fish to the restaurants until evening. Under immediate gross settlement, his bank account was often dangerously low, but it was always full again by nightfall. Thus, the bankers reason, it would be better if the system had some flexibility. Since Poseidon is good for the money and only owes Athena for incidental reasons due to who paid who today, why not defer settlement another day? So the banks agreed that while eventually settling was critical to the system, banks could borrow from each other for one night at a special interest rate, which they called the overnight rate. Just as villagers could go into debt to each other in order to resolve a timing-mismatch, so banks could go into debt to each other for exactly the same reason.

Acceptance. The villagers begin to wonder: what is money? Stones are obviously money, but so are banknotes and even bank deposits. For example, every time the bookseller sells a book, he is either paid in stones directly or he is paid with a banknote. After a while, the bookseller realizes something: he hasn’t seen a stone in a while. Everyone buys from him using banknotes, and he doesn’t even convert that banknote to stones. He simply deposits the banknote at his bank, and then banks settle the debt later, sometimes days later. The bookseller realizes that once he’s handed a banknote, he considers himself paid. Of course, if he only viewed stones as money, he would not be paid until he converted this banknote into stones. But he goes to bed each night with only a number on a balance sheet to tell him he has money. The villagers begin to wonder if maybe all the things they thought mattered about special gray stones—durability, portability, scarcity—were not the real reason people were willing to accept them as money. Maybe money was just anything that another person would accept as settlement for a debt. If this were true, then a banknotes were also money.

Creation. An extremely profitable businessman came to Zeus Bank for a loan, but the banker has a problem. Her storehouse of stones is nearly empty, and she cannot issue more debt without another villager handing over more stones as deposits. But then she thinks about the bookseller. The bookseller accepts banknotes as payment and buys goods for his family using banknotes as well. He has not asked for his stones in the storehouse in years, and the banker does not even think of herself as storing his particular stones anywhere. She only has a pile of stones in the storehouse, and she can’t remember the last time she worried about running out of them. What she does worry about is the residual payment owed at nightly settlement. Sometimes she is paid a little, sometimes she pays a little, depending on payments across the village. And if she owes more than she expects, she can borrow at the overnight rate. In her mind, the real risk is not a villager asking for their stones. It’s her overnight interest payment growing if she keeps rolling her debts forward. This is the risk that she must and can manage. So she takes out her balance sheet, and simply writes down a new line: a liability in the form of new deposits for the businessman and an asset in the form of this man’s debt to the bank. Her sheet balances. This isn’t an accounting trick in her mind, and she doesn’t even think about it as creating money, because she isn’t creating stones. The liability or deposit is simply a claim for stones against her bank. The profitable businessman can now, if he wants, ask for real, physical, special gray stones, and she could give them to him. But he won’t! He will only ask for banknotes and repay his debt in banknotes. Thus, with a stroke of the pen, the businessman has banknotes to expand his business, and the ingenious banker’s residual payments shift, imperceptibly, day over day, as slightly more money in the village is a claim against the stones in her storehouse.

Centralization

Squeeze. Every autumn, all the farmers in town withdraw their stones from their banks to pay the the agricultural workers who bring in the harvest. These are typically poor, itinerant workers who do not have bank accounts. They always want to be paid in stones. On a normal night, the banks’ nightly settlement is easy because everyone in the village is paying everyone else, and so the residual payments between banks is small. The zoologist pays the architect and the architect pays the bookseller and the bookseller pays the fishmonger and the fishmonger pays the zoologist. Money circulates. But around harvest time, many banks struggle to settle because their stones have been withdrawn to pay agricultural workers. Money flows in one direction. The banks fear this night, because often the residual payments are very large. The bankers call this night a credit crunch because the ability to extend credit is restricted, as many banks are suddenly short on stones. The stones do not disappear; they simply leave the banking system temporarily, until the agricultural workers spend their money.

Gridlock. One harvest night, Athena Bank owes Poseidon Bank a large residual payment of one hundred thousand stones, but Athena’s vault is empty because its customers had to pay workers’ wages. As usual, Athena asks Poseidon for an overnight loan, but this time Poseidon says no. Athena argues that while its vaults are empty, this is only due to the seasonal harvest. Eventually, money will flow back into Athena as its customers—many of whom borrowed money to prepare for the harvest—repay their debts. But Poseidon has its own debts to pay very soon and depositers who might ask for their stones back at any moment. Also, Poseidon cannot tell whether Athena made good or bad loans. All Poseidon can see from the outside is that Athena does not have stones. Most of the other banks are similarly constrained by the harvest’s drain on their stones, and Athena simply cannot settle its debt. The problem with the harvest night credit crunch is that Athena cannot create money that Poseidon will accept. Athena can expand its balance sheet to create new deposits that the bookkeeper will accept as money. But these new deposits mean nothing to Poseidon. Money is something that the other party will accept as the settlement for a debt, and so deposits at Athena is not money to Poseidon. But if Athena cannot pay Poseidon, then Poseidon cannot pay Hermes, and so on. The banks cannot settle, and this harvest night, the banking system finally goes into gridlock. The bank leaders and village elders agree to meet the next morning to resolve the crisis.

Backstop. The next morning, the largest bank in the village, Zeus, proposes a solution. It argues that the banks should create an organization that acts as an intermediary between lender and debtor banks during a crisis. Zeus calls this a clearinghouse. The clearinghouse could inspect any member bank’s balance sheet and issue paper certificates against the bank’s assets. Other banks would trust the clearinghouse because it was a neutral third party, run by all the member banks. At first, Poseidon balks at this idea. It argues that you cannot settle a debt by making another one. This is why Athena cannot simply loan itself money and why Poseidon does not want another promise from another bank. But Zeus argues that these certificates are not promises; they are money between banks! If two villagers transact without a bank, the only thing that is money between them is stones. But if two villagers use an intermediary such as a bank, then a hierarchy emerges. One villager can pay another using a banknote and both parties go to bed knowing that there is no debt. The debt is moved up the hierarchy, to debt between banks. But what happens when the banks cannot settle? Zeus argues that the fix is simple and even obvious: the banks should move the debt up the hierarchy by creating a kind of bank-of-banks! Finally Poseidon agrees—what choice did the bank really have any way? —and a clearinghouse is created. The clearinghouse inspects Athena’s balance sheet and then issues a fairly-valued certificate against its assets. Athena pays Poseidon with this certificate, and now Athena has no debt to Poseidon but rather has debt to the clearinghouse. And Poseidon can pay Hermes with a clearinghouse certificate, and so on. And soon, the argicultural workers start buying beer and food and clothing, and stone money starts flowing through the village and back into each bank’s storehouse. Soon, every bank is able to repay its certificate loan, and the banking system survives the harvest gridlock.

Centralization. Over time, the banks agree with Zeus that these certificates were yet another form of money. Between villagers, stones were money and even banknotes were money because neither was any villager’s liability and both were accepted at face-value and without any discount, which the banks called at par. Similarly, between banks, clearinghouse certificates were a kind of money because they were not the liability of any individual bank and they were accepted at par. However, with time, the banks came to dislike the clearinghouse. Zeus was the largest bank and even a competitor and yet had outsized influence in the process. The village elders realized that the clearinghouse, as a bank-of-banks, was the most powerful financial organization in the village. So the village elders stepped in and decided that the village needed an official bank-of-banks, which they called the central bank. They called all the other banks commercial banks. The central bank would serve essentially the same role as the clearinghouse, but rather than being run by member banks, it would be a new administrative arm of the village government.

Reserves. The central bank opened a bank account for every bank in the village. Unlike the clearinghouse, banks had no choice. They could not opt in or out of membership. They were required by law. And rather than issue certificates, the central bank said it would issue reserves. The central bank said that certificates were ad hoc emergency money, issued as part of a voluntary system of member banks, while reserves would be official bank money, issued by the central bank. Furthermore, by law every bank had to keep a certain amount of reserves in its account at the central bank, as a fraction of the amount of deposits it owed its customers. This made reserves money between banks, because now banks needed and wanted to have reserves and because they were accepted at par as settlement for debt between banks. To get more reserves, a commercial bank would borrow from the central bank against the assets on its balance sheet. This moved bank debt up the financial hierarchy, just as villager debt was moved up the hierarchy by banks. And just as villager debt was made flexible by intermediation and money creation, so bank debt was made flexible by the central bank, which could simply create reserves by expanding its balance sheet.

Inflation. Over time, debt in the village grew. The commercial banks were comfortable with the debt in the village, because now they could always settle their debts to other banks by going into debt to the central bank instead. And the central bank was comfortable with all the debt from commercial banks, because it could always expand its own balance sheet to create more reserves. However, as more and more villagers and businesses paid for goods with debt, the price of goods in the village went up. For example, the rancher could only raise so many cows per year, but now people were offering him more stones for each cow. So the prices of cows went up. And so on for other items in the village. The villagers called this increase in prices over time inflation. The villagers speculated that inflation was caused by the village creating money faster than it could create value. A few wise villagers noticed, however, that the problem with inflation was not with stones. The stone supply had barely changed in years. When the village experienced inflation years ago, it was when the flood cut open the river embankment and revealed more special stones. At that time, the impact was moderated because the value of a day’s labor collecting stones was reduced as the value of a stone went down. But now inflation was being caused by the stroke of a banker’s pen, and this labor was essentially free.

Policy. The central bankers thought about the problem of inflation, and they realized that they could control the price and thus the quantity of reserves, which in turn would control the price of money for the villagers. Just as a commercial bank could encourage more villagers to deposit money by offering a higher interest rate on deposits, so the central bank could encourage more banks to hold reserves by offering a higher interest rate on reserves. And since banks were were required to hold reserves as a fraction of the debts on their balance sheet, this meant that the banks would loan less money to villagers. So if the central bank increased the interest rate it offered on reserves, more banks would hold reserves and thus decrease their lending to villagers. And if the central bank decreased the interest rate it offered on reserves, fewer banks would deposit their reserves and thus increase their lending to villagers. So the central bank started to manage the problem of inflation by changing the overnight interest rate on reserves.

Hierarchy. The villagers have constructed a hierachy of money. Villagers settle debts with stones, bank deposits, or banknotes, while banks settle debts with reserves. So reserves are money between banks, while banknotes and deposits are money between villagers. This gave the central bank enormous power. It could change the price of credit throughout the entire village by changing the interest rate on reserves. And in a crisis, it could act as the lender of last resort, creating elasticity in the system by lending when no other bank could. The villagers have built a hierarchical system that allows for both elasticity and discipline in the money supply.

Exchange

Currency. The village has built a financial system that uses special gray stones as money. But over the mountain pass is another village which uses special red stones as money. And over the river is another village which uses special blue stones as money. And so on. In fact, there are many villages in the region, and they each use their locally available special stones as money. In each village, the villagers refer to their stones as simply money, but when discussing money as an idea that transcends all the villages, they refer to special stones as currency.

Trade. The merchant has a problem. The red-stone village is near rich clay deposits and makes excellent pottery, which he wants to bring back to his village to sell. However, the merchant only has gray money, which is not money in the red village. But after some initial bartering, he convinces the merchants in the red-stone village to accept his gray stones as payment. He argues that while gray money is not money to them, it is not worthless either. They can, for example, spend the gray stones in his village when they travel there for business, or they can exchange the gray stones for red stones with other red villagers who plan to travel to the gray village. The red-stone merchants eventually agree, and they sell their pottery for gray stones. But they include a markup on the price, since gray money is inconvenient and must be converted. Over time, all the villages trade with each other. However, trades are limited, because not every merchant wants the inconvenience of being paid in a foreign currency and because imported goods are expensive due to the markup.

Exchange. An entrepreneur notices that many merchants have red stones that they do not want. They trade with the red-stone village because it is worthwhile, but they would prefer to be paid in gray stones. The entrepreneur thinks that the inverse problem must exist in the red-stone village: those merchants must have gray stones that they do not want. And so she forms a business: she buys red stones from the merchant in her village using gray stones, and then she travels over the mountain pass to the red-stone village and buys gray stones with red. The villagers in town start to call her a currency trader. Just as a horse trader specializes in trading horses, the currency trader specializes in trading currencies. The currency trader quotes her price as exchange rate, which reflects her estimate of the relative value of stones in two villages. This rate fluctuates, as the money supply and the prices of goods in both villages slowly drift. And of course, she adds a markup or spread onto this rate for her services. Currency trading is very profitable, and over time, many exchanges pop up. As exchanging currencies becomes easier and cheaper, the villages trade more.

Correspondence. But the currency trader has a problem: transporting stones between villages is dangerous and laborious. So she opens bank accounts in all the villages in the region, and rather than trading stones, she trades banknotes. The banks notice her work and that their customers are often receiving foreign currency, and they wonder: why not simply accept banknotes from other villages and then perform this exchange themselves? Then they could collect a currency exchange fee. A gray merchant could receive a red banknote, deposit it in his local bank, and receive gray deposits in return. His bank would then warehouse the foreign currency and eventually exchange it for gray money. The process could be similar to nightly settlement in a single village. And so the banks open accounts with all the other banks, and they hire currency traders to manage exchange rates and their growing balances of foreign currencies. The bankers call this correspondent banking. And so just as payments between villagers created debts between banks, trade between villages starts creating debts between banking systems.

Exposure. Correspondent banking made trade between villages easier. Now a gray bank could simply accept a red banknote from one of its customers. However, this red banknote was only a promise from a bank in another village. Ultimately, the gray bank needed to know that the red-stone village bank was good for the money. As with nightly settlement, the residual payment between banking systems was typically small. The gray village bought pottery from the red village, while the red village bought cows from the gray village. Money circulated. But the central bankers worried about the political and economic health of the other villages. They thought about their own struggles with inflation and credit squeezes, and wondered what would happen if these happened in another village. There was no central bank above villages. What if another village failed to repay their debts? The gray village could create gray money, but it could not create foreign currency, force a foreign bank to pay its debts, or enforce its laws on foreign bankers. And so as the debts between villages grew, the central bankers monitored the political stability and economic health of their trading partners. They reasoned that a foreign currency was only as good as the village that issued it.

Default. Like other villages, the red-stone village funded itself through taxes. However, the government also funded itself with debt: banks, businesses, and individuals would give the elders money, and the elders would promise to repay the debt with interest. The bankers called these promises bonds. Many people liked to own bonds, because it seemed like a relatively safe way to make interest. However, over many years, the red-stone village borrowed more and more by selling bonds. The village’s debt became very large, and after a few poor harvests, many local businesses struggled and tax payments dwindled. A wealthy lawyer in the red village worried about his government. He worried that his central bank might pay off its bond debt by issuing yet more bonds, this time by creating reserves and selling the new bonds to commercial banks. The debt would roll from public bondholders to commercial banks, and the central bank would pay for this by expanding its balance sheet, by simply creating money. He knew that when this happened, there would be more red money in the system chasing the same amount of goods, and so the red village might experience inflation. So every so often, this lawyer would go the currency trader in town and convert some of his red banknotes to black banknotes, since he thought the black-stone village had the strongest economy. At first, the currency trader was happy to exchange one red banknote for one black banknote. But soon, as the red-village experienced inflation, many people in the red-stone village wanted black stones instead of red. The currency trader started demanding two red stones for one black stone, then three, and then four. The red-stone village’s economy continued struggle, because now importing goods was more expensive, since red stones were worth less relative to other currencies. Finally, the red-stone village told the other villages in the region that it would not repay its loans, since it could not risk creating more red money without extreme inflation. The bankers called this a default.

Reserve. During the red-stone village’s debt crisis, no one thought that black stones were completely safe. Rather, many villagers simply preferred to hold black stones rather than red. Like the lawyer, everyone trusted the black-stone village more. This is because the black-stone village, which was high in the mountains, was the wealthiest village by far. It had a strong military, a robust economy, transparent monetary policy, and a fair judicial system. People trusted that black money would retain its value. Over time, black money had simply become the most trusted money in the region, and merchants from all the villages found themselves transacting with black money because everyone had some. When a merchant was offered a black banknote, she would happily accept it; often, she would not even bother taking it to a currency trader to convert. Like the bookkeeper who thought himself paid when he received a banknote, the merchant thought herself paid when she received black money. She did not think, “This money is better than my money.” She simply didn’t bother to exchange it. And during any sort of financial crisis, people would quickly exchange their domestic money for black money. The central bankers noticed this, and they started to refer to black money as the reserve currency. They used the word “reserve” because, much like central bank reserves, black money acted as a settlement asset, this time between banking systems.

Fiat

Devaluation. The purple-stone village is also struggling. The village specializes in making clothes; it has spinners and weavers, knitters and dyers, tailors and dressmakers. However, the village struggles to export clothes, since other villages also make their own clothes at competitive prices. So the village’s bankers propose an idea: what if the purple central bank expanded its balance sheet to create reserves and then used those reserves to buy foreign currencies. Then there would be more purple stones relative to foreign currencies, which would decrease the price of purple money. The bankers called this currency devaluation. Why, the village elders ask, would they want to do that? The bankers reply that if purple money is cheaper relative to, say, black money, then in the black-stone village, purple clothes would be cheaper than black clothes. And so black-stone villagers would buy more purple clothes. Of course, this would mean that the purple village would struggle to import goods, but it would thrive at exporting them. After much debate, the elders agree, and the purple central bank begins devaluing its currency. Some villages enjoy the cheaper clothing from the purple village and allow their local clothing industries to struggle, while other villages protect their local industries by levying a special tax on imported clothes, called tariffs. Over time, many villages devalue their currencies to become more competitive, while others impose tariffs to protect their domestic industries.

Conference. The central bankers debate monetary policy. They debate topics like currency devaluation, extreme inflation, and banking system defaults. They realize that trade between banking systems is lacking cooperation and flexibility. Each village is engaging in competitive or protectionist policies that limits free trade. And a village default impacts everyone, since there is no backstop. So the elders agree that they should meet and discuss a resolution, and they gather in mid-summer at a beautiful hotel in the black-stone village. After much debate, the leaders decide to formalize a few things. First, they agree that black money would be the region’s official reserve currency, and that a single black banknote would always be convertible into thirty-five black stones. Second, they decide that each central bank would keep its exchange rate with the black currency fixed. They called this dynamic a currency peg. This meant that each central bank would maintain a balance of black money in reserve and would then buy or sell this black money in exchange for its own currency, in order to maintain the exchange rate. For example, if red stones were worth too little relative to black stones, the red central bank would buy red stones for black. The idea behind this system was that that if black money was stable and if every other currency was pegged to black money, then every other currency would also be stable. Finally, they agree that some flexibility was needed in the system, and they create a clearinghouse for the central banks. This would be analogous to a clearinghouse for banks within a single village: if any central bank struggled to defend its currency peg due to liquidity issues, this new clearinghouse could lend as a last resort. In theory, this system would prevent currency devaluations and protectionist policies, limit the fallout of debt defaults, and add flexibility during gridlocks.

Privilege. This status as the region’s reserve currency gave the black village an important advantage. Other villages had to make and sell goods in order to acquire money used to trade. But the black village could, within limits, acquire goods simply by issuing money and debt that everyone else wanted to hold, because people preferred to save and trade using black money, and now because central banks needed to maintain some black money in reserve. This made debt cheaper for the black village, and the black government could fund public programs more easily, because everyone was happy to hold black bonds. Furthermore, black villagers could buy cheap goods and services from across the region, because everyone wanted black money.

Dilemma. However, the success of black money created a dilemma. Over time, the other villages accumulated vast quantities of black banknotes and debt denominated in black money. This meant, however, that there were many claims for black money across the region. And just as the teacher worried about convertibility of his bank deposits into special gray stones, so central banks wondered about convertibility of black money into special black stones. As long as few banks tried to convert, this was not a problem. But as more and more black money flowed through the system, the central banks wondered: was every black banknote really worth thirty-five black stones? And thus a dilemma arose: the more successful black money was, the harder it became for the black central bank to maintain the promise of convertibility.

Float. The black-stone village elders had a problem. There was too much black money in the system, relative to black stones held by the black central bank. To maintain convertibility, they would need to make black money more expensive. They could buy back black money using foreign currencies, but they were constrained here. There was much more black money than any other currency. And they could raise the central bank’s overnight interest rate and thus raise the price of money in the village, but this would discourage villagers from taking out loans. It would hurt the black village’s economy. In other words, the black central bank was struggling to defend its own kind of peg, that of convertibility of a black banknote into thirty-five special black stones. And so after much discussion, the elders of the black-stone village made an extraordinary announcement: the black central bank would no longer exchange its banknotes for special black stones at all. Anyone could trade black stones, but their price in terms of black banknotes would not be fixed by convertibility; the parlance of the central bankers, the price would float.

Fiat. At first, elders and bankers and traders around the region were shocked. Even the black village’s central bankers worried about what would happen next. And yet nothing happened. Everyone in the black village still had to pay taxes with black money. Wages, loans, and contracts were still denominated in black money. Commercial banks settled debts using reserves from the black central bank. And the black-stone village was still the strongest economy in the region, with a large military, a liquid and transparent financial system, and a relatively fair judiciary. People across the region still preferred to hold black money over any other, even though a black banknote was now just a piece of paper which could not be converted into special black stones. The bankers called this new system fiat money, because its value depends on the institutions and economy of the black village, not on convertibility into a commodity whose supply was governed by labor. Of course, the elders of the black village were still constrained. They could create unlimited amounts of black money, but they could not create unlimited amounts of goods from the black village: eggs, bread, cloth, wine, jewelry—these all had to be produced by people in the black village. So if the black central bank created money recklessly, they might experience inflation, and other villages might lose trust in the system. But within reason, fiat money gave the black village immense flexibility and power, while still maintaining the village’s status as the region’s reserve currency.

Hierarchy

In the beginning, special gray stones were money. However, the villagers ran into a problem with stone money: it was inflexible. So the villagers created debt, but a villager could not settle a debt by making more promises. And so banks emerged as a layer above stone money. Now villagers could settle their debts with banknotes, because banknotes were a promise from higher up the hierarchy. Then the banks ran into the same problem: a bank could not settle a debt to another bank by creating more of its own deposits. And so the central bank emerged as a layer above bank money. Now banks could settle their debts with reserves, because reserves were a promise from higher up the hierarchy. Finally, the banking systems themselves ran into the same problem but with currencies: one village could not settle a debt to another village by creating more of its own currency. And so a reserve currency emerged as a layer above. Now villages could settle their debts with reserve currency, because the reserve currency was a promise from higher up the hierarchy.

And so the pattern was: within each level, money was whatever the counterparty accepted as final settlement, and this could be promise if it was backed by the level above. The black village sat atop this hierarchy, with a promise to convert black banknotes into real, physical, special black stones. But in the end, this too was just a promise, and the black village was able to decree, by fiat, that black money just is. The black village could do this because black money was the most widely accepted form of final settlement. But the system rests on trust. And if the system rests on trust, then the trust can erode through bad governance, corruption, poor fiscal policy, and competition. But for now, black money is the best money in the world.