If you're just looking for the results, below is a custom color picker based on the color space written in Javascript and a sample procedural generation algorithm in Python (Javascript equivalents are in the page source) - feel free to take this math and go have fun depicting our diverse world!

The goal of this project was to define a color space that makes it easier to build inclusive color tools for a variety of contexts - such as character creators or digital art. If you see something here that sparks your curiosity, I would love for you to stick around and read below this section to learn more!

# Plug the output of one of the select_point implementations into to_rgb(t, u, v)

def select_point(r_square: float = 2.) -> tuple[float, float, float]:
    """Uniformly sample from the sphere deterministically"""
    radius = r_square ** (1. / 2)

    phi = uniform(0, 2 * math.pi)
    costheta = uniform(-1, 1)
    n = uniform(0, 1)

    theta = math.acos(costheta)
    r = radius * (n ** (1.0 / 3))

    t = r * math.sin(theta) * math.cos(phi)
    u = r * math.sin(theta) * math.sin(phi)
    v = r * math.cos(theta)

    return (t, u, v)

def select_point(r_square: float = 2.) -> tuple[float, float, float]:
    """Uniformly sample from the sphere using rejection sampling"""
    radius = r_square ** (1. / 2)
    R = radius + 1

    while R > radius:
        t = uniform(-radius, radius)
        u = uniform(-radius, radius)
        v = uniform(-radius, radius)

        R = (t**2 + u**2 + v**2) ** (1.0 / 2)

    return (t, u, v)

def to_rgb(t, u, v) -> tuple[int, int, int]:
    x = (t - 0.15) / 0.45
    y = (v - 1.2 * t ** 2 + 0.2 * t + 0.655) / 1.84
    z = u / 3.6

    r = 28.77438370854 * x + 36.78307445559 * y - 19.69766918644 * z + 187.1436241611
    g = 35.38327306318 * x - 2.009931981182 * y + 47.93462563172 * z + 137.1073825503
    b = 36.14733717939 * x - 43.54346996173 * y - 28.50821294135 * z + 108.2241610738

    return int(r), int(g), int(b)

Overview

What colors are we? The short answer is maybe something like “brown” and the long answer is very, very long. Representing the broad range of human skin tones digitally is a hard problem. Often, a limited set of colors is presented as being good enough to cover the full spectrum of diversity. However, in using a specific set of colors, large groups of people are unable to accurately be represented, or might be unintentionally excluded.

The goal of this work is to identify the broadest inclusive range of colors in the RGB color space that correspond to plausible, but simplified skin tones. In particular, the aim was to identify simple, “good enough” equations which define that area, allowing the range to be used in a variety of contexts. Calling the equations "good enough" is intended to keep the limitations of this work at the forefront - the results are a useful starting point, but should not be taken to be authoritative.

Introduction

Although there have been improvements in the set of colors that are presented as representative of us, there's still a gap that needs to be closed. Emojis say we're 5 shades (or cartoon-yellow); a makeup brand might say 50; and a character creator might shrug and tell you to pick from all 16,777,216 options. If you look outside - or just at yourself - you'll quickly notice that none of those can compare to the variety of reality; one person is not just one color. Despite that, it can be useful to try to boil things down to fewer values. The Unicode Consortium and makeup companies can figure out their own ranges, but I believe we can find a better solution that's somewhere between "several" and "several million."

Taking the digital art world as an example, images such as the one below are often circulated in an attempt to assist other artists in identifying plausible colors.

a lumpy rectangle shape with a spectrum of skin colors smeared together

"Flesh Cloud" by Tumblr user shiroxix

In the video game world, nowadays the preset colors are often wide ranging and supplemented with a general color picker, but it would be even better if the initial experience presented better options.

a skin color picker with 18 choices and an extra spot for bringing up a generic color picker

Screenshot from character creator for the early access game Paralives

Limitations

Taking a step back to reality, this work has a number of inherent limitations.

As mentioned before, skin tones are much more complicated than a single color. They vary widely between different areas of the body and are subject to complex biological processes. The perceived color of the skin is affected by blood flow, concentrations of melanin, complex scattering of light through the layers of the skin, as well as things like vitiligo, freckles, hyperpigmentation, scarring, and other common variations.

Secondly, a variety of health conditions can cause people to have skin tones that are well outside what might be perceived as plausible. Argyria can often lead to skin that is blue-gray in color; high bilirubin can cause skin to be yellowish or greenish.

Additionally, it's important to note that I am one person, not a researcher, and subject to my own general biases on top of my own perception of color. As far as I am aware I don't have color vision deficiency, but a lot of the choices I made are completely subjective.

Finally, colors are not perceived consistently across display types and viewing environments. RGB values look different between different screens, and people look completely different under different lighting conditions.

Broadening this work to address some of these limitations would be an interesting area of further investigation - a goal of this project was to be good enough for simple use cases, but these limitations might be more problematic in other contexts. The results here will mainly be applicable in contexts that relate to generating simplified representations of people.

Methodology

What are all of those numbers, and how did you get them?

Content warning - unscientific methodology below. In other words: good enough is fine if you're an engineer.

Summary:

1. Manually label colors in RGB in order to get a rough approximation of the dataset shape
2. Perform a principal component analysis (PCA) with N=3 on the dataset to change the shape into something easier to work with
3. Use your preferred graphing software to manually create equations that map a sphere in the target space onto the transformed data in the XYZ, or PCA space

Manually Labeling Colors

a webpage with sets of different colored faces on the left and right - the faces on the left have colors more like skin tones, and the ones on the right are purples, blues, and greens

Data labeling UI

Sometimes the best place to start is by doing a ton of tedious work. I didn't label every color in RGB, but there certainly were a lot. This is the UI I built for labeling - just a simple webpage where you click to move a face between the yes and no side. Initially they were just squares, but I ended up drawing a face and slapping the colors onto it. Gazing into my lumpy research assistant's eyes made it much easier to quickly go “yeah I can imagine that person walking around”.

Graphing that data, you get the following shape - for visualization it's kind of like a banana shape that swoops between 0, 0, 0 and 255, 255, 255. The curve is more towards red and away from blue.

a 3D graph of points - the colors range from darker on the bottom left to lighter on the top right

Matplotlib graph of the labeled data

This is where I got stuck for the longest. For your sake, I'm going to skip over all of my dead ends and struggles along the way. There were many. Far too many. Things involving convex hulls, regression misadventures, 4th degree polynomials, and the worst looking code I've ever written.

The result of all of that was that I decided I needed some way to transform the shape into something easier to work with: that's when I learned about principal component analysis and was immediately inspired.

Humanities Intermission

Although all of the math and technology here is fun, it's important to address the fact that technology exists in a social context. Stating things plainly: lighter skin colors have both presently and historically been celebrated and prioritized while darker skin colors have been marginalized and maligned. Racism and colorism are systemically present in many cultures, in both overt and subtle ways.

Below are some great videos, essays and projects that address these issues through a variety of lenses. I highly recommend taking a look, especially if you want a break before we get into the math.

In her video series The Darkest Shade, Nyma Tang reviews the darkest shades from a variety of makeup brands. In her own words, "It's important for makeup brands to make products for all shades and as someone with a darker skin tone, I want to be able to help others who struggle to find the same!"

In a similar vein, Kat Blaque's video (for introspective hot people) Youthforia's Blackface Foundation is about a makeup brand that released a dark shade that was literally black - far darker than what would be plausibly useful.

Finally in the makeup space, the Vox video How beauty brands failed women of color talks about the history of discrimination in the beauty industry and the limited availability of deeper shades. There's a lot of great expert interviews as well as a discussion of the intersection with Black history.

Taking a look at video games, Me, On The Screen: Race in Animal Crossing: New Leaf by Austin Walker is an excellent essay about the author's struggle to see himself represented in video games. In the years since, Animal Crossing has gotten much better about inclusivity but the essay also goes into his history of not seeing himself - or seeing characters that look like him be stuck in racist tropes.

The Humanae photography project by Angélica Dass is "an unusually direct reflection on the color of the skin, attempting to document humanity's true colors rather than the untrue labels “white”, “red”, “black” and “yellow” associated with race." In a way it's really similar to the work here - except with a focus on documenting and conversing instead of describing, and using Pantone shades over hex codes.

For more direct resources, the video It's not a Coincidence. It's Colorism. by Tee Noir gives a description of what colorism is, and a few ways it manifests in modern pop culture.

Finally, Writing With Color is an excellent blog with many resources about writing various kinds of diversity. The blog is run by a team with expertise in many different areas, and they often take reader questions. In particular I found their post Words for Skin Tone | How to Describe Skin Color really interesting. As more of a reader than a writer, it's fascinating to see behind the scenes on how to make better word choices.

Principal Component Analysis

Briefly, principal component analysis (PCA) is a way to rotate and stretch a dataset so that the main directions of variation lie along the axes of the coordinate system. To make that more concrete - recall the banana shape of my dataset from before. In its original form, the banana was sort of floating in space. PCA took that shape and placed in on the ground so that it was symmetrical about the axes and aligned nicely. It also did some slight stretching and rescaling, but that's less important.

The graph from before and after applying the PCA transformation

two 3D graphs with an arrow labeled PCA pointing left to right - on the left is a curve from the origin to the top right, and on the right is a similar curve but aligned with the axes

Aside from the modified dataset, another output of the analysis is a matrix that can be used to take a point from RGB space and translate it into the resting-on-the-floor space. I refer to this as PCA space, or XYZ space since those are the variables I use for it.

Manual Function Fitting

To summarize what we've done so far: we started with data in RGB space (the manually labeled dataset). PCA gives us a way to take points in that space and transform them into another space - XYZ space.

The dataset in XYZ space, graphed in Desmos

3D graph of a cloud of points

Now the goal is to fit some function to all of these points. Because it's a dense shape, if we can define the surface of it with an equation, we can modify that equation to include the points on the inside as well by switching the equation to being an inequality (eg from x² + y² + z² = 2 to x² + y² + z² ≤ 2).

The process for fitting the function is as follows:

1. Start with a function for a sphere: R² = t² + u² + v² (see Adjusting R² for why a sphere is used)
2. Define t, u, and v in terms of x, y, and z
3. Modify how t, u, and v are defined until the sphere stretches to match up with the expected data
4. Invert the relationships in order to define x, y, and z in terms of t, u, and v

What the graph looks like when t = x, u = y and v = z

3D graph of a cloud of points with a sphere overlapping with the middle

If you have (t = x, u = y, v = z) as a starting point, you can start to play with those relations until you start to get an intuition for how changes to the equations will end up shifting your sphere around.

So I did that! A lot. To put it plainly - I manually fit functions to my target cloud of points. There is no regression here, I literally did guess and check and eyeballed the function fit. I did this in Desmos 3D - if you haven't used Desmos in a while it's amazing how good it was for this.

What my Desmos “workspace” ended up looking like.

graphing calculator display - on the left are a set of equations, and on the right is a 3D shape that lines up with the cloud of points

The functions that relate (t, u, v) to (x, y, z) can be used to translate a point between those two spaces. Principal component analysis gave us a way to move between RGB and XYZ, and then these new equations give us a way to move between XYZ and TUV. Linking up all the transformations gives us the functions that take us between TUV to RGB.

Am I endorsing this methodology? No. But also kind of. In this specific situation it's what worked to get it done and get functional equations. It would be great to see someone do this properly, with nice clean symbolic regression and really good training data. But I figured since the training data I labeled was middling quality at best, I'd probably have a better time using my guess and check and then eyeballing the results.

And all that being said, I think I would endorse the overall method I used, if not the specifics. That being "Label Data" → "PCA" → "Fit A Spherical Equation".

Results

From the equations, the sample code at the top of the page is just a short translation away. The next section is the description of the components of the Picker UI, which also serves as an explanation of what TUV space actually looks like, and what properties it has.

Picker UI

the color picker interface from the top of the page with parts that control T, U and V labeled

Sorry - this one's not interactive

Just like how a typical color picker allows you to adjust red, green and blue values (or hue, saturation and value), this color picker also has 3 independent values to adjust - referred to in the code samples as T, U and V.

Component	Aspect	Variable Name
Up/Down On Square	Deep/Fair	T
Left/Right On Square	Flushed/Ochre	U
Left/Right On Slider	Cool/Warm	V

What's really interesting about these controls is that the concept each is adjusting is completely an output of the principal component analysis. Or in other words, I didn't know that these would be what the directions controlled until after I had made the picker. It's a really neat demonstration of what principal component analysis does - it rotates data to maximize how meaningful each axis is.

Now, if you're experienced with color spaces you might be thinking that this looks a lot like HSL/HSV. However, when I was doing my first experiments with the labeled data those color spaces didn't quite capture what I was looking for. I think in particular that although T and U seem similar to Value and Hue, they don't completely match up. And the biggest difference is between V and Saturation. V ranges from blue to orange, while Saturation would look more like gray to orange; although at some (T, U) values V might look a lot like Saturation, overall it doesn't quite correlate.

Adjusting R²

What is R²? R² is the radius of a sphere in the skin tone color space. Because one of the steps of translation from RGB to TUV involves a sphere, it makes spheres a natural shape to work with in TUV space. A lot of the cool properties that arise from that fact are just a side effect of spheres having a lot of useful properties.

To start with a visual, below is a table of R² values, and the result of using a sphere with that value:

R²	Description	Sample
0.1	Almost no variation
0.5	Minor variations
1.0	Minimum value to have decent variety when randomly picking
1.5	Good variety while maintaining realism
2.0	Potentially cartoonish - random selection will have occasional outliers
2.5	Randomly selecting in this range might give unexpected values, but useful in a picker UI
10	Maybe something for a fantasy setting?	Regenerate Faces

So what's going on with that table? Because a sphere is defined by a radius, you can define useful sampling ranges using only a single value and then easily select points from within that sphere. For example, in the sample implementation of the tone generation function I use a radius of 2. This is the radius that was used when developing the color space and generally will result in a broad range of tones. However, if you find there are implausible results generated using that radius you can decrease it by a bit in order to reduce the variation in generated colors.

The fact that the sphere shrinks along a radius means that the reduction in variation actually does not result in a large reduction in representativeness. In other words, reducing the radius doesn't lead to just chopping off deep or fair skinned colors - deep skin tones, fair skin tones, flush skin tones, ochre skin tones, cool skin tones and warm skin tones are uniformly decreased in variation.

Taking R² all the way down to 0, we can identify the origin of the color space. If the space is well formed then this origin point should be a very neutral ambiguous tone. If the origin seems biased in a particular direction, then that would indicate that the space needs more fine tuning. Essentially, the origin point can be used to identify bias in the space mapping.

At the other extreme, you can increase R² in order to allow for more variation. As an example, I do this in the color picker UI. In a picker UI the user can ignore implausible colors, so there's little issue with increasing the variation too far. In a color generation context you'd likely want to avoid having implausible colors popping up. There is likely not an ideal R² value that works in all contexts, but having a single parameter to tweak makes it easy to adjust the equations to your needs.

In Summary

We have a color space, we have a picker for it, we have a procedural generation algorithm, and we have a methodology for creating new spaces or modifying the existing one. Overall, I'm thrilled with this result. It's certainly not perfect, but I don't think a simple and completely correct solution exists. I've already used the picker in a digital painting and it was exactly as I was hoping it'd be - and I'm looking forward to using the generator in my next projects.

What's Next?

First is feedback: if you at all found this interesting or useful, please let me know! Although I did this work for my own use, hearing from others is also so rewarding. Also please reach out with any questions or feedback - especially if you spot something I didn't think about.

Although I would love to post an email address for accessibility, I think the safest solution is to use the issues page of the repo for this project to contact me. But open to suggestions on alternatives for that as well!

A task for me is to clean up my code and post it to the repo for this page to make it easy for people to recreate and iterate on this work.

Future Work

Refining The Space

In defining the color space so precisely and using specific R² values, there is a risk that some colors have been excluded or marginalized. However, that same concreteness also allows for extremely precise critiques of the color space, and therefore extremely precise improvements. As mentioned before, my process involved a lot of subjective eyeballing - a great iteration on that would be to have multiple people, perhaps experts, labeling data and then measuring the output more precisely against that.

Skin Variations and Conditions

A direction mentioned earlier would be to look into modeling simplified versions of various conditions. What modification might you be able to apply to base tones to simulate that person having jaundice, or argyria, or becoming pallid? What colors are freckles, vitiligo, scars, hyperpigmentation, or stretch marks relative to a person's base tone?

Technical Improvements

From a technical direction, making the equation generation more formalized is another potential direction to go. Starting with the shape of equations I manually determined, symbolic regression on those against better training data might lead to an even better color space definition.

There's also the potential for optimizing the equations for different contexts. I've chosen to write them out in code as they are to be the most readable to a broad audience. However, keen mathematicians might notice that one of those steps looks a lot like a matrix multiplication, which can be rewritten to be more efficient in certain contexts.

It might also be interesting to see what the results look like if you transform the data into another color space before doing the principal component analysis and equation fitting. There's a chance the equations end up being simpler or more representative. Because of the sphere, the shape of the surface in the color space you build from will be maintained and it will mainly scale with R² values. For an illustration of this idea, see the graph script in the repo's code.

Thank Yous

• My friends, for listening to me ramble and for coming up with great suggestions
• This video, for helping to break my roadblock by explaining principal component analysis
• All of the creators mentioned in the Humanities section, for sharing their works and thoughts
• Tools
  • matplotlib, a library for graphing data in Python
  • scikit-learn, an ML library with an implementation of principal component analysis
  • Desmos, a free online graphing calculator
  • SymPy, a symbolic mathematics library
• You, for reading!

The concept of recursive self-improvement (RSI) dates back to I. J. Good (1965), where he defined an “ultraintelligent machine” as a system that can surpass humans in all intellectual activities and design better machines to improve itself. Yudkowsky (2008) used the phrase “recursive self-improvement” for a specific feedback loop: an AI uses its current intelligence to improve the cognitive machinery that produces its intelligence.

This feedback loop in modern AI may indicate the model rewriting its own weights directly, or more broadly the model improves the training pipeline and the deployment system, which in turn enables a better successor model with improved performance across economically valuable tasks. The speed of research development in AI has been shown to drastically accelerated in frontier labs (Anthropic; OpenAI).

I explicitly mention “deployment system” because the layer between the raw model and the real-world context seems to be as important as the model’s raw intelligence (i.e. the evals right after pretraining). Harnesses are important components of AI deployment, as shown by successful coding agent products such as Claude Code and Codex. A harness is the system surrounding a base model that orchestrates execution and decides how the model thinks and plans, calls tools and acts, perceives and manages context, stores artifacts, and evaluates results.

This one post will focus on research around harness engineering and how it contributes to RSI. Much recent work on auto-research, self-improving agents, and evolutionary program search can be organized around this question. Other work on model self-play, synthetic data, test-time training and a broader theme of continual learning also matches the RSI vision (e.g. Yuan et al. 2024, Chen et al. 2024), Zhao et al. 2025, Choi et al. 2026)) but they will not be the focus of this post.

Harness Design Patterns

Compared with early agent frameworks, “agent = LLM + memory + tools + planning + action”, harnesses engineering additionally include workflow design (e.g. loop engineering), evaluation, permission controls, and persistent state management. It is no longer only prompt templates, but closer to runtime and software system design: how the model observes, acts, memorizes, checks itself, and improves.

The design should be deliberately simple and generic to enable generalization, likely with reference to existing software engineering practices to benefit from prertaining knowlege. There is also a strong analogy between operating systems and harnesses. Similar to an OS, a harness should encapsulate complicated logic while keeping the interface simple. Meanwhile, configs, tool interfaces and other protocols may gradually become standardized across the industry.

Pattern 1: Workflow Automation

Defining a workflow in which the model can operate, test, and iterate is a key design for automation. Karpathy’s autoresearch repo (https://github.com/karpathy/autoresearch) is a clean example of how such a workflow can be constructed. A common workflow follows a goal-oriented loop of plan, execute, observe/test, improve, and execute again until the goal is achieved. The process may trigger proactive requests to users for clarity in task specification or execution preference.

A simplified Codex agent loop: the agent calls tools and tool responses affect the model's next generation.
(Image source: OpenAI codex agent post)

The workflow graph also emphasizes the model analyzing its own trajectories and failure cases and then iterating on its progress through an “agent runtime” rather than a static prompt template.

Pattern 2: File System as Persistent Memory

A recurring pattern in long-horizon agent systems is simple control over rich states and artifacts. A harness should not carry the entire workflow and all logs in context; instead, it should keep durable state in files. In long-horizon agentic rollout, artifacts such as experiment logs, code diffs, paper summaries, error traces, and past rollout trajectories often grow much longer than the context window that the model has trained for.

Learning how to read, write, and edit the file system (commonly via bash commands) is a foundation skill for LLMs, and thus managing persistent memory in the simple form of files naturally benefits from improvements in core model capability.

Pattern 3: Sub-agent and Backend Jobs

A harness can spawn multiple subagents to execute in parallel and monitor backend jobs. This is useful when the main agent needs to search multiple hypotheses, run experiments concurrently, or delegate isolated subtasks without polluting the main context. The parent agent then needs a small process manager: launch jobs, inspect logs, cancel failed runs, and merge results back into the main agent thread.

The key design choice is to make parallelism explicit and inspectable. If subagent outputs only live in a transient chat context, they quickly become obselete and hidden. If they are stored as files, logs, and status records, the model can recover after interruptions and reason over its own execution history.

Case study: Coding Agent Harness

The core interface of mainstream coding agents has become stabilized across Claude Code, Codex, OpenCode, and Cursor-style agents. They commonly use a loop like:

With access to a set of tools, the coding agent is able to develop and debug issues in a given repository, similar to how human developers are equipped with IDEs.

(Not a comprenhensive list; shown for demonstration. Read this if interested.)

Group Tool definitions File system - File discovery: glob, grep, ls
- File read: read, read_many
- File modification: write (a whole new file); edit (string exact-match replacement); multi_edit; apply_patch (applies a structured patch/diff) Shell execution Run commands: bash, PowerShell IO lsp, git tools like git_status, git_diff, git_commit External context MCP tools, Skills Web search web_search, web_fetch, browser tools Artifacts Read docs, images; generate HTML, images Backend processes Such as: CronCreate, CronDelete, CronList Agent delegation Such as: spawn_agent, resume_agent, wait_agent, list_agents, close_agent, interrupt_agent, etc.

Harness Layer vs Core Intelligence?

It is hard to forecast how much the future of RSI will rely on harness engineering, but the near-term path of RSI is unlikely to start as a model directly rewriting its weights. My prediction of a practical near-term path is:

1. Harness engineering will evolve in the direction of meta-methodology (i.e. improving the machinery for getting better answers, not just improving the answer itself). The harness system itself becomes an optimization target, with fewer heuristic rules and more general mechanisms.
2. In turn, mature harnesses enable auto-research for model self-improvement loop and smarter models prevents harnesses from overengineering and keep the system sustainable.

Eventually it is possible that many harness improvements will be internalized into core model behavior, but the interface with external context and tools should remain. We have seen a softer version of this pattern with prompt engineering: manual prompt tricks became less central as instruction tuning and model reasoning improved, but the need to specify goals, constraints, context, and evaluation did not disappear.

Harness Optimization

The progression in the object being optimized in the harness system is roughly: instruction prompts → structured context → workflow → harness code → optimizer code. As the model becomes more intelligent and powerful, we move toward more complex targets and generic methods.

Context Engineering

Simply appending all the tool responses and model generations into the context can quickly grow out of control as the agentic job horizon increases significantly. Context management is a layer to construct a more structed and concise context for LLM and manage persistant states. There is no doubt that long-context research will keep on making progress but at the moment long-context intelligence and context engineering sometime intertwines.

Agentic Context Engineering (ACE; Zhang et al. 2025) treats context as an evolving playbook rather than an increasingly lengthening prompt. It has three components to maintain one context playbook of bullet points, each with an identifier and a description.

1. Generator: produces task trajectories, with reference to bullet points.
2. Reflector: distills insights from successful and failed trajectories.
3. Curator: updates the structured context with incremental, itemized entries.

The framework of Agentic Context Engineering (ACE). (Image source: Zhang et al. 2025)

To prevent context collapse and brevity bias during iterative rewrites, one key design choice in ACE is that the curator does not rewrite a full prompt blob. It instead outputs a collection of structured, itemized bullets in the form of (identifier, description), and these bullets are merged into a structured context logbook with deterministic logic. The context items are refined and deduplicated periodically.

The fact that ACE learns insights from rollouts helps us move toward self-managed memory, but the update rules and the overall workflow are still handcrafted. To move toward a more self-improving loop, Meta Context Engineering (MCE; Ye et al. 2026) separates the mechanism (how to manage context) from the artifact content (what is in context), running skill evolution at the meta-optimization level and context optimization at the base level.

An MCE skill $s \in \mathcal{S}$ defines a context function $c_s=(\rho_s,F_s)$ and maps an input $x$ to context $c = F_s(x;\rho_s)$, where:

• $\rho_s = {\rho_1,\dots,\rho_m}$ are static components (prompts, knowledge bases, code libraries).
• $F_s = {F_1,\dots,F_k}$ are dynamic operators (search, selection, filtering, formatting).

The bi-level optimization is to find the best context $c_s^*$ given skill $s$ on the training data, while the outer loop finds the optimal skill that provides the best performance on the validation set:

$$ \text{Inner: }c_s^*=\arg\max_{c_s}J_\text{train}(c_s;s)\quad \text{Outer: }s^*=\arg\max_{s\in\mathcal{S}}J_\text{val}(c_s^*) $$

The skill database tracks the history of previous skills, context functions and eval metrics $\mathcal{H}{k-1} = {(s_i,c_i,J_i^\text{train}, J_i^\text{val})}{i=1}^{k-1}$. A meta-level agent performs agentic crossover over prior skills to create a new skill given a task $\tau$: $s_k=\text{crossover}(\tau,\mathcal{H}_{k-1})$.

Then a base-level context engineer executes the skill $s_k$ and learns the context function from rollout feedback $\mathcal{R}k$, guided by the current skill: $c_k=\text{engineer}(\tau,s_k;c{k-1}^*,\mathcal{R}_k)$.

The framework of Meta Context Engineering (MCE): meta-level skill evolution searches over context-management mechanisms, while the base level optimizes the task context. (Image source: Ye et al. 2026)

MCE does not enforce a heuristic rule for how to structure context as ACE does. It uses free-form skills to store the most important knowledge for a task, and evolves the skill and the skill-conditioned context iteratively together. Implementation-wise, a context function $c$ is instantiated as a collection of files in a dedicated directory, including both static (skill.md) and dynamic (context and data rollouts) components. Both meta-level and base-level optimization are executed in agentic coding envs with a standard tool set,

$$\mathcal{T}={\texttt{Read},\texttt{Write},\texttt{Edit},\texttt{Bash},\texttt{Glob},\texttt{Grep},\texttt{TodoWrite}}$$

Meta-Harness (Lee et al. 2026) moves another level deeper: the optimized object is the code that determines and optimizes what information should be stored, retrieved, and presented to the model. “Meta-” in its name means it is a harness for optimizing harnesses.

The Meta-Harness outer-loop optimization algorithm. (Image source: Lee et al. 2026)

The proposer for creating a new harness is itself a coding agent and the final output is a collection of harness candidates on the Pareto frontier.

• The entire execution history is accessible via a file system, and thus the coding agent uses commands like grep or cat to read through it instead of shoveling everything into a single prompt context.
• The proposed harness is a dictionary in the file system containing its own source code, scores, rollout trajectories, and state updates.
• The mete-harness loop iteratively creates new harnesses, and only qualified ones are kept.

The performance of Meta-Harness on (Left) text classification with a small number of iterations and (Right) TerminalBench-2. Note that the search in the TerminalBench-2 experiment is initialized from Terminus-KIRA and Terminus-2, two very strong harnesses. (Image source: Lee et al. 2026)

Still, the important lesson is clear: once harness design becomes an executable search space, a strong coding agent can exploit the same design space human engineers use.

Workflow Design

Workflow design in harness engineering can be handcrafted by domain experts. Taking auto-research as an example, various frameworks have been proposed and tested. The AI Scientist system (Lu et al. 2026) builds a pipeline to propose research ideas, write code, run experiments, analyze results, write a manuscript, and perform peer review. Meng et al. (2026) make verifiability the central design constraint in ScientistOne, where every claim (citation, numerical, methodological, conclusion) must trace to an evidence source and is audited by Chain-of-Evidence checks.

AI Scientist pipeline for idea generation, experimentation, paper writing, and review. (Image source: Lu et al. 2026)

The Autodata agent (Kulikov et al. 2026) is designed to work as a data scientist for generating training and evaluation data. The main agent manages a challenger that proposes problems, a weak solver, a strong solver, and a verifier/judge, aiming to synthesize data at the “just right” level of difficulty, meaning that the strong solver succeeds but the weak solver fails.

In Autodata, the challenger prompt is updated iteratively according to feedback from the solvers and verifier. The limitation here is that synthesized tasks are used to fine-tune weak solvers but not strong solvers; if the loop cannot iteratively improve the strong model, it is more like indirect distillation over a generated prompt distribution, with less RSI flavor.

Autodata agentic workflow design for generating synthetic training and evaluation data around challenger, solver, and verifier roles. (Image source: Kulikov et al. 2026)

The design space for workflow is enormous, and naturally we can think of workflow design as a search problem, and therefore we should be able to find good solutions by algorithms rather than only manually craft them. Following this direction, Automated Design of Agentic Systems (ADAS; Hu et al. 2025) formulates agent design itself as an optimization problem, “meta-agent search” where a meta-agent proposes new designs of agentic workflows.

1. Initialize an archive of agentic workflows with simple agents such as CoT and self-refine.

2. Ask a meta-agent to program new agents, all in code, inspired by existing solutions in the archive.

   • The meta-agent first generates a high-level description of the new workflow, and then implements it in code.
   • The draft program then goes through two self-refine steps (i.e. ask the model to provide feedback and then ask the same model to refine the previously generated outputs based on the feedback; Madaan et al. 2023) by the meta-agent to check its novelty.

3. Evaluate each new candidate and add successful ones back to the archive.

4. Repeat steps 2-3 until the maximum iteration count is reached.

Illustration of Automated Design of Agentic Systems (ADAS).
(Image source: Hu et al. 2025)

AFlow (Zhang et al. 2025) represents an agentic workflow as a graph, where nodes represent LLM-invoking actions and edges implement logical operations in code. The workflow optimization relies on MCTS (Monte Carlo Tree Search):

1. Initialize the starting workflow $W_0$ in the tree with a template.
2. Select a workflow node using a soft mixture of score and uniform exploration.
3. Expand it by asking an LLM to produce a modified workflow conditioned on its evaluation performance.
4. Execute and evaluate the new workflow.
5. Add it back to the tree if the new workflow shows improvement within a budget of $N$ rounds.
6. Repeat steps 2-5 and stop when the top-$k$ average score plateaus or hit the budget.

AFlow optimization process over a tree of workflow candidates. (Image source: Zhang et al. 2025)

Experiments of AFlow in QA, code, and math tasks showed decent improvement of AFlow over manually designed workflows and ADAS.

AFlow experiments in comparison to manual methods and ADAS. (Image source: Zhang et al. 2025)

Self-Improving Harness

Either context engineering or workflow design is only one part of a harness. We need to search through the entire design space and optimize context-management logic, workflow, permissions, and many other harness components together. As we have seen in work like Meta-Harness, ADAS, and AFlow, ✨code✨ is a universal language for defining programs and systems. In simple words, a harness is code that programs how prompts, tool calls, subagents, control flow, memory, and workflow logic work together. If an LLM can optimize the code that executes agents, it can access a much larger design space than hand-written prompts.

Self-Taught Optimizer (STOP; Zelikman et al. 2023) is one of the early examples of recursive scaffolding improvement. A seed improver $I_0$ at step $t=0$ takes an initial solution $s$, a utility function $u$, and a black-box language model $M$, and returns an improved solution $s’$, that is, $s’ = I(u, s; M)$. The goal of STOP is not directly to improve $s$ but to improve the improver $I$ itself.

First, let’s define the meta-utility as the average utility of a given improver function $I$ over a collection of downstream tasks $\mathcal{D}$:

$$ \hat{u}(I) \triangleq \frac{1}{\vert\mathcal{D}\vert}\mathbb{E}_{(u,s)\sim \mathcal{D}}[u(I(u,s; M))] $$

Because improving the improver function is an optimization problem itself, we can recursively get a new version of $I_t$ based on $I_{t-1}$’s performance measured by meta-utility via a self-improvement update:

$$ I_t=I_{t-1}(\hat{u},I_{t-1};M) $$

Algorithm of Self-Taught Optimizer (STOP). (Image source: Zelikman et al. 2023)

In their experiments, the improved improver discovered various strategies, such as genetic algorithms, decomposing and improving parts, multi-armed prompt bandits, simulated annealing, varying temperature, and beam/tree search. This is analogous to how a harness workflow can be represented as an object for optimization.

Examples of self-improvement strategies discovered by STOP. (Image source: Zelikman et al. 2023)

A cautionary result in Zelikman et al. (2023)’s findings is that STOP improved mean downstream performance across iterations with GPT-4 but degraded with weaker models like GPT-3.5 and Mixtral. Recursive structure alone is not enough. The base model must be capable enough to improve the mechanism. This implies that harness improvement enables better deployment of the model but intelligence is still the core.

Lin et al. (2026) investigated the dependency of harness evolution on model capabilities in more details. They disentangled two axes: (1) harness-updating refers to the capability of producing useful harness edits and (2) harness-benefit denotes the capability of utilizing the updated harness, to achieve better task solving. Interestingly a range of model of different sizes and core intelligence, from Qwen3.5-9B to Claude Opus 4.6, were observed in their experiments to show similar harness updating capability; the 9B harness proposer/evolver is able to write a skill procedurally isomorphic to Opus. To best utilize a harness, a model needs to invoke skills/tools correctly and timely and be good at long-horizon instruction following.

Main results: (A) harness updating capability is measured flat across a range of models from Qwen2-32B to Opus 4.6; (B) harness benefit capability is non-monotonic where middle tier models benefit the most. (Image source: Lin et al. 2026)

A more recent work, Self-Harness (Zhang et al. 2026), relies on LLM agents to improve their own harness via a propose-evaluate-accept loop.

Self-Harness uses a loop of weakness mining, bounded harness proposal, and validation to update a harness. (Image source: Zhang et al. 2026)

The loop in Self-Harness has three stages:

1. Weakness mining: cluster failures into verifier-grounded failure patterns.

   • The current harness $h_t$ is used to evaluate on tasks and execution traces are collected for analysis.
   • Note that two runs can share the same verifier outcome in the error logs on the surface, such as timeout or missing artifact, while having different causal mechanisms. Therefore we need a failure record of rich information, containing the terminal verifier-level cause, the causal status of the relevant agent behavior, and the abstract agent mechanism exposed by the trace, to uncover the root causes.

2. Harness proposal: propose bounded harness edits based on mined failure patterns.

   • The same model is invoked under $h_t$ as a proposer.
   • The model is provided with a bounded proposal context: (1) the editable surfaces of the current harness, (2) the verifier-grounded failure patterns from the evaluation system, (3) records of passing behaviors that should be preserved, and (4) summaries of previously attempted edits.
   • Harness edits should prefer recurrent error patterns that are addressable (e.g. not task-specific difficulty) and can be resolved by narrow changes.
   • Harness edit candidates should be distinct and diverse.

3. Proposal validation: validate and merge qualified edits to create a new harness $h_{t+1}$.

   • Candidate edits are evaluated by regression tests on held-in $D_\text{in}$ (for testing whether the weakness is resolved) and held-out $D_\text{out}$ (for checking whether other unknown issues were introduced) splits.
   • Candidates are accepted only if they have no regression on both held-in and held-out data.
   • Accepted candidates are merged to update the harness to $h_{t+1}$, while rejected candidates are logged without changing the active harness.

When running MiniMax M2.5, Qwen3.5-35B-A3B, and GLM-5 on Terminal-Bench-2, Self-Harness was shown to learn model-specific harness instructions that target at different weaknesses of different base models and improve held-out pass rates.

Self-harness type of work does raise my concerns that if a program is allowed to edit the OS system, abstraction boundaries are broken. The editable surface needs to be properly designed and the permission control and security layers need to live outside this loop. All the challenges around reward hacking still remain.

Agentic Harness Engineering (AHE; Lin et al. 2026) see the bottlenecks of harness evolution are around observability—that is, when a rollout fails, we need to know which component is responsible for that and every edit should be grounded by evidence.

The framework creates a closed loop with 3 observability pillars:

1. Component observability: every editable harness component has a representation in the file system so the action space is explicit and tracable.

   • A harness contains 7 components: system prompt, tool description, tool implementation, middleware, skill, sub-agent configuration, and long-term memory.
   • Each failure pattern is mapped to one component so the edit can be more targeted.

2. Experience observability: analysize and summarize a large amount of raw trajectories into a hierarchy of evidence and failure patterns.

   • Each harness generates $k$ traces.
   • Use an agent (“Agent debugger”) to analysis the trajectories each stored in one file and generate per-task analysis report on the root cause for the failure or success.
   • All the per-task reports are aggregated into a benchmark overview for the next step, and raw traces can be accessed if needed. This layered access structure is more token efficient.

3. Decision observability: every edit is paired with a prediction for the next round to validate.

   • An agent (“Evolve agent”) reads the repo and decides which component to edit, and then produces the edit and the reasoning behind it.

   • Every edit is a file-level, falsifiable claim and can be verified in the next round, under two constraints:

     • (1) Edits are only applied to the harness workspace. the runs directory, tracer, verifier, and LLM configuration are read-only, which disables a set of reward hacking (e.g disabling the verifier, swapping the model, or raising the reasoning budget) and thus it can keep every recorded gain attributable to harness edits.
     • (2) Edits are evidence-driven, with a manifesto entry: the failure evidence’s name, the inferred root cause, the targeted fix, and a predicted impact comprising both expected fixes and at-risk regressions.

On Terminal-Bench-2, AHE achieved better than human-designed harness (OpenCode, Terminus-2, Codex) except for Hard tier and a few other self-evolve baselines (ACE, TF-GRPO). The same frozen harness, without further evolving, transfers to SWE-bench-verified, indicating that the evolved harness is able to encode engineering experience into harness components rather than doing benchmark-specific optimization.

Evolutionary Search

Evolutionary search is an optimization method inspired by natural selection (see my old post on evolutionary algorithm). It evolves a population of solutions by mutating them and only keeping those with high “fitness” in the crowd. Evolutionary search comes in handy when (1) the search space is extensive or weirdly shaped; and (2) it is hard to optimize directly with gradients but easy to evaluate solutions. Harness search seems to be a good fit here.

Evolutionary search has been used in prompt engineering in the past studies. Promptbreeder (Fernando et al. 2023) optimizes task-specific prompts through a rich set of mutation operations, and interestingly the mutation prompts (i.e. instructions to an LLM to mutate a task prompt) are themselves also improved through evolution. GEPA (Agrawal et al. 2025) combines reflection-based prompting with evolutionary search and uses natural language reflection over trajectories of trial and error to propose prompt updates.

Novikov et al. (2025) introduced AlphaEvolve as a coding-agent evolutionary search system, which stores a pool of candidate programs and prompts frozen LLMs to generate diffs for improvement. As the system repeatedly evaluates child programs and keeps successful ones, it discovers better solutions in time.

How AlphaEvolve works. (Image source: Novikov et al. 2025)

A few details matter in the design of AlphaEvolve:

• The prompt includes parent programs, results, instructions, and sometimes meta information.
• The coding agent has access to the full repo, but code regions for improvement are explicitly marked with # EVOLVE-BLOCK-START and # EVOLVE-BLOCK-END.
• Meta-prompt co-evolves with instructions and context as suggested by LLM, in a similar way as how we evolve solution programs.

Ablations show the evolution procedure, context in prompts, meta-prompts, full-file evolution and the use of stronger LLMs.

Ablations show the value of everal designs in AlphaEvolve. (Image source: Novikov et al. 2025)

Recent variants such as ThetaEvolve (Wang et al. 2025) combines evolutionary search with RL and in-context learning, and DemoEvolve (Che, et al. 2026) augments the self-rollout archive with human expert demonstrations as reference experience for harness-level diagnosis and editing. ShinkaEvolve (Lange et al. 2025), on the other hand, introduced three new components to improve LLM sampling efficiency:

• More sample-efficient exploration by designing parent sampling to balance performance rank and offspring count.
• Code-novelty rejection sampling by discarding candidates that are too similar to the existing population based on embedding-based cosine similarity.
• Identifying good patterns in successful solutions in a meta-scratchpad to guide future mutation.

Unlike the methods above, which focus on solution improvement, Darwin Gödel Machine (DGM; Zhang et al. 2025) explicitly targets the evolution of an editable harness-code repository with an LLM-based coding agent. Precisely, this agent is allowed to modify its own harness. A follow-up work on Hyperagents (Zhang et al. 2026) introduced a meta-agent to control how to modify existing task agents to create new ones.

1. Start with one coding agent in the pool.
2. In each iteration, pick one parent with a probability proportional to its performance and inversely to the number of children it has, to modify and branch off to produce new agents.
3. The selected parent agent examines its own benchmark evaluation log and then proposes improvements to its own harness codebase to generate a new version of the coding agent. Code editing is implemented with two basic tools: (1) bash (args: <bash_command>) and (2) editor (args: view/create/edit <file_path>).
4. New coding agents are evaluated, and only those with sufficiently high performance are added back into the pool.
5. Repeat steps 2-4 until some stop criteria hit.

DGM is harness evolution under a fixed model. In experiments with Claude 3.5 Sonnet as the base LLM and simple initial harness configs, the DGM-discovered agents are comparable to or outperform handcrafted agents on SWE-bench Verified (20% to 50%) and Polyglot (14.2% to 30.7%).

This family of methods works well when candidate solutions are automatically evaluable and candidate fitness is easy to quantify, such as matrix multiplication, GPU kernel optimization, algorithm contests, datacenter scheduling. It struggles with domains where evaluation is slow, ambiguous, or mostly heuristic-based. The compute efficiency and effectiveness of evolution are also concerns.

Joint Optimization with Model Weights

Harness evolution changes the non-parametric system around the model. To enable full self-improvement, the model can totally be allowed to update its own weights at the same time. The weight update can be implemented via improvements in the model training pipeline or continual learning at test time. The topic of continual learning is worthy of its own post in the future.

SIA (Hebbar et al. 2026) is an early attempt to combine harness improvement and model-parameter updates in the same optimization loop, with three components in the design:

• Meta-Agent: proposes the initial harness.
• Task-Specific Agent: executes the task.
• Feedback-Agent: chooses whether to update the harness or the model weights based on recent trajectories.

The Feedback-Agent in SIA decides the next iteration type. (Image source: Hebbar et al. 2026)

There are a few confounding choices in SIA’s experiments that make the results hard to interpret. For example, the task-specific agent is much weaker than the models used for the Meta-Agent and Feedback-Agent (gpt-oss-120b vs Claude Sonnet 4.6), and the baselines are too weak to cross-reference cleanly against related methods. I would consider the direction interesting, but the evidence provisional. Yet many challenges, such as training stability and Goodhart effect, still remain open.

Continual Harness (Karten et al. 2026) experimented in long-horizon gameplay setting with harness updating and co-learning a policy model by distilling a strong teacher model’s labels on low-reward trajectories.

Future Challenges

The AI Scientist line of work is a strong demonstration that an expert-designed harness can coordinate a large portion of auto-research loop, experimented in the form of writing research papers. But paper production is not identical to scientific discovery. A system can write a plausible manuscript while still having fabricated citations, implementation drift, or weak experimental results.

Trehan & Chopra (2026) tested whether LLMs can go from a research idea to a paper with minimal scaffolding and basic tools (i.e., read_file, write_file, llm_search, list_files). Each idea had a dedicated workspace where agents could generate and read documents as part of context. They experimented in three domains (world models, multi-agent RL, AI safety & alignment), with each domain containing 45-50 high-quality seed documents to inspire new ideas. Only four ideas were selected by human experts to run through the full pipeline, and only one was fully executed into a paper. They observed six recurring failure modes in the experiments:

• Bias toward training-data defaults: use old libraries, stale commands, standard formats, or assumptions not grounded in the actual repository or dataset.
• Implementation drift under execution pressure: when implementation becomes technically complex, the model may move toward a common simpler solution rather than the proposed method.
• Memory and context degradation: long-horizon projects lose critical details unless logs are written as persistent artifacts.
• Over-optimism: the model declares success despite noisy or failed experiments, similarly observed as “p-hacking and eureka-ing” pattern by Bubeck et al. (2025) where models can introduce “numerical duct tape” and declare victory when signals are still noise.
• Insufficient domain intelligence: the model lacks tacit craft knowledge, e.g. predicting implementation complexity, judging whether an experimental result is plausible, or knowing which baselines matter.
• Weak scientific taste: experiments may be executable but fail to answer the right question.

Toward full RSI, researchers have made real progress, but several bottlenecks remain.

1. Weak and fuzzy evaluators. Many research claims do not have a fast and precise verifier, and the same is true for many real-world tasks. Current self-improvement loops work best for tasks when evaluation metrics are measurable and objective, similar as how RL works.

Research taste, novelty, and long-term scientific value are much harder to measure. For example, research taste often mixes problem framing, experimental design, and judgment about which surprising results are worth pursuing and which failure cases are worth retries.

2. Context and memory lifecycle. Memory grows as AI agents become more autonomous and independent. A useful harness needs to manage context and memory to complement existing limitation in long-context generation while still maximizing the success of long-horizon tasks. Since humans are able to maintain memory through our life time, I see an anoloy here that context engineering will and should become a core part of intelligence, rather than staying in the software system layer.

3. Negative results. Researchers are incentivized to publish successful results and thus literature is biased toward successes. LLMs trained on a vast amount of data (mostly human created, at least for now, lol) may be bad at deciding when to abandon a hypothesis, report a negative result, or even acknowledge a failure due to the imablance of success vs failure cases in data. A research harness should make failed attempts easy to preserve, as learning from failure is the best way to trim down the task search space.

4. Diversity collapse. Evolutionary and RL loops tend to exploit known high-reward patterns. We need mechanisms to prevent the population from collapsing into variants of the same solution. This is especially critical for open-ended research, where the best path may initially look worse under the current evaluator.

5. Reward hacking. A self-improvement loop optimizes whatever signal it is given. If the reward comes from unit tests, the agent may overfit to tests; if it comes from a judge model, it may learn reward hacking tricks specific to this judge; if it comes from benchmark scores, it may exploit benchmark artifacts.

The evaluator and permission control should likely sit outside the loop that evolves harness, with held-out tests, trace audits, and human review at decision points that matter—how much oversight can be scaled up and automated remains an open research area.

6. Long-term success. An extrinsic loop of optimization works on rewards outside of individual rollouts that we can simulate in training sandbox.

Take coding agent as an example. Coding agents have already increased daily productivity in software engineering, but many optimization goals are still too short-term. It can often complete the task at hand, but less obvious how it should protect the long-term health of a repo collectively maintained by hundreds or thousands of engineers. Standard sandbox-based RLVR-style training rarely captures maintainability, ownership boundaries, migration cost, backwards compatibility, or future debugging burden.

7. The role of humans. Humans should move up the stack, not be removed from the loop, meaning that human should provide oversight at the right time, at the right abstraction level and our system design should consider when and how to set up such touch points.

Many challenges listed above need human’s feedback and steering. After all, we are building the technology for better future of humanity, not other way around.

Citation

Please cite this work as:

Weng, Lilian. “Harness Engineering for Self-Improvement”. Lil’Log (Jul 2026). https://lilianweng.github.io/posts/2026-07-04-harness/

Or use the BibTeX citation:

@article{weng2026harness,
  title = {Harness Engineering for Self-Improvement},
  author = {Weng, Lilian},
  journal = {lilianweng.github.io},
  year = {2026},
  month = {July},
  url = "https://lilianweng.github.io/posts/2026-07-04-harness/"
}

Appendix: Some useful benchmarks

• PaperBench: replicate 20 ICML 2024 Spotlight and Oral papers from scratch, including understanding paper contributions, developing a codebase, and successfully executing experiments.

  • Each replication task is decomposed into smaller, individually gradable tasks.
  • 8,316 rubrics in total, co-developed with the paper authors.
  • The best model at the time (Claude 3.5 Sonnet, ~21%) does not outperform ML PhDs.
  • Includes PaperBench, PaperBench Code-Dev (a lighter version), and JudgeEval.

• CORE-Bench: evaluate computational reproducibility of published research.

  • 270 tasks based on 90 scientific papers across computer science, social science, and medicine.
  • Tasks involve reproducing results from provided code and data.
  • Includes multiple difficulty levels and both language-only and vision-language tasks.
  • The best reported agent at the time (GPT-4o and GPT-4o-mini) achieved only 21% accuracy on the hardest task.

• ScienceAgentBench: evaluate LLM agents for data-driven scientific discovery.

  • Extracts 102 tasks from 44 peer-reviewed publications in four disciplines (math, chemistry, biology, geography).
  • Covers basic data-science tasks in these domains: data processing, model development, data analysis, and information visualization.

• RE-Bench: evaluate frontier AI agents on realistic ML research-engineering envs against human experts.

  • 7 challenging, open-ended ML research-engineering environments.
  • Each environment = (scoring function, starting solution, reference solution); each can be run with 8 or fewer H100 GPUs.
  • Examples: optimize a kernel, run a scaling-law experiment, fix an embedding, fine-tune GPT-2 for QA, etc.
  • Includes data from 71 eight-hour attempts by 61 distinct human experts.
  • Human experts achieved non-zero score in 82% of 8-hour attempts; 24% matched or exceeded strong reference solutions.
  • Best AI agents scored 4× higher than humans at a 2-hour budget, but humans had better returns to longer budgets and exceeded agents at 8-hour and 32-hour settings.

• MLE-bench: evaluate ML engineering agents on offline Kaggle competitions.

  • Contains 75 ML-engineering competitions curated from Kaggle.
  • Tests training models, preparing datasets, running experiments, and submitting predictions to grading scripts.
  • Uses Kaggle public leaderboards as human baselines.
  • Best setup in the paper, o1-preview with AIDE scaffolding, reached at least Kaggle bronze-medal level in 16.9% of competitions.
  • Includes resource-scaling and contamination analyses.

• KernelBench: evaluate correctness and speed for generated GPU kernels.

  • 250 PyTorch tasks to evaluate whether LLM can write fast and correct kernels.
  • The evaluation metric fast_p = the percentage of generated kernels that are correct and faster than baseline.

References

[1] Good, I. J. “Speculations Concerning the First Ultraintelligent Machine.” Advances in Computers, 6:31–88, 1965.

[2] Yudkowsky, Eliezer. “Recursive Self-Improvement.” LessWrong, 2008.

[3] Choi, et al. “Anchored Self-Play for Code Repair.” ICML 2026.

[4] Zhao, et al. “Absolute Zero: Reinforced Self-play Reasoning with Zero Data.” arXiv preprint arXiv:2505.03335, 2025.

[5] Yuan, et al. “Self-Rewarding Language Models.” arXiv preprint arXiv:2401.10020, 2024.

[6] Chen, et al. “Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models.” ICML 2024.

[7] Zhang, et al. “Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models.” ICLR 2026.

[8] Ye, et al. “Meta Context Engineering via Agentic Skill Evolution.” arXiv preprint arXiv:2601.21557, 2026.

[9] Lee, et al. “Meta-Harness: End-to-End Optimization of Model Harnesses.” arXiv preprint arXiv:2603.28052, 2026.

[10] Lu, et al. “Towards end-to-end automation of AI research.” Nature, 651:914–919, 2026.

[11] Meng, et al. “ScientistOne: Towards Human-Level Autonomous Research via Chain-of-Evidence.” arXiv preprint arXiv:2605.26340, 2026.

[12] Kulikov, et al. “Autodata: An agentic data scientist to create high quality synthetic data.” arXiv preprint arXiv:2606.25996, 2026.

[13] Hu, Lu, and Clune. “Automated Design of Agentic Systems.” ICLR 2025.

[14] Madaan, et al. “Self-Refine: Iterative Refinement with Self-Feedback.” NeurIPS 2023.

[15] Zhang, et al. “AFlow: Automating Agentic Workflow Generation.” ICLR 2025.

[16] Zelikman, et al. “Self-Taught Optimizer (STOP): Recursively Self-Improving Code Generation.” COLM 2024.

[17] Zhang, et al. “Self-Harness: Harnesses That Improve Themselves.” arXiv preprint arXiv:2606.09498, 2026.

[18] Fernando, et al. “Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution.” arXiv preprint arXiv:2309.16797, 2023.

[19] Agrawal, A. et al. “GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning.” arXiv preprint arXiv:2507.19457, 2025.

[20] Novikov, et al. “AlphaEvolve: A coding agent for scientific and algorithmic discovery.” arXiv preprint arXiv:2506.13131, 2025.

[21] Lange, Imajuku, and Cetin. “ShinkaEvolve: Towards Open-Ended And Sample-Efficient Program Evolution.” arXiv preprint arXiv:2509.19349, 2025.

[22] Wang, et al. “ThetaEvolve: Test-time Learning on Open Problems.” arXiv preprint arXiv:2511.23473, 2025.

[23] Zhang, et al. “Darwin Gödel Machine: Open-Ended Evolution of Self-Improving Agents.” arXiv preprint arXiv:2505.22954, 2025.

[24] Zhang, et al. “Hyperagents.” arXiv preprint arXiv:2603.19461, 2026.

[25] Yuksekgonul, et al. “Learning to Discover at Test Time.” arXiv preprint arXiv:2601.16175, 2026.

[26] Riaz, et al. “Epistemic Uncertainty for Test-Time Discovery.” arXiv preprint arXiv:2605.11328, 2026.

[27] Hebbar, et al. “SIA: Self Improving AI with Harness & Weight Updates.” arXiv preprint arXiv:2605.27276, 2026.

[28] Trehan and Chopra. “Why LLMs Aren’t Scientists Yet: Lessons from Four Autonomous Research Attempts.” arXiv preprint arXiv:2601.03315, 2026.

[29] Bubeck, et al. “Early science acceleration experiments with GPT-5.” arXiv preprint arXiv:2511.16072, 2025.

[30] Starace, et al. “PaperBench: Evaluating AI’s Ability to Replicate AI Research.” ICML 2025.

[31] Wijk, et al. “RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts.” ICML 2025.

[32] Chan, et al. “MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering.” arXiv preprint arXiv:2410.07095, 2024.

[33] Chen, et al. “ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery.” ICLR 2025.

[34] Siegel, et al. “CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark.” TMLR 2024.

[35] Ouyang, et al. “KernelBench: Can LLMs Write Efficient GPU Kernels?” arXiv preprint arXiv:2502.10517, 2025.

[36] Lin, et al. “Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents.” arXiv preprint arXiv:2605.30621, 2026.

[37] Lin, et al. “Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses.” arXiv preprint arXiv:2604.25850, 2026.

[38] Karten, et al. “Continual Harness: Online Adaptation for Self-Improving Foundation Agents.” arXiv preprint arXiv:2605.09998, 2026.

[39] Che, et al. “DemoEvolve: Overcoming Sparse Feedback in Agentic Harness Evolution with Demonstrations.” arXiv preprint arXiv:2605.24539, 2026.

On August 3, 2006, I uploaded the first version of pandoc to my website, releasing it under the free GPL license. Pandoc 0.1 consisted of about 3000 lines of Haskell code, with no dependencies aside from GHC’s standard library. It could convert Markdown, reStructuredText, HTML, and LaTeX documents into any of these formats, plus RTF or S5. I had no idea at the time that this would just be the first of over two hundred releases over the next twenty years; that the project would become the most popular program written in Haskell; that I would spend countless hours on bug-fixes, improvement, and project management; that I would collaborate with programmers in many other countries; that pandoc would come to support over fifty document formats; that it would allow automatic generation of citations and bibliographies; that it would become integrated into academic writing tools like Quarto and Jupyter Notebook; that it would be installed on millions of computers around the world.

How did this happen? I want to take advantage of pandoc’s birthday to tell the story of the project, as best I can remember it.

Prehistory

People often ask: Why is pandoc written in Haskell? There could have been good answers to this question: Haskell is a very good language for writing this kind of application. But in fact, I didn’t decide to write a document converter, then decide to use Haskell for it. I decided to use Haskell, and then decided to write a document converter in it.

I had heard about Haskell from the blog of a philosophical logician friend, Greg Restall. Of an introductory book on Haskell, he said: “I’m glad that this wasn’t the textbook in my introductory computer science course, long ago in 1986. If it were, I may have fallen in love with computing and never become a philosopher” (consequently.org).

Intrigued by this (and not heeding Restall’s warning about the potential effects on my future philosophical productivity), I read A Gentle Introduction to Haskell to get a basic understanding of the language. But the only way to really learn a programming language is to write something in it. I saw that Haskell was good for writing parsers and compilers, and it came with a really nice parser combinator library (parsec), so I decided to write a Markdown parser.

At that time, there were implementations of Markdown in Perl, Python, Ruby, and PHP; they all transformed Markdown directly to HTML through a sequence of regex transformations. Pandoc took a different approach. It parsed the Markdown using parser combinators and produced a real abstract syntax tree (AST), which it could then render to HTML or another format. This was a more reliable architecture (avoiding many quirks of the regex versions). It was also a more extensible one: by writing N parsers (“readers”) and M renderers (“writers”), one could support N × M conversions. Soon I added a reader for reStructuredText, because I kept a lot of my lecture notes and handouts in that format. And I added a writer for LaTeX, because I wanted to be able to produce PDFs. Then I added a writer for Markdown, so I could start to convert my reStructuredText notes to Markdown. And from there the project just snowballed.

Thus, a project that started out as nothing more than the product of procrastination was nurtured by the joy of writing in Haskell and by its increasing usefulness for my own academic work.

First releases (2006–8)

In August 3, 2006, I decided to make the source code available on my website. By now pandoc supported HTML, LaTeX, RST, and Markdown as input and output formats, and RTF as an output format; also PDF via LaTeX.

The first release

I made no attempts to advertise the project, other than emailing two friends. This was before social media (which I’ve never used anyway), before GitHub, and before Hackage, the Haskell package repository. But apparently some people stumbled across it on my website and started using it. In October I was contacted by a Turkish developer, Recai Oktaş, who was trying to get certified as a Debian developer and wanted to package pandoc for Debian linux. So I worked with him to do that. This was a great learning experience for me and it greatly increased the visibility of the project.

During 2007, I continued to improve pandoc, largely guided by my own needs. Version 0.3 added the DocBook writer and the now-standard syntax for footnotes in Markdown. Version 0.4 added support for Markdown tables, definition lists, super/subscript, strikeout, and enhanced ordered lists, as well as writers for groff man pages and ConTeXt. This was the first release to go on the Hackage Haskell package repository, which was started in 2007. The Hackage archive and the new cabal-install tool, which automatically resolved and fetched dependencies, opened up the possibility of depending on external packages.

Pandoc 1 (2008–17)

Pandoc 1.0 was released in September 2008, with new writers for MediaWiki, GNU Texinfo (contributed by Peter Wang), OpenDocument (contributed by Andrea Rossato), ODT, and delimited code blocks (now called “fenced”) with automatic syntax highlighting. Support for ODT requires the ability to create a zip archive, and at the time there was no Haskell package for this, so I created one (zip-archive), using the excellent binary package for binary parsing and serialization. Support for syntax highlighting required a syntax highlighting library, which also did not exist in Haskell. For this, I wrote highlighting-kate, which parsed the XML syntax definitions used by the Kate text editor and turned them into Haskell code highlighters. This allowed pandoc to support a large number of syntaxes right off the bat. This version also contained support for automatic generation of citations and a bibliography using CSL style, using Andrea Rossato’s citeproc-hs library.

Throughout this period, I was involved in discussions with other Markdown implementers on the (now defunct) markdown-discuss mailing list. The syntax for delimited code blocks, which pandoc supported long before GitHub popularized fenced code blocks, was worked out in collaboration with Michel Fortin, the maintainer of PHP Markdown Extra. I took care when adding extensions to pandoc’s Markdown to pay attention to prior art, for example copying PHP Markdown Extra’s definition list syntax. During this period, I also became aware of many ambiguities in Markdown’s syntax—a situation I would later try to improve in the commonmark project.

The next big change to pandoc came in version 1.4 (released in January 2010), which introduced a flexible template system, replacing hard-coded headers and making pandoc’s output much more customizable.

In 2010, we moved from Google Code to GitHub, which would do even more to increase the visibility of the project. Further releases in 2010 and 2011 added support for EPUB output, Org-mode output (due to Puneeth Chaganti), and Textile input (due to Paul Rivier). Pandoc also gained support for converting TeX math to MathML (for DocBook or HTML), via my texmath library.

Pandoc 1.9, published in 2012, finally made it possible to produce Word docx output. To handle the equations properly, I added support for Word’s OMML format to texmath. This release also added an AsciiDoc writer and support for Beamer and DZSlides, and in 1.9.3 we gained a DocBook reader (with contributions from Mauro Bieg, who became a long-time contributor).

In 2013, we focused on several features that made pandoc much more flexible and customizable. The first was a fine-grained system of Markdown “extensions,” allowing support for the many variants of Markdown that were then proliferating. The second was the ability to include YAML metadata blocks in Markdown, with arbitrary structured fields that populate template variables. The third was the ability to create custom writers in Lua, allowing ad hoc output formats to be supported by users. The fourth was the introduction of JSON filters—user-created programs that transform a JSON serialization of the pandoc AST, allowing the document to be customized between the parsing phase and the rendering phase. Citation processing was moved from the core of pandoc into an external filter, pandoc-citeproc.

This era saw the addition of reveal.js, EPUB v3, DokuWiki, and FictionBook2 output; OPML input and output; and Haddock and MediaWiki input. Notable contributors include David Lazar (Haddock) and Sergey Astanin (FictionBook2).

The year 2014 saw the arrival of three new contributors who would go on to make many contributions to the project. Albert Krewinkel added support for Org-mode input; Jesse Rosenthal added a Word docx reader (complete with track-changes awareness); and Matthew Pickering (at the time a student at Oxford whom I “advised” as a Google Summer of Code Student) added support for EPUB and Txt2Tags as input formats. Supporting EPUB input required being able to convert MathML equations, so Pickering also worked on texmath. We were in very different time zones, and I remember waking up every morning to find all the work Pickering had done during the night. (Pickering has gone on to become one of the core maintainers of the ghc compiler.) All of these contributions were released in pandoc 1.13, together with Clare Macrae’s DokuWiki writer.

Since 2012, I had been involved in a working group that aimed to produce an unambiguous specification of Markdown’s syntax, initiated by Jeff Atwood and including representatives from GitHub, Reddit, and Stack Overflow. The group held intensive discussions in 2012, which petered out in 2013. I still believed in the project and didn’t want to let the work we’d done go to waste, so I sat down in August 2014, before the academic year began, and wrote up a spec for Markdown, as well as parsers in JavaScript and C. I sent the draft spec to John Gruber for comment and did not get a response, so a few weeks later we posted the spec. At this point, Gruber strongly objected and demanded that we not call the project “Standard Markdown,” so we changed the name to “commonmark.” The project has been a success, in that with a few exceptions, most Markdown processors implement the commonmark spec for their core rules. (Commonmark does not concern itself with extensions.)

Pandoc 1.14 (2015) added support for commonmark and a number of extensions (at first via bindings to the C library libcmark, but later, in 2020, via my Haskell packages commonmark, commonmark-extensions, and commonmark-pandoc). I intend eventually to replace pandoc’s legacy Markdown parser with a commonmark core, but there are still a few key extensions that have not been implemented, so pandoc users must still choose between parsing their documents as markdown (Markdown with pandoc’s extensions) or as gfm or commonmark or commonmark_x (commonmark with a number of extensions). Ironically, although I was the author of the commonmark spec, pandoc still uses a pre-commonmark Markdown parser!

The next year brought some important changes in the pandoc AST, with the addition of image and link attributes, a SoftBreak element (enabling pandoc to preserve line breaks from the original source, or wrap, depending on a command line setting), and a LineBlock element. MarLinn added an ODT reader, Chris Forster added a TEI writer, and Ivo Clarysse added support for DocBook 5.

Pandoc 2 (2017–23)

Pandoc 2.0 (released in 2017) brought some big architectural changes, worked out in collaboration with Jesse Rosenthal. In the past, most of pandoc’s readers (parsers) and writers (renderers) had been “pure” (that is, they had Haskell types that prevented them from having any side effects, including I/O operations). But some formats needed to be able to do I/O for a fully faithful conversion. (For example, reStructuredText has a syntax for including files, so the parser needs to be able to read files; in some other formats, images require explicit sizes, so a renderer has to be able to read image files, perhaps fetching them using HTTP, and determine their sizes.) We designed a system that allowed pandoc readers and writers to run in any instance of the PandocMonad typeclass, and we provided both a pure instance (which could be used for controlled testing, and in situations where we wanted to forbid I/O) and an instance that allowed I/O operations. The system also provided a way to handle images included as resources in formats like docx or EPUB.

The other big change was the introduction of Lua filters: filters running in an embedded Lua interpreter and operating directly on the pandoc AST, requiring no software other than pandoc itself and offering far better performance than JSON filters. This was made possible by the massive efforts of Albert Krewinkel, building on the hslua, a Haskell-Lua bridge library.

In addition, pandoc 2.0 introduced the raw attribute syntax in pandoc’s Markdown, and support for GitHub-flavored Markdown, Emacs Muse (Alexander Krotov), TikiWiki, Vimwiki (Yuchen Pei), Creole (Sascha Wilde), groff ms, and JATS. The old highlighting-kate was replaced by the new skylighting, which offered better performance and more accurate interpretation of KDE syntax definitions. A PowerPoint writer (due to Jesse Rosenthal) soon followed, as well as support for FictionBook2 (Krotov) and man (Yan Pashkovsky and me) as input formats.

In 2018, the project received a generous $100,000 donation from Handshake, which we used over the next five years to give small stipends to the most active maintainers.

In 2019, support for ipynb (Jupyter notebooks) was added, allowing pandoc to be used in data science workflows, and Jira wiki markup was supported as an output format. With pandoc 2.8, it became possible to specify collections of default options using defaults files.

Users had long complained that pandoc’s model of a table was too restrictive, not even supporting row and colspans. After extensive discussion of what was needed in a table format, Christian Despres designed the new types for tables and modified all of the readers and writers to use it (a big job).

At this point pandoc had supported citation resolution for many years, by means of the pandoc-citeproc filter that used Andrea Rossato’s citeproc-hs. This was slow and somewhat buggy, and Rossato had long since disappeared from the scene, so I wrote a Haskell citeproc library from scratch, using just the CSL spec and test cases. Pandoc 2.11 depended on this library and offered far better citation support: faster, more faithful to CSL, and with no need for an external filter. In order to get citations to sort properly, I had to write a another library (unicode-collation) implementing the Unicode Collation algorithm in pure Haskell.

During this era Pandoc came to support conversions between bibliography database formats: BibTeX, BibLaTeX, and CSL JSON, EndNote XML and RIS; conversion from CSV and TSV to pandoc table formats; conversion to Markua; and conversion from RTF. With pandoc 2.15 a --sandbox option was added, which guarantees that pandoc’s parsers and renderers have no I/O side effects. (This was possible because of the PandocMonad abstraction we added back in pandoc 2.0.) With pandoc 2.16.2 it became possible to write custom readers in Lua to complement the custom Lua writers that had been added in 2013. And with pandoc 2.19.1 it became possible to run pandoc as a web server exporting an API.

Pandoc 3 (2023–present)

By 2023, pandoc had become a very big, monolithic project. Some users wanted a leaner program, one that didn’t include a full web server and Lua interpreter. So with the pandoc 3.0 release, we split pandoc into four parts: pandoc remained the Haskell library, pandoc-lua-engine brought the Lua integration, and pandoc-server exposed the library over HTTP as an API. The command-line program, now in the pandoc-cli package, could optionally be compiled without server or Lua support. We also introduced a native Figure element in the AST and a “chunked HTML” writer for multi-chapter HTML books and documentation.

The first versions of Typst, a modern LaTeX competitor with incremental compilation, were released in 2023. I wanted to help the project by providing an easy on- and off-ramp, making it easy for others to try Typst. It turned out that creating a Typst reader for pandoc required implementing an interpreter for a fairly full-featured programming language. The result was the typst package on Hackage. Typst support was added in pandoc 3.1.3.

In 2018 I had published an essay “Beyond Markdown” in which I described the six features of Markdown that I thought had created the most difficulties, both for writing a spec and for implementations, and I explained how I thought these flaws could be fixed in a future Markdown-like light markup syntax. In 2022, I published a syntax description for such a syntax, djot, together with code in Lua, JavaScript and (later) Haskell. Pandoc 3.1.12, published in 2024, added djot as both an input and output format.

Subsequent releases in 2024 and 2025 saw the addition of an ANSI writer for formatted terminal output and a reader for the mdoc and POD formats (all due to Evan Silberman), a reader and writer for an XML representation of the pandoc AST (massifrg), a vimdoc writer (reptee), a PowerPoint reader (Anton Antich), an Excel spreadsheet reader (Anton Antich), and a BBCode writer (reptee), and an AsciiDoc reader (supported by my asciidoc package).

Pandoc 3.9, released in February 2026, included support for compiling pandoc to WASM, which allowed a full-featured version of pandoc to run in the browser. Most of the key work was done by TerrorJack. The GUI interface “pandoc for the people” was designed with the help of Claude Opus.

I still work on pandoc almost every day. Most of this work doesn’t involve the kind of new features or architectural changes I have focused on in this narrative. Mostly it consists in fixing small bugs, making tiny improvements, reviewing issues and pull requests, repairing infrastructure (continuous integration, building releases, code signing, website), improving documentation, and engaging in discussions with maintainers and users.

Statistics

Pandoc currently supports 51 input formats and 76 output formats, thus 3876 distinct conversions (not counting the variants that are possible by adjusting extensions).

Pandoc’s conversions

The four core packages (pandoc, pandoc-lua-engine, pandoc-server, pandoc-cli) consist of 85,684 lines of Haskell code, not including tests. If one includes dependencies that exist mainly for the sake of pandoc (texmath, typst, djot, commonmark, asciidoc, citeproc, and the pandoc/Lua interface packages), this number approximately doubles.

On GitHub, 7346 issues have been resolved.

Over 600 people have contributed to pandoc over the years. The top twenty contributors (measured by numbers of source lines changed) are:

Contributor Lines changed Years active John MacFarlane 372,317 2006– Albert Krewinkel 77,136 2014– Jesse Rosenthal 39,664 2014– Christian Despres 15,314 2019–2021 Alexander Krotov 8,657 2017–2019 Matthew Pickering 6,919 2014–2015 MarLinn 4,142 2015 Evan Silberman 3,478 2024– Nikolay Yakimov 3,362 2014–2020 Mauro Bieg 3,044 2012–2020 Emily Bourke 2,196 2021 Yan Pas 2,035 2018 reptee 1,732 2025 Anton Antich 1,552 2025 massifrg 1,171 2025– Nathan Gass 1,011 2010–2011 Tuong Nguyen Manh 801 2022– Joseph C. Sible 767 2020–2024 Clare Macrae 759 2013–2015 Sergey Astanin 718 2011–2012

Here are the twenty contributors who have contributed over the longest spans of time:

Contributor Years active John MacFarlane 2006–2026 Albert Krewinkel 2014–2026 Andrew Dunning 2015–2026 Nikolay Yakimov 2014–2025 Thomas Hodgson 2015–2026 Mauro Bieg 2012–2022 Kolen Cheung 2016–2025 Pablo Rodríguez 2014–2023 Pascal Wagler 2019–2026 Felix Yan 2016–2023 Sergei Trofimovich 2011–2018 Tristano Ajmone 2017–2024 Frerich Raabe 2015–2022 Salim B 2017–2024 Yihui Xie 2014–2020 Sascha Wilde 2017–2023 Jose Luis Duran 2013–2019 Jesse Rosenthal 2014–2020 John Muccigrosso 2016–2022 Jan Tojnar 2020–2026 Brian Leung 2018–2023

Retrospective: the choice of Haskell

As I noted at the beginning, I didn’t choose Haskell because I judged it to be the best language to use for a project like pandoc. But was it?

It’s hard to answer this confidently, because I’m not very familiar with what would now be the most obvious alternative: Rust. But I have created and maintained significant projects in a number of languages, including Pascal, C, Ruby, and JavaScript/TypeScript. I don’t think I would have been able to manage a project like this in my spare time if it had been written in one of these languages.

Haskell has a number of features that have been very helpful in developing pandoc:

• Its algebraic data types give us a very clean, ergonomic representation of a structured document
• Its strong type system, which gives you a compiler error if you don’t combine the types of things in the right way, allows one to make big changes to the program with confidence that you’re not breaking anything; the compiler will show you everything that needs to be changed, and when the code compiles, you are very often done. When working with languages without a strong type system, e.g. Python and JavaScript, the lack of these safeguards always make me afraid to make big changes, especially when I am maintaining code long after I’ve written it.
• Haskell is a pure language; nothing can have side effects that aren’t explicitly allowed for in the types. If you have a pure function, you know it won’t create a file or delete one or make a web request or launch missiles or change a global variable. This is extremely useful for preventing bugs. In pandoc we also use it to give us a really strong guarantee that, when run in sandbox mode, the readers and writers won’t touch the file system.
• The choice of Haskell has also led to a high quality and low volume of contributors (a combination that is good for a project without a lot of resources).

From what I have seen, Rust appears to have many of the good features of Haskell, while producing faster, more memory-efficient, and more compact code. But Haskell still strikes me as more “ergonomic,” better suited to express abstractions, and just closer to the ideal of a language that helps the developer think.

Whither Pandoc

I plan to continue improving pandoc. There are many ways in which it can be improved. But sometimes I wonder how long such a tool will continue to be necessary.

Just as current LLMs can do a very good job translating from one human language to another, they can do a decent job translating from one document format to another. In my small tests, ChatGPT did a good job translating from Markdown to HTML, and a decent (but notably worse) job converting to reStructuredText. My guess is that you could write a document in a light markup language you just had invented, and an LLM could do a decent job guessing your intent and translating it to HTML or another format.

Perhaps, then, in the future, people will no longer have a need for tools like pandoc. As things stand now, though, I think that using pandoc to convert texts has several large advantages over relying on an LLM. The first is ecological; it simply requires far less energy for the same conversion. The second is that pandoc’s output is deterministic; if you convert your text with pandoc, you’ll always get the same result, and you’ll be able to predict what that result is. The third is that, for the moment at least, pandoc’s conversions are going to be more reliable. But that could change in the coming years. Indeed, a time may come when LLMs can produce more reliable conversions than pandoc or anything that works like it.

In designing the commonmark spec, we had the goal of interpreting complex strings in the way that a human would naturally interpret them. This turns out to be quite difficult to achieve: witness the complex rules for emphasis. What we found is that, no matter how complex we made the rules for nested emphasis, it was always possible to come up with cases where the algorithm diverges from the meaning a human would naturally find in the string. In such cases, I would often remark, “until our programs have AI, we are going to have edge cases like this; at some point we have to accept that and stop trying to develop more complex rules.” Interestingly, now we do have tools that can understand (or at least simulate understanding) of the meaning and intent of the text, and can potentially do better at recognizing the formatting intended by the author than any light markup syntax that could be designed.

Whatever the future may bring, I am proud of the 20-year history of this project, which has saved people all over the world countless hours of drudgery. Happy 20th birthday, pandoc!

---

In honor of this occasion, I have produced some pandoc mugs and stickers:

• Coffee mug with conversion diagram
• Sticker with pandoc cartoon
• Sticker with pandoc logo
• Mug with pandoc logo

During the last two weeks of January 1975 Buckminster Fuller gave an extraordinary series of lectures concerning his entire life’s work.

Introduction

These thinking out loud lectures span 42 hours and examine in depth all of Fuller’s major inventions and discoveries from the 1927 Dymaxion house, car and bathroom, through the Wichita House, geodesic domes, and tensegrity structures, as well as the contents of Synergetics. Autobiographical in parts, Fuller recounts his own personal history in the context of the history of science and industrialization.

The stories behind his Dymaxion car, geodesic domes, World Game and integration of science and humanism are lucidly communicated with continuous reference to his synergetic geometry. Permeating the entire series is his unique comprehensive design approach to solving the problems of the world. Some of the topics Fuller covered in this wide ranging discourse include: architecture, design, philosophy, education, mathematics, geometry, cartography, economics, history, structure, industry, housing and engineering.

The “Everything I Know” video series is also available online at archive.org/details/buckminsterfuller.

Un-Edited, Un-Cut

The printed work below is a transcript of those lectures. Painstakingly typed word for word from audiotapes, these transcripts are minimally edited and maximally Fuller. In that vein you will run into unique Bucky-isms: special phrases, terminology, unusual sentence structures, etc. Because of this, as well as the sheer volume of words, we expect you may find places that need editing, refining and improving. Therefore, we invite you to participate! We hope that by your using it as an active resource you can, through your comments, suggestions and feedback, become a participant in the process of annotating, editing, footnoting, updating and illustrating the information it contains. This way it will become progressively more useful to more and more people. The more it is used the more useful it can become! Send us your edits by simply sending us a copy of the page(s) that you think need changes, marked with your suggestions and edits by email on our contact page. We will then make the appropriate adjustments to be integrated and published in the newer versions of the work over time.

We are grateful to make this work available and look forward to its evolution into an evermore useful, refined, and expanded document.

— The Buckminster Fuller Institute

Acknowledgments

We would like to gratefully acknowledge and thank JoAnne Ishimine, whose care and dedication in transcribing the entire 42 hours of the “Everything I Know” series as a volunteer was an inspiration in getting this project off the ground. Her contribution is a striking example of what one individual can do, and in this project she has made a big difference.

We would also like to acknowledge Ed Applewhite for his foresight and commitment in producing the outstanding outline which he prepared while in attendance at the lectures, and for which he has over the years allowed us to include as part of the “Everything I Know” publications.

In addition we would like to thank dedicated volunteers Russell Chu, Jim Morrisett, Robert Orenstein and the many BFI staff members who assisted in various aspects of the preparation of materials.

Copyright

First Edition

Published by the Buckminster Fuller Institute
Contact us for more Information

Copyright © 1997 Estate of R. Buckminster Fuller

Sections

01

Section 1

Everything I know: Section 1 Part 1 We tried to think about the most primitive information we have regarding our extraordinary experience, is that, I think we choose the fact that, all humanity has always been born …

02

Section 11

Everything I Know: Section 11 Part 1 Somebody asked me last night, just at the end if I was going to talk about love. And I said, I’m bound to talk about love. And Andy, one of you, was married, at Christmas …

03

Section 10

Everything I Know: Section 10 Part 1 I went into many angles of the developing of the omni-medium transport and experiences that I had, I did speak to you about the problem of the men who worked for me not daring …

04

Section 9

Everything I Know: Section 9 Part 1 As you know, we’ve been through now about 26 hours, and I’ve been able to keep a picture of what I have said for 26 hours, and the I haven’t used any notes, but …

05

Section 8

Everything I Know: Section 8 Part 1 I made a diagram last summer of the trigonometric functions, and I thought I would complete it for you, and in trying to get it run to your head, because I am a little slower at …

06

Section 7

Everything I Know: Section 7 Part 1 We left off at the experience of witnessing the every sphere in closest packing, changing and becoming a space, and a space becoming a sphere. We’ve been through …

07

Section 6

Everything I Know: Section 6 Part 1 I’ve been covering really very large patterns with you very deliberately, and many people ask me a question about being a comprehensivist, and then being competent. And …

08

Section 5

Everything I know: Section 5 Part 1 At our first meeting I reviewed what I could remember of my conscious input of what it is I am conscious of when I say I am thinking. Remember we came out then with the …

09

Section 4

Everything I know: Section 4 Part 1 You recall that I talked a great deal about Pattern Integrity, and there was one episode in my life that I think really dramatizes the pattern integrity. In 1930, I was asked to …

10

Section 3

Everything I know: Section 3 Part 1 I talked quite a lot last time about technology and the at present very popular viewpoint that technology is something that has been introduced to our life on our planet here by …

11

Section 2

Everything I know: Section 2 Part 1 I think it’s important for all of you to share very intimately with me what I do in the way of conscious disciplining of myself as we meet. I am an experientialist. My …

12

Section 12

Everything I Know: Section 12 Part 1 Cynde gave me a very beautiful drawing and quotation from Cheyenne Indians on it was a marriage in the north and a marriage in the south, and then the east and the west concepts …

How a branch-free loop and byte-space arithmetic let GitHub case-fold every byte of code search at >45 GiB/s on a single core.

Suppose a user searches for café and your corpus contains CAFÉ, or they type straße and you’ve stored STRASSE. To make these count as matches, you need a canonical form that erases case distinctions, so that two strings which differ only in case compare equal. That form is case folding, and it shows up wherever text is matched rather than displayed: search engines, regex (?i) flags, case-insensitive usernames and hostnames.

It’s a basic operation, but at GitHub we run it a lot. Blackbird, GitHub’s code search engine, indexes over 180 million repositories—more than 480TB of source code. Every byte is case-folded before we extract ngrams and build the index, and for every potential query result, another (implicit or explicit) case folding operation is needed to locate matches. At that scale, the speed of even a basic operation starts to matter.

This post is about how we made it fast, and it starts somewhere counterintuitive: the biggest win in the ASCII fast path came from removing an optimization, not adding one. It turns out to be faster to sweep the whole buffer with no branches than to stop early at the first non-ASCII byte. We open-sourced the result as a Rust crate called casefold.

Folding is not lowercasing

It is tempting to reach for str::to_lowercase, but lowercasing and folding are different operations with different goals:

Lowercasing is for display, and it’s locale- and context-sensitive: Greek final sigma lowercases to ς at the end of a word and σ elsewhere, and Turkish I lowercases differently than English I. Case folding is for comparison, and it’s deliberately context-free and locale-independent. The point is a relation that stays stable and symmetric, so that if A folds to match B, B folds to match A in any locale. The Unicode Character Database ships an explicit CaseFolding.txt for exactly that.

The two operations diverge on real characters—ß, İ, final sigma—which is why lowercasing as a stand-in silently produces wrong matches. This crate implements only the simple (1-to-1) folds—statuses C and S in CaseFolding.txt—and not the multi-character “full” folds (ß → ss) or Turkic locale folds (the dotted İ). This isn’t an unusual choice: common tools and regex engines like ripgrep make the same restriction, and being consistent across tools is important.

The counterintuitive core: Don’t stop early

We deal mostly with source code, so the text we fold is overwhelmingly ASCII and making it run at memory speed is the single most important thing we can do. Everything else just has to keep the rare non-ASCII path from spoiling it.

The fold of an ASCII letter is trivial—A..=Z map to a..=z, everything else is unchanged—so the ASCII pass is really just “sweep the buffer, lowercase in place.” Ask any LLM for it and you might get something like this:

let bytes = s.as_bytes_mut(); 
for (i, b) in bytes.iter_mut().enumerate() { 
    if *b >= 0x80 { 
        break; // non-ASCII at index i: hand the rest to the Unicode path 
    } 
    if b.is_ascii_uppercase() { 
        *b += 32; // 'A'..='Z' → 'a'..='z' 
    } 
}

It looks ideal: do the cheap byte work, and the instant you hit a non-ASCII byte, break and let the “real” Unicode path take over: “only do the cheap work until you have to.” On an Apple M4 this runs at about 3 GiB/s. That sounds fine in isolation, but it is more than 15× short of “optimal” because of the if branches.

Let’s delete every branch, line by line:

• if b >= 0x80 { break } → don’t stop at all. OR every byte into an accumulator and test it once, after the loop: high_bit_acc |= *b. Same information (was there any non-ASCII byte?), zero branches in the body.
• The A..=Z range test → make it arithmetic. b.wrapping_sub(b'A') < 26 is true exactly for A..=Z (any other byte wraps to ≥ 26), yielding a 0/1 mask with no branch.
• The conditional write → fold the mask into the store. | (is_upper << 5) sets bit 5—turning an upper-case letter lower-case and being a no-op on everything else—the byte is always written, never branched on.

What’s left has no branch in its body and no early exit:

let mut high_bit_acc: u8 = 0; 
for b in &mut bytes { 
    high_bit_acc |= *b; // detect any non-ASCII byte 
    let is_upper = b.wrapping_sub(b'A') < 26; // branchless A..=Z test 
    *b |= u8::from(is_upper) << 5; // set bit 5 → lowercase, else no-op 
} 
if high_bit_acc & 0x80 == 0 { 
    return bytes; // pure ASCII: already folded in place, no second buffer 
}

A loop with no data-dependent control flow is trivially vectorizable: LLVM emits 16-byte-at-a-time NEON and the whole thing runs at > 45 GiB/s—essentially memory bandwidth. And we come out of the pass already knowing, from high_bit_acc, whether there’s any non-ASCII work left to do.

How much did each step matter? Measuring the cumulative ladder on pure ASCII (Apple M4, 5.7 KB buffer):

Version	Throughput	Vectorized?
naive (break + branch test)	3.1 GiB/s	no (0 vector instrs)
→ branchless test/write, keep break	2.6 GiB/s	no (0 vector instrs)
→ drop the early-exit break	7.6 GiB/s	partially (25 vector instrs)
→ branchless test + write (the loop)	>45 GiB/s	fully (41 vector instrs)

The early-exit is what gates vectorization: keep the break but make the body perfectly branch-free and you still get zero vector instructions (~2.6 GiB/s); a data-dependent loop exit is enough on its own to keep the loop scalar. Only once the break is gone can the compiler vectorize. The final step—making the upper-case fold branchless—then turns a partially vectorized loop (which still compiles the conditional store to a compare-blend-masked-store, ~7.6 GiB/s) into the straight-line arithmetic that hits memory bandwidth.

Note: Branchless is a pessimization in scalar code. Look again at the table: making the body branchless while keeping the break (2.6 GiB/s) is actually slower than the naive branchy loop (3.1 GiB/s). The asm explains why. The branchy version only stores a byte when it actually changes one; its conditional strb is skipped for every lowercase letter, digit and space (the vast majority of real text), and the well-predicted branch that guards it is nearly free. The branchless version replaces that rarely taken store with an unconditional strb every iteration, writing back all ~5,700 bytes instead of just the handful of upper-case ones. Extra write traffic for no benefit. Branchless-write only wins once the loop vectorizes, because then the store becomes a single 16-byte vector write regardless of content, and the per-byte cost disappears. The lesson: a branchless body is worth it only as the enabler for vectorization. On its own, in scalar code, it can cost you.

There’s also a middle ground, and it’s what standard libraries use. Instead of testing one byte at a time, [u8]::is_ascii scans a machine word at a time—on a 64-bit target it tests 16 bytes per iteration by OR-ing two u64 lanes and checking all their high bits with a single & 0x8080_8080_8080_8080 mask. You can build the ASCII fast path on top of that: chunk-scan to find the ASCII prefix, then run the branchless (vectorizable) convert over it. That keeps the early-exit ability—it still bails on the first non-ASCII block—while letting both halves go fast. The catch is that it reads the data twice (once to scan, once to convert), landing at about 23 GiB/s—roughly half of the single-pass branchless sweep, and ~7× the naive break loop. A solid, general-purpose default; just not the absolute ceiling when you control the whole loop and can fold detection and conversion into one branch-free pass.

Wouldn’t fusing the two passes be faster? It’s the obvious next thought: keep the chunked early-exit but convert each 16-byte block right after you’ve confirmed it’s ASCII, reading the data only once. Measured, it’s ~2.6× slower—8.7 GiB/s versus the two-pass 23. The inner block convert still vectorizes to a single 16-byte op, but now there’s a data-dependent early-exit branch every 16 bytes, and that branch pins the loop to one block at a time: the compiler doesn’t unroll or software-pipeline across blocks, and each iteration pays the full load→test→branch→convert→store latency with nothing to hide it behind. Split into two passes, each one is clean: the scan is a branch-light, store-free word scan that races through memory, and the convert is the fully-vectorized branch-free sweep at >45 GiB/s. Two fast, branch-free passes beat one branchy fused pass—even though the fused version touches the data half as many times. It’s the same lesson one more time: in the hot loop, the branch is the enemy.

Avoiding the heap

Forty-Five GiB/s also means doing zero unnecessary allocation. simple_fold takes the input String by value, owning the heap buffer it can mutate and return it. If the OR-accumulator’s high bit was clear, the input was pure ASCII already folded in place. We hand the same allocation straight back, no second buffer and no copy. Otherwise, we memchr to the first non-ASCII byte and scan the tail from there, leaving the output buffer unallocated (a null write cursor) until we hit a character that folds to different bytes. Text whose multibyte content never folds—CJK, Hangul, Kana, Arabic, Hebrew, symbols—also returns the original allocation untouched, never copying a byte.

Why a second buffer rather than rewriting in place like the ASCII pass? Because folding can make the string longer: almost every fold preserves the UTF-8 length or shrinks it, but two outliers grow—U+023A (Ⱥ) and U+023E (Ɀ) are 2 bytes each yet fold to 3-byte characters (ⱥ, ɀ). Once one appears, the output no longer fits in the input’s bytes, and we need somewhere new to write.

We allocate that buffer once, sized for the worst case, rather than growing it as more folds appear. Incremental reserve calls would mean re-checking capacity, occasionally reallocating, copying everything written so far, and juggling extra length/capacity bookkeeping; a single up-front allocation lets a raw write cursor run straight to the end with none of that. And since the cursor is null until that first growing/changing fold, it doubles as the “have we allocated the extra buffer yet?” flag.

Sizing it needs a bound on growth, and those same two outliers give it: every 2 input bytes yield at most 3 output bytes, capping the output at 1.5× the input—exactly the capacity we reserve:

out = Vec::with_capacity(bytes.len() + bytes.len() / 2 + 4); 

After that the loop writes through a raw pointer with no capacity checks and calls set_len once at the end. Two more details keep it branch-light. The run of unchanged bytes between two folds is moved with a single copy_nonoverlapping rather than byte by byte. And each fold unconditionally writes all 4 bytes of a little-endian word before bumping the cursor by only the folded length (1–4)—dropping a branch on the output length from the hot path, with the + 4 in the reservation as the headroom that makes the final character’s over-store safe.

Making Unicode cheap too

When a character does fold, we still don’t want to fall off a cliff—decode UTF-8, hash lookup, re-encode. Unicode 16.0 has 1484 simple-fold mappings, but they’re a very sparse and *very structured relation. Four observations shrink them to 1776 bytes and let the fold run without ever decoding a full character.

Even on the non-ASCII path, the overwhelming majority of characters do not fold. The hot operation isn’t really “fold this character,” it’s “does this character fold?” Almost always no. The table has to make that negative test as cheap as possible; the actual folding is the rare case on an already-rare path. That priority is what shapes the layout below—the page bitmap exists precisely so a non-folding character is rejected in a single bit test, straight from its leading UTF-8 bytes, without decoding or scanning anything.

This is exactly why a HashMap<u32, u32> is the wrong shape for the job, not just a bigger one. A hash map is optimized for the hit: it finds a present key in roughly one probe, and only spends extra work (more probes, full key comparison) when load factor or collisions bite. But our workload is dominated by misses—characters that aren’t in the table at all—and a miss is a hash map’s least favorite query: it still has to hash the key, jump to a bucket, and walk the probe sequence far enough to prove absence.

Foldable code points cluster into 64-code-point “pages”

Foldable code points bunch together. Slice the code space into 64-code-point “pages” and the ~1484 folds touch just 59 of ~1960 possible pages. A one-bit-per-page presence bitmap answers the negative test on its own: a clear bit is a definitive “no fold”—copy through, done—which is what makes fold-free scripts cheap. Only on a set bit do we consult a second structure, a cumulative-popcount side table that ranks the page (how many populated pages precede it) to find its slice of entries, storing nothing for the ~1900 empty pages.

let (word_idx, bit_idx, c_len) = if lead < 0xE0 { 
    (0usize, lead & 0x1F, 2usize) // 2-byte: word 0 
} else if lead < 0xF0 { 
    ((lead & 0x0F) as usize, bytes[read + 1] & 0x3F, 3) // 3-byte: word = nibble 
 
} else { 
    ( 
        (((lead & 0x07) as usize) << 6) | (bytes[read + 1] & 0x3F) as usize, 
        bytes[read + 2] & 0x3F, 
        4usize, 
    ) // 4-byte: merge 2 bytes 
}; 
// reject without decoding: clear bit ⇒ no fold 
if word_idx >= PAGE_BITMAP.len() || (PAGE_BITMAP[word_idx] >> bit_idx) & 1 == 0 { 
    read += c_len; 
    continue; 
} 

Because word_idx depends only on the lead byte (and, for four-byte sequences, the first continuation byte), the bitmap load can be issued early.

Within a page, folds come in runs

A set page bit tells us something on this page folds, but not which code points or to what. The obvious encoding is one entry per foldable code point—but that is both bulky and slow to search: a page can hold dozens of folds, and we’d have to scan them all to find the one matching the current code point. The structure of the data rescues us again. Adjacent code points overwhelmingly share the same delta to their fold: A–Z all map +32, and Latin Extended is full of alternating runs like 0x0100, 0x0102, 0x0104, … where every second code point folds. Instead of per-code-point entries we store runs—start, end, stride, delta—and a 1-bit stride flag covers both the contiguous and the every-other case. This interval compression collapses the ~1484 individual folds into just 238 runs across the 59 pages (≈four per page), leaving the within-page search only a handful of entries to look at instead of dozens. This range-with-delta encoding (including the stride trick) is borrowed from Go’s unicode package, whose CaseRange records store a Lo/Hi range plus per-case deltas, with an UpperLower sentinel marking the alternating blocks. Runs are split at the page boundaries so a run never straddles two pages.

A run record is two clean bytes

With both endpoints inside one page they fit in 6 bits, split across two arrays: RUN_END_LOW[i] = end & 0x3F (the scan key) and RUN_START_STRIDE[i] = (start & 0x3F) | ((stride − 1) << 6) (read only on a hit). Because each key is one clean byte, the within-page search can go wide: rather than comparing cp & 0x3F against the runs one at a time, we load 8 end_low bytes into a single u64 and test all of them at once with one branchless SWAR step—(chunk | 0x80…80) − broadcast(low) & 0x80…80 sets the top bit of every lane whose key is ≥ cp & 0x3F. A single bit-scan of that mask (the keys are sorted, so the first set lane is the run we want) finds the slot. A page holds ~4 runs on average; that one 8-wide compare almost always resolves the entire search in a single step. One unlucky page does hold 30 runs, which puts the compare inside a short loop that strides eight keys at a time—but that loop trips at most a handful of times on exactly one page in all of Unicode, and never on the common ones. Either way: no per-run branch, and no code-point reconstruction anywhere.

/// Offset of the first run with `end_low >= low_v` in a page of `n` runs, 
/// or `n` if none. Scans 8 `end_low` bytes at a time via SWAR. 
#[inline] 
fn scan_end_low(lo: usize, n: usize, low_v: u8) -> usize { 
    const HIGH: u64 = 0x8080_8080_8080_8080; 
    const ONES: u64 = 0x0101_0101_0101_0101; 
    let bcast = (low_v as u64).wrapping_mul(ONES); 
    let mut base = 0; 
    while base < n { 
        // RUN_END_LOW is padded by 8 bytes so this read is always in bounds. 
        let chunk = u64::from_le_bytes( 
            RUN_END_LOW[lo + base..lo + base + 8] 
                .try_into() 
                .expect("8-byte slice"), 
        ); 
        // `(b | 0x80) - low_v` keeps its high bit iff `b >= low_v` (no 
        // cross-lane borrow). The first set lane is the first run `>= low_v`. 
        let ge = (chunk | HIGH).wrapping_sub(bcast) & HIGH; 
        if ge != 0 { 
            let j = base + (ge.trailing_zeros() / 8) as usize; 
            return if j < n { j } else { n }; 
        } 
        base += 8; 
    } 
    n 
} 

Folding is a little-endian byte addition

On a little-endian machine the folded character’s UTF-8 bytes, read as a u32, equal the source bytes (as a u32) plus a per-run constant. A parallel BYTE_DELTA[i] table then turns the whole fold into a masked load, one wrapping_add, and a 4-byte store:

let word = u32::from_le_bytes(next_four_bytes) & length_mask; // keep this char's bytes 
let folded = word.wrapping_add(BYTE_DELTA[i]); // the fold, as one byte add 
write_u32_le(dst, folded); // store all 4 bytes... 
dst += utf8_len(folded); // ...advance by the folded length

Both lengths in that snippet—the length_mask for the source character and the advance by the folded length for the destination—come from one more tiny trick. A UTF-8 sequence’s length is fixed by the top four bits of its lead byte, letting the 16 possible lengths pack one nibble each into a single 64-bit constant (0x4322_1111_1111_1111); the length is then a shift and a mask, (LEN_BITS >> (4 * (lead >> 4))) & 0xF—no if chain, no table memory, nothing for the predictor to get wrong. (A count leading ones—(!lead).leading_zeros()—would also work, since a lead byte carries one leading 1-bit per byte of the sequence.)

/// Number of bytes in the UTF-8 sequence whose lead byte is `lead`. 
#[inline]
pub fn utf8_len(lead: u8) -> usize { 
    const UTF8_LEN_BY_LEAD: u64 = 0x4322_1111_1111_1111; 
    ((UTF8_LEN_BY_LEAD >> (4 * (lead >> 4))) & 0xF) as usize 
}

Because we advance by the folded length, this even handles length-changing folds—U+212A KELVIN SIGN (3 bytes) → k (1 byte), or U+023A Ⱥ (2 bytes) → U+2C65 ⱥ (3 bytes)—by writing fewer or more bytes than were read. That’s the part we believe is genuinely new: every other folder we looked at—ICU, Go’s unicode, Rust’s regex, CPython, glibc—decodes UTF-8 to a code point, applies the fold there, and re-encodes (even SIMD folders decode first). Doing the arithmetic in byte space skips both the decode and the encode, which is exactly why this path can outrun a hash map that already has the answer tabulated—the hash map still has to decode its key and encode its result. The byte-space arithmetic assumes the input is well-formed, shortest-form UTF-8—every code point encoded with the minimal number of bytes. Reading the source bytes as a u32 and adding a per-run delta only lands on the correct folded encoding when the source is in canonical form; an overlong encoding (a code point padded into more bytes than necessary, e.g. / as 0xC0 0xAF) has a different byte pattern and would break the length_mask and the delta arithmetic. This is not a real restriction in Rust—&str/String are guaranteed to hold valid UTF-8, which by definition rejects overlong sequences—but a caller feeding raw bytes from elsewhere must validate (or otherwise normalize) them first.

The ASCII shortcut in the tail loop

One more shortcut rounds out the tail loop. Remember the first pass already lowercased every ASCII byte, so when the scan meets an ASCII byte in the tail it advances a single byte and moves on—no page probe, no table touch at all. And it doesn’t copy that byte either: unmodified bytes (ASCII and non-folding multibyte alike) aren’t moved one at a time. The scan just keeps walking until it reaches a character that actually folds, then flushes the whole unchanged run between the last fold and this one with a single copy_nonoverlapping. Mixed text—CJK with ASCII spaces and punctuation, or code with the occasional accented identifier—therefore races through the ASCII filler and only consults the bitmap for genuine multibyte characters, copying in bulk rather than byte by byte.

Putting it together: the whole table

Component	Bytes
PAGE_BITMAP (1 bit per 64-cp page)	248
POPCNT_SAMPLES (cumulative popcount)	32
PAGE_OFFSET (per populated page)	60
RUN_END_LOW (scan key, end & 0x3F, +8 pad)	246
RUN_START_STRIDE (start & 0x3F | stride)	238
BYTE_DELTA (little-endian fold delta per run)	952
Total	1776

That’s 9.6 bits per fold entry, over half of it the BYTE_DELTA side table we trade for the decode-free path; the index + run records alone are ~4.4 bits/entry.

Next to the obvious alternatives, that 1776 bytes is an order of magnitude or more smaller—and unlike most of them, it never decodes a character:

Representation	Size
Naïve [(u32, u32); 1484]	~11.6 KB
regex-syntax’s case_folding_simple table	~70 KB
Go’s unicode.SimpleFold (orbit + ASCII + ranges)	~7.3 KB
A runtime HashMap<u32, u32>	~17 KB
This crate (paged bitmap + packed runs)	1776 B

Where it lands against the alternatives

On the common case, ASCII, folding runs at memory bandwidth (>45 GiB/s), more than an order of magnitude ahead of other real folders and more than 50% faster than the (non-equivalent) str::to_lowercase function. To get a rough “upper bound” for the non-ASCII case, we measured the optimized Utf8 decoding + encoding round trip without performing any actual case folding using the simdutf crate. This experiment achieves consistently about 2GB/sec and is only about twice as fast than our solution for the worst case all-folding input. A naive hash map trails everything on all workloads.

The three columns are real case folders that produce identical output: simple_fold (this crate), simd_normalizer (the simd-normalizer crate), and HashMap (naive CaseFolding.txt lookup). The workload rows are chosen to simulate different scenarios from typical to worst case:

Workload (input size)	simple_fold	simd_normalizer	HashMap (byte path)
Pure ASCII (5.7 KB)	>45 GiB/s	1.21 GiB/s	213 MiB/s
Chinese/Japanese/Korean, no folds (8.1 KB)	2.95 GiB/s	1.97 GiB/s	558 MiB/s
Symbols / Myanmar, no folds (9.0 KB)	2.96 GiB/s	1.56 GiB/s	410 MiB/s
Worst case: Latin/Greek/Cyrillic (Unicode U+0000–U+FFFF), all folding (8.8 KB)	869 MiB/s	922 MiB/s	334 MiB/s
Length-changing folds (1.7 KB)	1.26 GiB/s	716 MiB/s	233 MiB/s

Treat the absolute figures as illustrative, not portable: the whole design leans on auto-vectorization, SWAR, and little-endian byte arithmetic, so the numbers—and even the ratios between rows—can shift substantially on a different microarchitecture (a wider or narrower vector unit, different memory bandwidth, a big-endian target, x86 vs ARM).

More details can be found in the performance section of the README.

Take this with you

Case folding is about as basic as text operations get, which is exactly why it was worth the effort: we run it across every byte we index. The wins came from two ideas that both cut against instinct—sweep the whole buffer branch-free instead of stopping early, and do the fold as byte-space arithmetic instead of decoding to a code point. Together they let the common case run at memory bandwidth and the rare fold run without a decode, in a table small enough (1776 bytes) to stay resident. The decode-free byte-space fold is the piece we believe is genuinely new; it’s why this path can beat a hash map that already has the answer.

There’s surely more to find here, and we’d like to see it. The crate is casefold; the generated table and full design notes live alongside the source.

Calendars and Missing Dates#

In 1582, under the inter gravissimas papal bull, the Catholic world transitioned from the Julian Calendar to the Gregorian Calendar. This new calendar shortened the length of the year from 365.25 days to 365.2425, a reduction of just 0.002%.

One side effect of this is the modern leap year system, in which every fourth and 400th year is a leap year, but no other year divisible by 100 is. For example, this means that 1900 was a normal year, 1904 was a leap year, and 2000 was also a leap year.

Another side effect (and the topic of today’s post) was that the Julian and Gregorian calendars no longer agreed on the date. Calculating each from 1 to to 1582 AD, the Julian calendar had slowly accumulated “drift” relative to the Gregorian calendar, lagging eleven days behind it.

To correct for this drift, ten days had to be removed from the 1582 year, converting the entire system from Julian to Gregorian. Selecting an appropriate 10-day span took nearly 20 years (owing in no small part to the Catholic church’s reluctance to skip any holidays and desire to correct the Easter drift), but eventually the span of October 5 to October 14 was chosen.

The end result? On the fourth of October, 1582, citizens of the Catholic world* went to sleep and woke up ten days later, on the fifteenth of October, with a new calendar.

In essence, then, those ten days in 1582 never happened and simply do not exist within the Gregorian calendar system used almost universally in the West today.

So, how well do programming languages handle this range of dates?

Ruby#

Right off the bat, Ruby does the right thing - it simply does not allow a DateTime with an impossible Gregorian date to be created:

1
2
3
4
5
6
7
irb(main):001:0> require 'date'
=> true
irb(main):002:0> DateTime.new(1582, 10, 5) # October 5, 1582
ArgumentError: invalid date
	from (irb):2:in `new'
	from (irb):2
	from /usr/bin/irb:12:in `<main>'

This can be traced to the datetime_s_civil function in ext/date/date_core.c:

1
2
3
4
if (!valid_gregorian_p(y, m, d,
		       &nth, &ry,
		       &rm, &rd))
    rb_raise(rb_eArgError, "invalid date");

Python#

Python’s datetime module, despite claiming to represent a date object in “the current Gregorian calendar,” sadly does not do so:

1
2
3
4
5
6
7
8
Python 2.7.6 (default, Mar 22 2014, 22:59:56)
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from datetime import date
>>> date(1582, 10, 5) # October 5, 1582
datetime.date(1582, 10, 5)
>>> date(1582, 10, 5).ctime() # October 5, 1582
'Tue Oct  5 00:00:00 1582'

This is the case for both Python 2 and Python 3.

Perl#

Perl has no standard analog to Ruby’s DateTime or Python’s date, so I opted for the common DateTime module from the CPAN instead. Unfortunately, like Python, a proper error message for impossible Gregorian dates is notably absent:

1
2
3
4
5
6
7
#!/usr/bin/env perl

use DateTime;

my $dt = DateTime->new(year => 1582, month => 10, day => 5);

print $dt->ymd('/'), "\n"

This example runs without error (which is the error) on Perl 5.18, with DateTime 1.19.

Summary#

Despite the fact that dates between the 5th and 14th of October in 1582 are impossible to represent on the Gregorian calendar, two out of the three languages tested allow programmers to create “Gregorian” dates for those days.

This is certainly no Y2K or Unix epoch problem, but it’s a good indicator of how far we’ve come with respect to plain and simple correctness in date and time implementations. Even if it means adding an extra check to make sure that nobody creates a date within a 10-day range nearly 500 years ago.

I hope you’ve enjoyed this quick little exploration into calendars and their intricacies. If you’re still interested, there are a ton of cool mathematical formulas and historical justifications for the current calendar system (Zeller’s Congruence, Pre-Julian Calendars) worth taking a look at.

Happy Hacking!

Afternotes:

* As one might suspect, the non-Catholic churches (and religions) were less than inclined to obey an official papal bull. As a result, although the “official” calendar reform took place in the October of 1582, many protestant churches continued to use the Julian calendar well into the 18th century. The Eastern Orthodox church even continued well after that, only switching to the Gregorian calendar in 1929. These later switches required their own 10+ day removals in order to synchronize the Julian and Gregorian calendars.

P.S.:

On a whim, I checked ncal(1)’s Gregorian correctness. Interestingly enough, it may be the most correct of all. Instead of assuming that the Catholic adjustment is the most correct one, it takes a country code with the -s flag and attempts to determine when that country performed their adjustment.

For example, Italy (ncal -s IT 10 1582):

1
2
3
4
5
6
7
8
    October 1582
Su    17 24 31
Mo  1 18 25
Tu  2 19 26
We  3 20 27
Th  4 21 28
Fr 15 22 29
Sa 16 23 30

…and Great Britain (ncal -s GB 9 1752):

1
2
3
4
5
6
7
8
    September 1752
Su    17 24
Mo    18 25
Tu  1 19 26
We  2 20 27
Th 14 21 28
Fr 15 22 29
Sa 16 23 30

Although this likely takes a great deal of work, it’s by far the most correct of all.

Addendum 2026-06-30: John Costello points out that ncal(1) is still not perfectly correct, since it misses Sweden’s calendar shifts in the 1700s. Here’s a verbatim Gist of his analysis; thanks John!