UTF-8’s diner-placemat design had one demand: find your place in the bytes

Ken Thompson worked out UTF-8’s byte packing on a New Jersey diner placemat. The result then had to become a running system on a very short deadline. Apparently the right stationery for a text-encoding design was already under dinner.

Rob Pike’s first-person account, preserved by Markus Kuhn, corrects a story that gave IBM the design and Plan 9 the implementation. Pike credits the X/Open and IBM participants with creating the opportunity and pushing it forward, while attributing the actual design to Thompson with his own encouragement.

The proposed encoding had to accommodate real software. Existing filesystems and programs already interpreted some byte values in particular ways. Supporting more characters was therefore tangled up with preserving the behavior of tools that hadn’t been written for a newly enlarged alphabet.

Pike remembers a committee asking the pair to vet its FSS/UTF proposal. He described requirements and found at least one missing: the ability to recover a character boundary after starting inside a byte stream. The pair had room to propose a different scheme, provided they could implement it quickly.

Self-synchronization gives that requirement a practical shape. When continuation bytes have a distinguishable pattern, a decoder arriving partway through a multibyte character can find where the next character starts. A page number gets you back into a book; it doesn’t rewrite the sentence whose beginning you missed.

That last distinction matters for the everyday claim that corruption cannot spoil the rest of a file. Finding boundaries helps recovery. It doesn’t reconstruct lost bytes, validate every damaged sequence, or establish that an application responds correctly. A format’s useful recovery property is narrower than a magical promise that text never goes bad.

Pike recalls Thompson writing the packing code while he worked through libraries and other system pieces. By Friday, he says, Plan 9 was running the new encoding. The design was being exercised across a system, making the encoding’s contact with existing software more concrete than a successful demonstration on one sample string.

The archive adds something sturdier than a good dinner anecdote. Recovered code history and September 8 proposal mail give the sequence documentary support. Pike also admits uncertainty about how much the original proposal influenced them. A participant can remember the placemat vividly while leaving other parts of the evening less sharply defined.

The reproduced proposal should also remain a historical document. Its table includes longer sequences than today’s UTF-8 permits. Copying its old code into a modern validator would confuse the origin with the current specification. The interesting artifact deserves to keep its date attached.

The requirement list is the reusable part of the episode. A format used by existing tools needs explicit answers about compatibility and recovery before those tools depend on it. The missing placemat still leaves one small archival disappointment: Pike says he wished they’d kept it.