Simon Willison has released a prototype for storing text revision histories in SQLite that uses compression to keep file sizes tiny.
While walking his dog, the developer considered how to store previous versions of documents in a relational database. He suggested taking the full text of every prior version and placing it in a single JSON array of strings. Applying zlib or zstd compression to that whole array should remove redundant text effectively.
Willison discussed the concept with the GPT-Live voice mode in the ChatGPT iPhone app. Although sharing URLs to voice conversations is not yet possible, he provided a transcript of the discussion:
I have an interesting idea for a scheme for saving all previous versions of a piece of text that’s constantly edited in a SQLite database um column in as efficient a way as possible. Okay, so I built these kinds of systems in the past, and it’s always difficult to come up with a efficient way to do this. Like the easiest way is you have a row for every previous copy of the previous previous value of the string. But if it’s a long document Like20 kilobytes of data, that means that every single edit adds another 20 kilobytes of data to the database, right. So, what I’ve now thinking, is um compression would work really well, right? If you Bundle all of those different um Every every version of this document all the way back to the start if you were to apply a good compression algorithm to them that should basically wipe out huge amounts of the redund- the um redundant text, right Um, so what I’d thinking is how about really, really simple mechanism There is a history column on the single on this uh uh table and it’s a blob, it’s a BLOB so it stores binary data and then you just stick in there a Zlib or maybe even ZSTD um compressed JSON text array of all of the previous documents, and so you probably have two columns, right? You’d have a column that’s this magic JSON array of text You have a second column which is a JSON array of timestamps and that doesn’t need to be compressed at all, right. A timestamp can just be a uh- it’s an array of integers, right. Unix integers But that’s the whole scheme.
He then stopped the voice mode and sent a prompt to GPT-5.6 Sol Pro to build experimental prototypes in Python.
Use Python and Build experimental prototypes around this idea
The model worked for 38 minutes and delivered an answer plus the files found in the GitHub repository.
The approach works really well. 1,000 simulated revisions to a document resulted in 20.4 MB of raw revision text that compressed to 80.3 KB as Zstandard-compressed JSON array.
To avoid the overhead of decompressing and recompressing the entire array on every edit, the suggestion was to break the history up into multiple rows. Each row would contain a maximum of either 128 revisions or 3 MB of uncompressed JSON.
What it means
Developers can store long editing histories in a single database row without bloating the file size. They simply store a compressed array of all past versions alongside a separate array of timestamps.
