Simon Willison has released version 1.0 of condense-json, a Python library designed to reduce file sizes by replacing repeated strings with compact references. The tool scans input JSON for specific text patterns defined in a separate replacements object and substitutes them with a special syntax containing a unique identifier. This process creates a condensed format that can be reverted to the original structure using a companion function. Willison notes that the code has undergone sensible fixes over its year and a half existence before reaching this stable milestone.
The primary utility lies in storing large datasets within SQLite databases without inflating storage requirements. Willison applies this directly to logs generated by his LLM dataset tool, where identical prompts and responses appear frequently. By compressing these repeated elements, the library allows developers to maintain extensive historical records in a single file. The approach avoids complex schema changes while offering a straightforward method for data minimisation in Python applications.
* Replaces repeated substrings with a `$r` reference syntax
* Fully reversible via the `uncondense_json` function
* Optimises SQLite storage for datasets with high repetition
