Querying a folder of files by their metadata, without a database

You have just moved a few thousand records into their new home. Each one is a Slipcase container now — a single file carrying its own metadata inside it — and the files all came across. The migration looks done. Then comes the question that decides whether it actually went well: which records came across clean? Which are missing a retention date, or a reference number, or an owner? Which are still marked draft? You have a few thousand files and no way to see the answer by opening them one at a time.

The usual way to get that answer is to rebuild the thing you were trying to leave. Unpack every container into a spreadsheet and you have a spreadsheet that was true the moment you made it and wrong the next time someone adds a file. Load them into a small database and the folder and the database start disagreeing as soon as they drift, and the metadata you were so careful to keep inside each file is back to living somewhere else. Either way you are keeping a copy of the truth next to the truth.

A folder of Slipcase containers does not need that, because it is already a table. Every container carries its description the same way — a small TOML document Slipcase calls the flyleaf — so every file is a row and every flyleaf key is a column. The table is sitting on the disk. It does not need loading anywhere. It needs a way to be asked.

SlipQL is that way. Point it at a directory and ask in the three clauses anyone who has written a line of SQL already has in hand — select, from, where. To find the records a migration left incomplete:

$ slipql ./records --recursive
Connected to: ./records (recursive). Type "help" for commands, "quit" to exit.

slipql> select @path, title where not exists retention or not exists reference
| @path                     | title       |
| ------------------------- | ----------- |
| 2019/scan-0471.pdf.slpc   |             |
| legal/nda-draft.docx.slpc | NDA (draft) |
| q2/summary.xlsx.slpc      | Q2 summary  |
(3 rows)

Each row is a container; @path is where it sits under the folder you pointed at; the columns are flyleaf keys. The scan that is missing a title shows that cell empty rather than breaking the query, because flyleaf keys are ad hoc by design — and not exists retention is exactly the row you were hunting: a record whose retention was never set.

A few things make that answer one you can act on. SlipQL reads each container's flyleaf in place and keeps no index, so every query is a fresh scan and what comes back is what is on the disk now, not what was true when some export last ran. It compares values only within their type and never coerces, so a "7" typed where the number 7 belonged comes back as a mismatch it reports, not a match it invents — the audit catches the malformed as well as the missing. And rows arrive as the scan reaches them, so a query over a large tree starts answering at once and limit stops it early.

SlipQL is the command line — a step in a migration runbook, or a pipe into the rest of the tabular tools, since it will emit CSV straight into xled. When the job is instead to sit and read a folder by hand, Slipcase Query is the same engine behind a window: the same select, from and where, the rows on screen to scroll. One is for the pipeline; the other is for the afternoon you spend checking the set before you sign off on it.

And because the metadata lives inside each file in an open format — a .slpc is a ZIP with a TOML flyleaf, specified in the public domain — what you are querying is yours, not a proprietary store someone else has to grant you into.

“Which files say what” is the daily question in the migration, handover, and records work I am brought in on, and the honest answer is usually a spreadsheet nobody fully trusts. Getting a body of records onto a footing you can query directly, with nothing to keep in sync because the answers live in the files, is exactly the kind of problem I like being brought in on.

Get in touch See SlipQL →