I had five years and 218 Substack posts, exported as a pile of HTML files but nothing you could actually query. Here’s how I turned that archive into a real knowledge base, using structured approaches like taxonomies and batch processing.
If I’m honest, I’m not really someone who organizes as I go.
When I finally decided to try building my own LLM wiki as a folder of markdown files, I had accumulated 218 Substack posts.
(If you are new here, an LLM wiki is a simple, but organized knowledge base that you connect directly to your AI.)
Whenever I needed to reference or find an old post, I generally scrolled one post at a time on the Substack dashboard.
Of course, I always have a sense of my past writings. But without scrolling for twenty minutes, I could not tell you which post first made a given point, or whether I’d already made it three times under three different titles.
It is a problem that’s plagued me and sent me off looking for the perfect app.
I’ve written about the graveyard of note apps behind me ... Mem, Notion, Roam, Anytype.
Each one solved a different piece of the problem and then stalled out, because storage, editing, and AI access were all bundled into one product I’d eventually outgrow.
The fix was to stop treating those as one job. Separate the three layers, and you can swap each one as your workflow changes:
Storage — a folder of markdown files.
Structure — an editor that reads the files without adding its own syntax.
Access — an AI pointed at the folder instead of owning it.
That was the theory. Substack was the first real test of it at scale.
How I exported and organized 218 Substack posts
So I pulled all 218 posts off the internet and rebuilt them as a knowledge base.
Not a backup. Not an archive. Something an AI could actually query with precision.
The difference between a folder of old posts and a working wiki is more about what happens before you import anything.
At first, I thought this would be simple.
A wrong classification, once it’s live, is worse than no classification at all, because errors simply scale with the knowledge base.
Substack has an export button, so I clicked it, downloaded the zip, and opened it up expecting at least text files that were ready to go.
Instead, all I saw was a bunch of HTML files. One per post. Each file still had web page artifacts like subscribe buttons and pull quote widgets all in markup.
Now I’m not a coder, and writing a script really isn’t my thing. That’s where Claude Cowork came in.
I described what I had and what I actually wanted, and it wrote a script that stripped the Substack widgets out first, then ran the real HTML-to-markdown conversion on what was left.
But stripping the HTML only got me markdown files. It didn’t get me a knowledge base. Just 218 loose text files sitting in a folder, no better organized than before.
So before I let anything touch the real vault, I had to decide what I was actually building.
What counts as a domain.
What tags mean
What goes in the frontmatter of every single file, so an AI reading one of these posts later knows what it’s looking at or for.
Skip that step and the import is just digital hoarding with better formatting.
It just so happens, that I’d been reading Manny Silva’s book, Docs as Tests with AI, around the same time, and one idea stuck.
Don’t just trust that a piece of documentation does its job. Test it. Applied to 218 posts, that meant not dumping them all in at once and hoping everything got organized correctly.
Do them twenty at a time instead.
For each batch, Claude proposed a domain, a type, a set of tags, and handed the whole thing back as a table, one row per post that I could scan top to bottom to catch any major mismatches.
It was slow going for 218 posts, but that’s part of the point. A wrong classification, once it’s live, is worse than no classification at all, because errors simply scale with the knowledge base.
Once I approved a batch, Claude Cowork took the next step. A script picked up the approved table, wrote the frontmatter into each file, and moved it into its folder.
I checked the work, not every individual move.
Mapping content from the ground up
Doing this helped me better understand the nature of my content and how to build organization rules.
In a way, domains started clustering on their own.
None of this would matter if the schema only helped me look backward, though. What happens the next time I sit down to write something new?
I asked Claude to build a Map of Content for one of them, ai-and-writing, and it pulled ten clusters straight out of the classification tags. No hand-curation. All 91 files in that folder linked exactly once. Zero orphans.
To actually become a wiki, the content needs to be able to connect beyond just the folder structure. A MOC isn’t just a table of contents, but an actual map for important topics or key words that connect a level above folders.
I liked those MOCs enough to turn a few of them into their own Substack posts that point readers back through five years of writing organized by theme instead of by publish date. Something I’d been meaning to do for a while, and kept putting off, because doing it by hand is tedious.
None of this would matter if the schema only helped me look backward, though. What happens the next time I sit down to write something new?
I finished a draft a few weeks ago and ran it through a skill that scans a near-finished piece for link opportunities before it goes out. It doesn’t touch the writing itself. It just reads the draft, checks it against everything I’ve published before, and reports back where a backlink might be added.
It found three without me having to go digging anywhere. Every one of those 218 posts carries a published_url and a real domain and type in its frontmatter now, the same fields the batch tables assigned back when I was still checking classifications one table at a time.
The metadata I built for the import is what the skill reads to make the connection now.
That’s the actual payoff for all this. Not a tidier folder. A knowledge base that can cite itself and project to readers a more coherent body of work.
That skill is already up in my public Structured Skills Library, if you’re a paid subscriber and want to go pull it apart right now. I’m also sharing it directly here soon.
If you’ve got years of writing scattered across a platform’s servers, the export is the easy part. Substack will hand it to you in five minutes. The work is deciding what you actually want to be able to ask your own archive later.
Below is a short reference that paid subscribers can use to start this process themselves ... and of course, more specifics can be found in the Structured Skills Library.
