I spent the summer building a markdown knowledge base that AI helps me manage, then went looking for research that would tell me it was worth it. Four studies across twenty-seven years, including the first real evaluation of AGENTS.md context files, suggest the organizing work is easier to demand than to justify.
I’ve got to be honest with you. I’ve spent most of this summer building workflows around a personal markdown knowledge base that AI helps me manage.
Tags, links, folder structure, naming conventions. All the housekeeping I’ve quietly avoided for twenty years. Everything I hate about managing my own notes is exactly what keeps sending me off to try one more app.
Surely there’s a perfect one out there. There isn’t. I know there isn’t.
I go looking anyway, about twice a year. For a brain that would rather build the system than sit down and use it, this is a dangerous hobby.
I’m Lance Cummings. And welcome to my now monthly podcast that explores deep research on AI and writing.
After all this, I decided it might be worth looking into what kind of research has been done on personal knowledge bases.
Oh, yeah ... there’s lots of creators out there giving out all sorts of ideas and frameworks, but what does the research say?
Here are some of my initial findings.
I looked at four papers, written across twenty-seven years, are all circling the same question from different sides: what does it actually cost to make your own knowledge explicit, and who ends up collecting?
That question used to have an easy answer.
You did the filing for future-you, future-you rarely showed up, and the whole thing quietly rotted. What’s changed is that my knowledge base has a second reader now. An AI assistant searches it, and it searches differently than I browse.
So it sounds like a reason to finally do the work of organizing. And it might be.
But you might want to hold on a second.
The structure nobody wants to do
In 1999, Frank Shipman and Cathy Marshall point out in their article [Formality Considered Harmful] that the real problem is motivation and intention.
After years of building systems people wouldn’t use, the real problem came down to something simpler than interface design. Users were unwilling or unable to make structure explicit.
If asked to declare what a thing was or how it related to other things (👋 information types), people balked, worked around it, or filled the field with garbage.
Not from laziness. The cost of formalizing is immediate and the payoff is speculative and later. You know what you mean right now. You are, at that moment, the worst possible judge of what future-you will need.
This is probably one reason it’s so hard for content strategists to convince stakeholders to structure their content.
Their solution really ties to how I’m working with AI to build a personal knowledge base. Rather than demanding structure up front, systems should offer incremental, system-assisted formalization. The system flags structure you never declared and helps you make it explicit over time.
In other words, build your knowledge from the ground up with help. Start small and use AI to help you discover the organization that’s already there.
That’s more or less what I’ve been building.
It’s also how I nearly lost two folders in my own vault this month, because an assistant had been quietly filing things into subfolders I never asked for.
I got system-assisted formalization, but I wasn’t paying close enough attention to what the AI agent was doing.
What folders and tags were both missing
Jump to 2008. Amy Civan, William Jones, Predrag Klasnja, and Harry Bruce ran a study to find out what kind of organization works best, focusing mostly on tags and folders.
There’s no winner.
Each model has strengths and weaknesses across keeping, organizing, and re-finding information. What study participants needed was a clearer framework for the organization scheme itself.
Think here about things like taxonomies, ontologies ... or simply deciding what goes where, why, and for whom ... and how that “what” gets found.
That’s more important than what tool to use.
What they wanted was a way to say what the categories meant, and to change their minds later without everything breaking. The tool or method doesn’t give you that.
I found this out the hard way.
I counted the tags in my vault a few weeks ago: 1,988 of them. At least forty-five were the same idea written in different ways — “rag” and “RAG,” “pkm” and “PKM.”
There’s a danger that a search or an AI scan might only turn up half the entries for tags like these. The tagging system worked fine. I just had nowhere to record that those two spellings meant one thing or master guide that gives clear guidelines (without guessing in the moment).
That gap is what I ended up filling with three plain markdown control documents: a taxonomy document, tag thesaurus, and workflow overview.
More on that to come in future posts.
How AI is changing the way we think about organization
You’ve probably seen the trend already. To better control agentic workflows, people are starting to do this formalization work voluntarily — not in note-taking apps, but in code repositories, as context files for AI agents, files like AGENTS.md, CLAUDE.md, the README written for a machine.
But there are a few problems arising because developers are mostly controlling what should be called “documentation,” which needs a content specialist.
A 2025 preprint, Agent READMEs by Chatlatanagulchai and colleagues, studied 2,303 of these files across 1,925 repositories. What they found is that these files get maintained the way a build script gets maintained: small commits, constantly, forever.
They grow harder to read as they go. Nobody sits down and revises one.
The content skews hard toward the function ... not context:
implementation details — 69.9%
architecture — 67.7%
build and run commands — 62.3%
security — 14.5%
performance — 14.5%
So we tell the machine how to operate the thing, and almost never what the thing is for.
In the content world, we call this the “structure tax.” Maybe you don’t pay a price right now for structuring your knowledge ... but you will later.
That’s why this work is an investment, whether you are a company, a university, or just a solo creator.
The structure tax people refused to pay for themselves, they’re now paying for an agent. [pause]
The part that stopped me
Here’s where I have to complicate my own summer.
The obvious conclusion from all of this is that context files work, so write them. That’s certainly what I assumed while building mine.
Then in February, Thibaud Gloaguen, Niels Mündler, and colleagues at ETH Zurich published Evaluating AGENTS.md, the first rigorous test I know of. They ran coding agents three ways: with no context file, with one the agent generated for itself, and with one a human developer had actually written.
The machine-generated files didn’t help.
They made things slightly worse, and raised inference cost by more than twenty percent. The human-written ones were the only version that came out ahead, and they beat the machine’s version by a clear margin.
The temptation is to have AI generate this documentation for you ... its lots of work. Turns out that could be doing more harm than no context file at all.
But consider this carefully, the agents did follow the files.
When a file named a specific tool, the agent used it almost every time; when it didn’t, almost never. The instructions worked.
What didn’t work was the one thing every official recommendation tells you to put first: the repository overview. It’s the biggest chunk of these files, and it did nothing. Agents found what they needed just as fast without it.
In the end, this is a failure of content strategy. An overview is something the machine can already work out by looking around, but principles and processes needed to properly do that task ... not so much.
The files contained clear information types beyond just an overview. But, honestly, nobody in that study tested a context file designed that way. They varied the model, the prompt, the length, whole categories of content, and none of it moved the needle, because they were all the wrong knobs.
The one variable that mattered was a human deciding what actually belonged in the context file.
Two honest limits to these studies.
Both agent papers are preprints, not yet peer-reviewed. And they measure code tasks in repositories, a long way from retrieval in a personal knowledge base. I don’t think the finding transfers cleanly.
But nobody has run the knowledge-base version of that study, which is roughly where the next several weeks of this newsletter are headed. Stay tuned.
Takeaways for writers and content professionals
To be clear, this does not mean we should “stop organizing your notes.” Rather, we need to pay more attention to how we do it, especially when AI is part of the picture.
So the four things below are the ones I’d still do this week.
Write control documents, not more folders. The 2008 finding is the durable one. Before you reorganize anything, write down what your categories mean and which synonyms resolve to which preferred term. My tag thesaurus took forty minutes and fixed years of broken searches. No new app required.
Audit what your assistant has been organizing for you. If AI helps maintain your notes, run a structure check on purpose, full depth, no shortcuts. Incremental formalization is genuinely useful and it’s also invisible by default. I nearly lost two folders to that.
Treat your context file as configuration, and version it. If it evolves like code, maintain it like code. Small commits, a changelog, and periodic pruning. Files that grow without eyeballs get harder to read and control.
A/B Test your workflows. Run a task with your context file and without it. This is the least fun recommendation but can help you see whether a specific skill or documentation is actually needed..
If you want the practical side of this, the skills that run my knowledge-base-to-Blue drafting pipeline are posted in my public skills library. There are 18 of them up right now, and the number keeps moving.
... and I am updating them everyday, like software.
➡️ The method underneath all of them is what my Writing with Machines course walks through.
Pay it now … or pay it latter
The structure tax hasn’t gone away ... you’ve got to pay it somewhere.
We refused to pay it for ourselves because the payoff was invisible. Now we’re paying it for our machines, and the research says most of what we’re writing down is the wrong half.
Not because structure doesn’t help, but because nobody’s decided what belongs in it and what’s just noise wearing a filename.
That’s the part I can actually do something about. So instead of shopping for a new app this school year, I’m going to build my own, and pay attention to what goes in it.
If you know anyone building content workflows that might be interested in knowing the research behind the work, send this their way.
Leave a comment and let me know how you are using AI for your knowledge base.
I’m Lance Cummings. Until next time — mind what your system is quietly deciding for you.
