I had five years and 218 Substack posts, exported as a pile of HTML files but nothing you could actually query. Here’s how I turned that archive into a real knowledge base, using structured approaches like taxonomies and batch processing.

If I’m honest, I’m not really someone who organizes as I go.

When I finally decided to try building my own LLM wiki as a folder of markdown files, I had accumulated 218 Substack posts.

(If you are new here, an LLM wiki is a simple, but organized knowledge base that you connect directly to your AI.)

Whenever I needed to reference or find an old post, I generally scrolled one post at a time on the Substack dashboard.

Of course, I always have a sense of my past writings. But without scrolling for twenty minutes, I could not tell you which post first made a given point, or whether I’d already made it three times under three different titles.

It is a problem that’s plagued me and sent me off looking for the perfect app.

I’ve written about the graveyard of note apps behind me ... Mem, Notion, Roam, Anytype.

Each one solved a different piece of the problem and then stalled out, because storage, editing, and AI access were all bundled into one product I’d eventually outgrow.

The fix was to stop treating those as one job. Separate the three layers, and you can swap each one as your workflow changes:

• Storage — a folder of markdown files.

• Structure — an editor that reads the files without adding its own syntax.

• Access — an AI pointed at the folder instead of owning it.

That was the theory. Substack was the first real test of it at scale.

Subscribe now

How I exported and organized 218 Substack posts

So I pulled all 218 posts off the internet and rebuilt them as a knowledge base.

Not a backup. Not an archive. Something an AI could actually query with precision.

The difference between a folder of old posts and a working wiki is more about what happens before you import anything.

At first, I thought this would be simple.

Substack has an export button, so I clicked it, downloaded the zip, and opened it up expecting at least text files that were ready to go.

Instead, all I saw was a bunch of HTML files. One per post. Each file still had web page artifacts like subscribe buttons and pull quote widgets all in markup.

Now I’m not a coder, and writing a script really isn’t my thing. That’s where Claude Cowork came in.

I described what I had and what I actually wanted, and it wrote a script that stripped the Substack widgets out first, then ran the real HTML-to-markdown conversion on what was left.

But stripping the HTML only got me markdown files. It didn’t get me a knowledge base. Just 218 loose text files sitting in a folder, no better organized than before.

So before I let anything touch the real vault, I had to decide what I was actually building.

• What counts as a domain.

• What tags mean

• What goes in the frontmatter of every single file, so an AI reading one of these posts later knows what it’s looking at or for.

Skip that step and the import is just digital hoarding with better formatting.

It just so happens, that I’d been reading Manny Silva’s book, Docs as Tests with AI, around the same time, and one idea stuck.

Don’t just trust that a piece of documentation does its job. Test it. Applied to 218 posts, that meant not dumping them all in at once and hoping everything got organized correctly.

Do them twenty at a time instead.

For each batch, Claude proposed a domain, a type, a set of tags, and handed the whole thing back as a table, one row per post that I could scan top to bottom to catch any major mismatches.

It was slow going for 218 posts, but that’s part of the point. A wrong classification, once it’s live, is worse than no classification at all, because errors simply scale with the knowledge base.

Once I approved a batch, Claude Cowork took the next step. A script picked up the approved table, wrote the frontmatter into each file, and moved it into its folder.

I checked the work, not every individual move.

Mapping content from the ground up

Doing this helped me better understand the nature of my content and how to build organization rules.

In a way, domains started clustering on their own.

I asked Claude to build a Map of Content for one of them, ai-and-writing, and it pulled ten clusters straight out of the classification tags. No hand-curation. All 91 files in that folder linked exactly once. Zero orphans.

To actually become a wiki, the content needs to be able to connect beyond just the folder structure. A MOC isn’t just a table of contents, but an actual map for important topics or key words that connect a level above folders.

I liked those MOCs enough to turn a few of them into their own Substack posts that point readers back through five years of writing organized by theme instead of by publish date. Something I’d been meaning to do for a while, and kept putting off, because doing it by hand is tedious.

None of this would matter if the schema only helped me look backward, though. What happens the next time I sit down to write something new?

I finished a draft a few weeks ago and ran it through a skill that scans a near-finished piece for link opportunities before it goes out. It doesn’t touch the writing itself. It just reads the draft, checks it against everything I’ve published before, and reports back where a backlink might be added.

It found three without me having to go digging anywhere. Every one of those 218 posts carries a published_url and a real domain and type in its frontmatter now, the same fields the batch tables assigned back when I was still checking classifications one table at a time.

The metadata I built for the import is what the skill reads to make the connection now.

That’s the actual payoff for all this. Not a tidier folder. A knowledge base that can cite itself and project to readers a more coherent body of work.

That skill is already up in my public Structured Skills Library, if you’re a paid subscriber and want to go pull it apart right now. I’m also sharing it directly here soon.

If you’ve got years of writing scattered across a platform’s servers, the export is the easy part. Substack will hand it to you in five minutes. The work is deciding what you actually want to be able to ask your own archive later.

Below is a short reference that paid subscribers can use to start this process themselves ... and of course, more specifics can be found in the Structured Skills Library.

Read more

Four papers published this year measure whether agent skills actually work, and they all complicate the common advice about skills. It’s not necessarily how long the skill is. Here’s what works and how to spot what doesn’t using structured approaches and information types together.

I’ve spent this summer building a structured knowledge base — not just files I upload into AI, but one that helps me manage and organize them.

This has been the ghost I’ve been chasing, mostly through a graveyard of personal knowledge management apps. Every single one ended up in chaos.

This year I stopped looking and started building my own instead.

One working piece of that is the pipeline connecting my knowledge base to Blue, the project tracker I schedule this newsletter through. I published the skill behind it in Context Lab a few weeks ago. It’s called cyborgs-briefing.

This week, though, I was deep in the unglamorous part — revising those skills, reorganizing the documentation, and figuring out which had quietly gone stale.

Subscribe now

What is a skill?

If you haven’t worked with these, it probably sounds more complicated than it actually is.

Basically, an agent skill is a plain markdown file, usually called SKILL.md. It sits in your folder and tells an AI agent how to do one specific job. You can also install them directly into AI tools like Claude and ChatGPT.

The format is simple and doesn’t really change across AI models.

A few lines of metadata at the top say what the skill is called and when it should be used. Below that are the instructions in natural language. A skill folder can also hold supporting files:

• scripts the agent can run,

• reference documents it can pull from,

• templates and examples.

But the SKILL.md file is the only required part.

What makes it different from a prompt is persistence and scope. A prompt is something you type once for one conversation. A skill is written down once and read every time the job comes up, by whatever agent happens to be doing it.

That makes it documentation, with all the maintenance problems documentation has always had.

Years ago I wrote about the anatomy of a prompt — how a good prompt separates task from context from content instead of mashing all three into one paragraph. A skill works the same but at a larger scale.

How my skills actually fit together

Here’s my drafting process, simplified:

1. A draft starts. Sometimes I write it from scratch, sometimes I ask AI to draft from my notes. Often it’s both.

2. I iterate on the draft. Often section by section or by task.

3. Skills run passes on it. One audits for AI-sounding language. One scans for missing links or citations. One polishes the opening and writes the AEO summary.

4. I revise by hand. This customizes the text. I also track my revision habits and use that to help AI improve its revision skill.

5. Publish. This is more than just posting on the web. I have AI incorporate meta-data and then organize it into my knowledge base.

But as I continue to fine tune how skills support this process, I’ve been looking for research on what actually makes a skill good.

The research has interesting findings, but there’s a blind spot.

What the research actually found

Li and colleagues’ SkillsBench is the most direct evidence available.

They tested 86 real tasks with skills mounted versus without. Average performance improved, but here’s what matters: agents cannot reliably write their own skills.

When models tried to generate skills for tasks they’d been given, performance dropped. The skills need to come from someone else — someone who understands the task deeply enough to document it.

Let’s call that person a writer.

More generally, the benchmark shows three patterns:

1. Skills can backfire. Some tasks got worse when a skill was attached. The skill prescribed a single heavyweight process with no way to opt out. It fired every time, even when it shouldn’t. A skill with no off switch can do more harm than no skill at all.

2. Focused beats comprehensive. Skills that tried to cover every edge case and scenario underperformed. Tightly scoped skills, with constraints clearly marked, outperformed ones that attempted to handle every variation.

3. The agent has to sort what it’s given. More guidance doesn’t mean better performance if the guidance is tangled. The agent has to separate rules from steps from examples before it can use any of them. That sorting is overhead.

Wu and Zhang frame skills as procedural knowledge: knowing-how, a different kind of content from knowing-what. And Jiang and colleagues formalize a skill as four parts:

• a condition for when it applies,

• a policy for what to do,

• a condition for when it’s done,

• and an interface so something else can call it.

Ling and colleagues took a different angle. They looked at what people actually publish: 40,285 skills in a public marketplace, heavily concentrated in software engineering. What they found was “widespread intent-level redundancy” — the same capability, rebuilt over and over under different names.

You need writers, not just models

In the end, agents can’t generate their own good skills. When a model tried to write a skill for itself, performance tanked. Skills need to come from a human who understands the task deeply enough to document it carefully.

The bottleneck isn’t the model—it’s you, the writer. And you need a method.

Every finding above is a symptom of the same underlying problem, and the papers keep describing it with the wrong instrument.

SkillsBench measured length. Not because length is the actual problem, but because that’s what you can measure automatically. But the real issue is five different kinds of content get stacked into one file.

That organization is work an actual human writer should have done.

I’ve argued before that a skill is documentation, not a prompt. Documentation has a methodology for this, built over decades. Content gets sorted by the job it does for the reader. This is how writers have always organized complex information.

I use mostly five types.

When I restructured my own pipeline, I applied this grid to every skill.

The cyborgs-briefing skill that orients a drafting session is pure Concept and Task, which explains what a briefing is, then walks through the steps.

The structured-note skill I published this week sorts incoming content by type before it goes into the knowledge base — reference notes, concept notes, principle notes, and so on.

Both work because every block has one clear job. Both fail instantly if you mix the types without thinking about how they all connect.

Under this lens, all three failures reported in this research can be traced back to missing Principles.

Jiang and colleagues describe a skill as needing an entry condition (when to start), a process (what to do), and an exit condition (when you’re done). Missing that exit condition? The agent keeps working past “good enough.”

That’s a missing Principle. There’s no rule for when to stop.

The thirteen tasks that got worse? Same thing.

The skill had no boundary. There was no Principle saying “don’t use me if X.” So it fired every time, even when wrong.

And Ling’s marketplace redundancy happens when Principles aren’t separated from Process. Every small variation in the rules spawns a whole new skill file instead of an update to a shared one.

Manny Silva arrives at the same place from the practitioner side. I reviewed his book, Docs as Tests with AI, here in May. He argues that a skill needs three things — entry criteria, process steps, and an output specification.

He also argues that documentation should load in layers.

1. Metadata first — so the agent knows if this skill even applies.

2. Instructions next — what to actually do.

3. Resources last — details you only need if a step calls for them.

Separating these out into organized chunks helps both humans and readers understand and implement these skills.

His framework maps neatly to the five types.

• Entry criteria are Principle.

• Process steps are Task.

• Output specification splits into Reference (the shape) and Principle (the standard it must meet).

I would add to this anatomy a Concept type, making sure the AI understands key ideas about the task that are contextual and may be different from its training or what it finds in RAG or on the web.

Tacit knowledge is one of the important things the benchmarks can’t measure.

In his book, Silva describes a skill built from everything a practitioner consciously documented about her process … and it still failed.

She performed one comparison step so automatically she never thought to mention it. No amount of type-checking catches what you don’t know you know. That takes testing against real work.

Try it yourself

Try this the next time you are working with a skill. Go through your skill draft and label every block with the job it’s doing.

• explanation (Concept),

• lookup data (Reference),

• a rule (Principle),

• a flow (Process),

• or a step (Task).

Anything that doesn’t fit one of those five is either in the wrong place or shouldn’t exist. Anything that mixes two types should be split.

That’s it. No length audit needed. No word count. Just ask what job each block is doing.

Next: Inside a failing skill

Now that we have the framework, the real question is simple. What does a failed skill look like up close?

I pulled a real one from the SkillsBench benchmark that lost 13 percentage points of performance.

In the next piece, I’ll walk through that skill block by block, showing you exactly where it breaks. Understanding the problem is one thing. Seeing it in action is another.

I would love to hear what skills you are working with in the comments!

Leave a comment

Sources

• Xiangyi Li et al., SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks (arXiv:2602.12670, 2026)

• Yanna Jiang et al., SoK: Agentic Skills — Beyond Tool Use in LLM Agents (arXiv:2602.20867, 2026)

• Yaxiong Wu and Yongyue Zhang, Agent Skills from the Perspective of Procedural Memory: A Survey (TechRxiv, 2026)

• George Ling, Shanshan Zhong, and Richard Huang, Agent Skills: A Data-Driven Analysis of Claude Skills (arXiv:2602.08004, 2026)

• Manny Silva, Docs as Tests with AI — my review

• citation-check task and citation-management skill, benchflow-ai/skillsbench (GitHub)

Sorting what goes into your knowledge base isn’t the same as sorting what it actually is. This post walks through the structured-note skill I use to turn a bullet journal line, a voice memo, or a half-read article into one note typed by information type (concept, reference, principle, process, task) with a real worked example and the skill itself to try.

Throw a handful of mixed seeds onto bare dirt and something will definitely come up. You just won’t know what, where, or whether it’s anything you actually wanted.

My markdown knowledge base isn’t run that way. I don’t dump everything from the notebook into it and trust AI to sort out what grows.

It’s closer to a garden. Every note gets sorted before it goes in the ground, placed where it actually belongs, and given the conditions that kind of knowledge needs to take root.

Last week I wrote about why a $12 notebook beats every AI note app I’ve tried at separating storage, structure, and access. I promised I’d come back with how I’m building the digital half of that system.

“Digital gardening” is a common term for this kind of bottom-up approach, and like any kind of gardening, requires the skill of identifying and sorting what goes where.

Not everything from the notebook goes into the markdown knowledge base. Every knowledge base needs a filter. But that’s only the first step.

Before a note goes in the ground, it has to be sorted by what it actually is, not just what it’s about. That’s the job of what I’ve been calling the structured-note agentic skill that takes something I’ve written, said out loud, or read, and turns it into a note organized by the work each piece of it is doing.

In case you’re a newbie, here are the five core information types:

• A concept explains what something is or why it matters.

• A reference is a fact you look up and move past.

• A principle is a rule you can follow or break.

• A process describes something unfolding that you watch rather than perform.

• A task is steps you actually carry out.

I’ve written about this framework before, so we’ll keep the brief so we can get on with the structured skill.

Subscribe now

But those are certainly not the only information types, especially if you gather knowledge about more than just documentation. For example, many things I research make arguments built on specific warrants. You can get even more granular with claim, grounds, warrant, backing, qualifier, rebuttal, counterclaim. The basic rhetorical building blocks of most arguments.

The other addition is metadata like purpose, audience, tags related to my work, etc. It’s not required to get the benefit of typing your notes, but it’s what lets an AI — or future me — find the right note without rereading everything first.

To help make this process more efficient ... because, yeah, it can be a lot of work ... I created a structured-note skill.

This is the skill that shapes a note before I plant it. Sometimes it’s a line in the bullet journal. Sometimes it’s something I talked through out loud into Pocket (referal), my voice-note app, on a walk. Sometimes it’s an article sitting in Readwise Reader, highlighted and half-forgotten.

A few months back, all three fed the same note. I’d jotted a line in the notebook about breaking big goals into short “Sprints,” talked through the idea into Pocket, then finally sat down with Ryder Carroll’s piece on the same idea and ran the whole thing through the skill. Here’s a condensed version of what came out:

## Reference: what a Sprint is and the three requirements

A Sprint is a short, self-contained project covering a subset of a
larger goal. Unlike a phase, a Sprint is an end in itself — it
produces its own satisfaction, information, or motivation, rather
than existing only to lead to the next step. A Sprint has to have
no major barrier to entry, break down into clearly actionable
tasks, and stay under a month, ideally one to two weeks.

## Process: how Sprints sustain motivation

A long-term goal pursued all at once behaves like a marathon — the
longer it takes, the more it drains patience, and drained patience
is what causes projects to get abandoned. A Sprint interrupts that
drain with frequent, visible checkpoints of progress.

Claim: Sprints protect motivation better than open-ended goals.
Grounds: one writer lost motivation when his task app erased each
item on completion, leaving nothing to show for finished work;
switching to a typewriter, where pages physically piled up, made
progress tangible again. Warrant: visible, accumulating progress
is what sustains motivation on projects with no external
accountability.

## Principle: rules for keeping a Sprint workable

Keep every Sprint under a month. Estimate how long it'll take,
then triple that number and plan against the tripled one. If you
don't have enough time to meet even the tripled estimate, shrink
the Sprint's scope — don't let the timeline slip.

This pattern carries throughout the note, recording how the Sprint cycle actually unfolds, the steps for setting one up.

That middle section used to be typed as Concept, not Process, until I ran it back through my decision guide. These distinctions are harder than they look, and AI gets them wrong too, which is the whole reason I built the guide in the first place. It’s in the public skills library if you want to try catching your own mistypes or use it for a reference for AI.

Now I can more clearly find the specific kind of information I need ... but so can AI.

If you’re interested in trying the skill yourself, all my skills are available in my public skills library for paid subscribers. Fully adaptable, and I’d love to hear what changes you’d make. I’m also sharing an installable bundle in that library, so you can plug and play if you just want to take it for a test spin.

That’s the planting done. What’s still ahead is the garden itself, for example how the notes actually sit together once they’re in the ground, what an entity is versus what an information type is, and what I’ve started noticing growing in there that I didn’t plant on purpose.

I’ll get into that next.

Curious what the sorting actually looks like under the hood? Below is the skill itself for paid subscribers, copy-paste ready.

Read more

Most AI note apps bundle storage, structure, and access into one product and get at least one wrong — a $12 bullet journal doesn't. Drawing on structured authoring and Ryder Carroll's method, this piece shows how to separate what stays in the notebook from what graduates into a structured, AI-searchable knowledge base.

The best PKM I’ve used doesn’t sync, doesn’t have an AI layer, and has never been acquired or sunset by a startup out of money.

It’s a $12 notebook ... and it solves the three problems that I wrote about in my last piece on knowledge management (and what most apps get wrong): storage, structure, and access.

I know I promised a deep dive into the anatomy of a skill not too long ago, and honestly, it was scheduled for today.

But work I’ve been doing with my bullet journal turned out to be more relevant to how I’m thinking about personal knowledge management (PKM) right now, so that piece is back in the queue.

In fact, I’d argue that if you actually want to understand how a PKM works, or how a skill should be structured, you have to start at the most basic level there is ... analog.

I don’t write about this much here, but a notebook has been my first-layer PKM tool off and on for years, specifically the bullet journal method.

I slipped away from it for a stretch, chasing what I was sure would be the perfect AI-powered PKM instead.

I wrote a Substack note about that particular rabbit hole, but it turned out to be more relevant to my explorations than I thought.

These days I’m using my bullet journal as my launch pad for experiments with local markdown knowledge bases for AI (which is where all this is going!).

Prepping for a bullet journal community panel a few weeks back is what got me thinking about this connection again.

But wait ... I’m not talking about the versions of bullet journal with stickers and washi tape you see all over Instagram. I mean the original method, the one Ryder Carroll actually designed.

If you’re looking for a PKM that gets all three parts right, start with this kind of bullet journal.

I spent last week’s piece cataloging every AI note app I’ve adopted and abandoned and how the root of the problem was my inability to control and fine tune the three layers of knowledge management:

• Storage, or where it lives.

• Structure, or how it’s organized.

• Access, or how AI actually reads it.

Almost nothing on the market treats them as three individual units.

If you think about it, analog has been getting this right for centuries ... and is probably why it sticks around no matter how far technology advances.

I teach and research professional writing at a university, but the part of my job that’s actually shaped how I keep that notebook has less to do with the classroom.

For the past several years I’ve worked in structured authoring, a field that mostly lives inside technical writing and enterprise content teams who write documentation for medical devices, aircraft manuals, software APIs, etc.

Many of you reading this newsletter already know what structured authoring is. If you don’t, that’s normal. It’s not really a productivity philosophy, but a way to think about how we build and use information.

Share

What is structured authoring

For those of you unfamiliar, here’s what that looks like in practice.

An aircraft maintenance manual has to hold a torque spec for a bolt, a warning about what happens if you skip a step, and the numbered procedure for replacing a part.

Those look like they could all live on the same page in paragraphs, and in a bad manual, they do.

But they’re three different kinds of information, doing three different jobs for the reader.

• The torque spec is something a mechanic looks up and moves on. It’s a fact, and it shouldn’t change based on context.

• The warning is a rule. Follow it or don’t, but it’s not a step in a sequence.

• The replacement procedure is something the mechanic actually performs one action at a time.

Collapse those three into one undifferentiated block of text, and the mechanic reaching for a torque value in the middle of a repair has to read past a warning and a procedure to find it.

Keep them typed separately, and each one is exactly where it needs to be the moment someone needs it.

Most PKM approaches rely on storage schemes.

They tell you which folder or list something belongs in. They don’t ask what the thing actually is.

Knowledge items have a purpose. A fact is something you’ll look up later. A rule you’re supposed to follow. A step is something you’re going to perform.

That’s where structured authoring begins. Knowing what kind of information something is will tell you where it goes. You don’t need fancy acronyms.

Long before I had language for any of this, I was already doing a rough version of it every morning with a pen in my bullet journal.

Bullet journal as a structured authoring tool

Ryder Carroll doesn’t call bullet journalling structured authoring.

But he built the notebook around structural moves that map onto three key elements of structured authoring and information design.

A single source of truth. Carroll’s rule is that everything lives in one place, in order, as it happens ... not scattered across sticky notes, apps, and the backs of receipts.

That’s the same principle behind a single-sourced content repository for an airplane mechanic or software company.

There should be one authoritative location for information, so nobody’s ever working from a stale copy or hunting across three systems for the current version.

For us, the notebook is the repository. Its authority comes from being the only place you go to first.

That’s the storage layer that I wrote about in my last post.

Modularity. Carroll compares the bujo system to a Lego set built from discrete pieces you can rearrange without rebuilding the whole thing.

The clearest version of this is rapid logging, where every entry gets marked with a symbol before you write it, and that symbol is really just an information type:

• a dot for something you’ll perform,

• a circle for a fact tied to a moment in time,

• a dash for an idea you’re not ready to classify yet.

Three different jobs, sorted at the point of capture instead of after the fact ... the same way you would build an airplane mechanic manual.

That’s the structure layer part of the knowledge system.

Nothing in the notebook is readable by anything except me until I decide otherwise.

There’s no export button pointed at a model, no ambient permission sitting open by default.

That’s the access layer. You control who and what sees that notebook.

Iterative assembly. Carroll didn’t design the system top-down. He built it one challenge at a time.

Structured content systems get built the same way, when they work. Nobody architects the whole taxonomy in advance and gets it right the first time.

The bullet journal is essentially a structured authoring system that happens to run on paper, assembled by someone solving his own problem without ever needing the technical vocabulary for what he was doing.

It also happens to get the hardest of the three jobs right, by refusing to solve it prematurely.

What this changes about how I actually use it

Knowing that the notebook is already doing this work changes how I use it. Most of what lands on a daily page never leaves.

A task gets done or migrated.

An event passes.

A note gets read once, maybe referenced for a week, and then its job is finished.

The bullet journal is built for things that are useful now and stop being useful soon after, which is most of what a day is made of.

It takes effort to move something from your bullet journal to a personal knowledge management system, so it better count.

This means it is something worth finding and using again some day. When that is the case, I ask myself three things:

• Who’s actually going to read this again?

• How will that “who” find it when they come looking?

• Is this settled, or still moving (something I need to review)?

A note that’s personal, dated, and closed the moment I write it fails all three on purpose, so it stays exactly where Carroll put it, unread by anyone else, human or AI.

A note that’s an idea I’ll want back, findable by topic instead of by day, passes the test and gets pulled out into something more durable.

This is what I call a structured note, filed by what it is rather than when I happened to think it.

That’s also where my tool choices start making sense instead of looking arbitrary.

The notebook stays paper, on purpose. It’s fast, it has no sync delay, and it never asks me to decide anything before I’m ready to.

The structured notes live as plain markdown files I control, not in a fancy proprietary app or format.

(If you don’t know what markdown files are, they are simply text files that have simple formatting markups. It is the most durable and machine-readable format.)

I draft and edit in Obsidian, because it’s really just an interface for the text files. I can move and manipulate my markdown files without working with a third party.

And finally, I use Claude to help maintain the structure once it exists. AI agents help me flag when a tag has drifted, when a note’s type doesn’t match its content, or when something’s been sitting unclassified too long.

(You’re not tied to any of these tools if you are building a knowledge base in Markdown. More on that to come as well.)

Each tool is doing one job, in one layer, and none of them is trying to be the single place everything lives.

The notebook already is that place where the first draft of everything exists.

The five jobs a note can do

You may still not be sure what I mean by information typing beyond bullet journal’s simple bullets.

I’ve written about this framework in more depth before, in Reading with Information Types and What Information Actually Is, but here’s the short version that will get you thinking about what kinds of information are in your knowledge base (not just where).

When something graduates out of the notebook, I don’t just decide it’s worth keeping. I decide what job it’s going to do for me later, because that’s what determines how it gets written and where it lives.

There are five jobs a piece of writing can do, and almost everything I’ve ever wanted to save falls into one of them:

A concept explains what something is or why it matters ... the kind of thing you read once to understand.

---
title: What a structured note is
type: Concept
topic: personal knowledge management
---
A structured note is a note typed by the job it does, not the day it was written.

A reference is a fact you look up and move on from ... a number, a name, a definition you’d never read start to finish.

---
title: Obsidian
type: Reference
topic: tools
---
Local markdown editor. No built-in AI layer. My "human interface" for drafting and editing.

A principle is a rule you can follow or break...a guideline, a standard, a “do this, not that.”

---
title: Type and content must match
type: Principle
topic: note maintenance
---
Never let a note's type and its content disagree. If you catch one, fix it before moving on.

A process describes how something happens, when you’re watching it unfold rather than doing it yourself.

---
title: How a note graduates
type: Process
topic: bullet journal workflow
---
A note gets written, sits in the notebook, and either closes for good or gets flagged to graduate.

A task is a set of steps you actually perform, in order, to get something done.

---
title: Graduating a note
type: Task
topic: bullet journal workflow
---
1. Reread the note.
2. Decide if it's settled.
3. Retype it.
4. File it by topic, not by date.

The block above each example is metadata. It’s the advanced version of this move. You don’t need it to get the benefit of typing your notes.

But once you’ve been doing this a while, tagging a note with its type and topic directly is what lets you find it later without remembering a single word of what you wrote.

You’re searching the label. No more guessing at the content.

You can also add context there for AI which helps it use your knowledge better.

Naming the type tells me the shape it needs. A table. A rule. A sequence. A short piece of prose. No more guessing.

There are more types that I’m exploring, but these are the basic ones serve as the core to any knowledge base.

Sounds like a lot of work ...

It isn’t, most days, but it’s worth being honest about why I bothered building it, and the honest answer has changed in the last few years.

I’ve been building a personal knowledge base that an AI assistant helps me maintain, and typing turns out to matter even more.

When I browse my own notes, I can tolerate some mess. I remember roughly what I meant, even when the label’s wrong.

An AI searching the same notes doesn’t have that memory to fall back on. It only finds what it’s told to look for, and untyped, undifferentiated notes give it nothing to search against except raw text and hope.

If the pattern happens to match, it might still work, but it’s much more random than you’d want.

If a note is typed, an AI assistant can tell the difference between a fact I’m citing and an idea I’m still testing, without me having to explain it every time I ask.

So if you plan on integrating AI into your knowledge base, it’s worth the work. But it’s one payoff among several, and I’d have kept doing this even without it.

A typed system is faster for me to search six months later, when I’ve forgotten the exact words I used but still remember what kind of thing I was writing.

It’s easier to hand off to a student, a collaborator, or future-me, who can open a note and know what to do with it without reading the whole thing first.

And it scales as my work grows.

Carroll built a system that already knew this intuitively, years before anyone needed to worry about what an AI could and couldn’t read.

That’s the irony, I suppose. The paper notebook is about as close to a perfect PKM as you can get, because it never tried to do all three jobs itself.

Next, I’ll be sharing how I’m building my local knowledge base of markdown files and how I’m developing the workflows that maintain the system.

The creator economy looks like a new phenomenon of the social media age but technical writers and content strategists have been running the same playbook for decades ... giving away useful knowledge to build trust. This piece traces that history from Robert Horn’s modular content to Ann Rockley’s “intelligent content,” and shows how the original creators also built the structured content today’s AI depends on.

Part 2 of a series. Part 1 asked whether AI is killing your professional association.

AI isn’t the biggest disruptor in today’s knowledge environments.

That’s a case I’ve made informally for a while.

The creator economy is more disruptive than AI, but often goes unaccounted for.

In part one, I argued that AI didn’t kill the Society for Technical Communication ... or any of the other numerous professional associations that are struggling or even gone at this point. For example, STC had been losing members for twenty years before ChatGPT existed.

In this post, I’m going to make a case for what I think has been killing professional associations.

In short, people changed where they went to learn their craft, and who they trusted to teach it.

As knowledge and content has become easier to create, share, and find through the internet and social media, professional communities have been quietly gathering around “creators,” or professionals who create useful content direct to audiences, often for free.

We call it the creator economy, though you may or may not have heard that term.

Associations and institutions that adapt to this new economy survive, even thrive. Those who don’t, disappear.

That’s my theory anyway.

Subscribe now

What is a “creator”

When I say creator, I don’t mean influencer.

The two get lumped together, but they are opposites in a lot of ways.

An influencer grabs attention and rents it out to businesses, often posting about products in exchange for a fee or a discount code.

A creator makes something people actually want, like a newsletter, a podcast, a video series, a course. The audience forms around the usefulness of the work itself, as well as the “ethos” or credibility of the creator.

Ethos is just an old rhetorical word for earned trust. A creator builds it by giving useful work away, not by holding a title or charging dues.

The creator economy is now an entire ecosystem of these independent producers in spaces like YouTube, Substack, Patreon, Linkedin, and many other spaces.

Goldman Sachs sized the creator economy at roughly 250 billion dollars in 2023, with tens of millions of people creating for a living. It has only grown since then.

No single platform holds it together. Instead of going to an institution or an association, people follow a person who has been generous with what they know.

You might think this is a phenomenon of the social media age with YouTube stars and the first full-time bloggers.

But if you look a little earlier, being a creator has been around for much longer.

Content strategy and the creator mindset

Tech writers are creators at heart ... actually all writers are, really. That’s what we do. Create useful content for specific audiences.

That’s most visible in the discipline of content strategy, which is the work of planning, writing, structuring, and maintaining everything an organization publishes so that it stays useful instead of collapsing into a mess.

Kristina Halvorson, who helped name the field, called it planning for “the creation, publication, and governance of useful, usable content.”

Someone has to decide what a company’s help articles, product guides, manuals, and support pages should say, how they’re organized, how the same explanation gets reused across a website and an app and a printed guide, and how all of it stays accurate over years.

That person is a content strategist, and at the heart of every content strategist is a creator.

If you’ve never touched a content team, stay with me. This is really a story about how expertise earns trust now, in any field.

All businesses and institutions need to grapple with how the creator economy changes the ways people engage with information.

Creators before there were creators

Go looking for where content strategy came from and you find something that resembles the creator economy, only earlier and mostly offline.

, a content strategist who wrote the field’s own history, traces its founders, and I would argue that their credentials and association memberships have little to do with their success.

If you are not in the field, you’ve probably never heard of these people. That’s kind of the point.

Ann Rockley, , Kristina Halvorson all wrote self-published books, ran their own consultancies, and started conferences that people paid their own way to attend.

They earned trust the way a creator does now, by giving useful work away until people came to depend on it.

For example, Rockley is called the mother of content strategy because she showed up year after year teaching people how to manage mountains of content. That’s how you build ethos in the creator economy.

Or take Scott Abel. Since 2003 he has run something called The Content Wrangler ... more than a hundred free webinars a year, over sixty thousand subscribers, no membership card and no dues. Just a person who kept showing up with something useful, until an audience gathered around him.

Trust used to be handed to you by an institution, a credential or an association or a byline in the right journal. Now you earn it in public by being useful.

Let’s go even further back.

In the 1960s a researcher named Robert Horn built a method he called Information Mapping. Instead of writing documents as long unbroken prose, he broke knowledge into small, labeled, reusable blocks, so a reader could find exactly the piece they needed.

Around the same stretch of years, Vannevar Bush had already imagined a personal machine for linking and sharing trails of knowledge, and Ted Nelson had coined the word hypertext for writing that readers move through by their own path rather than start to finish.

These were people treating content as something modular and connected, built for an audience to use on its own terms. Decades before anyone uploaded a video, technical communicators and other content professionals were designing knowledge to be reused and shared.

We sometimes called these people technical writers. But they were are just technical writers. That’s why we keep changing the title: information developer, content strategist, content designer, content engineer.

The work keeps outgrowing the word for it.

Sell without selling

Content people have a phrase for the core move used by creators ... sell without selling.

You make something so useful that nobody feels sold to. This could be a troubleshooting guide, a clear walkthrough, a genuinely good explainer. No pitch. The trust does the work, and the sale takes care of itself later.

It’s the whole logic of the creator economy, and it was being studied inside technical communication years before creators turned it into a business model.

In the 1990s a writing scholar named Johndan Johnson-Eilola argued that technical writers had been badly miscast as support staff, when their real work was creating and shaping knowledge, the same high-value work economists were celebrating everywhere else.

More recently, two researchers, Amanda Wall and Clay Spinuzzi, looked closely at how content marketing actually persuades and found that it works precisely by not pitching, for example a set of genuinely useful documents that walk a reader toward a decision on their own.

Content marketing isn’t the same as “content strategy” in the tech writing world, but that’s exactly how many of these influential thinkers have built the communities around their work.

“Give value first” is a creator mantra now ... but a whole profession had it worked out a generation earlier.

Building content for machines

The same instinct that made content people early creators also made them the people who taught content how to talk to machines.

Remember Horn’s modular blocks? I’ve been talking about it for a while. That idea matured over several decades.

By the 2000s it had hardened into a formal standard called DITA, a way of writing content in structured, labeled chunks that software can sort, assemble, and deliver on its own.

Write something once, structure it well, and it can then flow into a manual, a help page, a printed guide, or a search result. Content people call this single-sourcing, and creators reinvent a version of it every time they turn one idea into a post, a thread, and a video.

Ann Rockley gave this the name intelligent content. Content that is both human-friendly and machine-processable. Modular, structured, and rich enough that a machine can find it, recombine it, and reuse it.

Decades before generative AI, a group of technical writers were deliberately building knowledge that machines could read and reassemble. That turns out to be a large part of why today’s AI systems work at all, and it’s close to my own research. Structured content is what makes these tools genuinely useful instead of merely fluent.

It’s also why, when AI writing tools arrived, many content people dug right into testing while other writers braced for a threat.

This is a field already trained to see writing as a system. You design content with users, not only for them, and you work inside a network of templates, tools, and now models that help assemble the final thing.

If your craft already treats writing as something built with technology and other people, a tool that drafts and remixes is just one more part of the network. Another node.

But a language model can be fluent. It cannot earn ethos.

Trust still comes from a person who shows up again and again, giving something useful before asking for anything back.

That was always the real product … and its pretty difficult to automate.

Evolve or die

Everyone will need to adapt to the creator economy before they adapt to AI.

Content people built audiences on free, useful work. They theorized the give-value model of content. They engineered workflows. They spent decades preparing knowledge for machines to read, a valuable asset today that is often overlooked.

Nearly every attribute we now call “creator” was ordinary practice in a field most people picture as the writers of dull instruction manuals.

And it’s not just their story anymore. Every professional is now being asked to do what these people always did ... build trust by giving knowledge away.

If technical writers and content strategists were the original creators, why did the institutions built around them, like STC, fall apart, while creator-style communities in the very same field grew?

That’s the question I’m digging into for my current research project ... and part 3 of this series.

Leave a comment

Substack’s new Pangram AI detector is not a tool that promotes trust ... or even works. Here’s the actual eight-step process behind my writing, and why provenance, not detection, is the question that actually matters.

As many know already, Substack turned on an AI detector last week. I am disabling this tool for all my Substack posts, but not for the reason you might think.

Built with a company called Pangram, this tool lets anyone scan a post more easily and get back a percentage of how much it thinks a machine wrote. It’s on by default, live across the platform now.

I will be turning it off on everything I publish.

Not because I’m worried about what it would find. No one should be surprised that I use AI hard, and I’m not interested in hiding that. I turned it off because I disagree with the premise these detectors are built on (and, honestly, it’s a waste of tokens).

I don’t necessarily disagree with ’s idea of building trust. I just don’t think this tool feeds into that purpose.

I do find the “how I make this” statement an interesting addition, but am disappointed that it’s attached to an AI detector ... making it more of a “defense” than an honest share of workflows.

If you don’t know, this statement is a place to explain your process instead of just submitting to a score. It’s not supposed to be a policing mechanism or some kind of confession, but that’s how it will be perceived when attached to an AI detector.

While I can’t imagine anyone wants to make a habit of reading these, I did think the idea deserved a real post, so I have a single place to point people whenever it comes up.

Then I can get back to what actually interests me ... understanding what happens between writers and machines, not proving I’m on the right side of a percentage that really doesn’t mean much.

For example, I ran a completely AI-generated draft through Pangram’s scanner. It came back 100% human. Then I ran a piece I’d actually written, edited by hand over several passes, sentences rewritten three or four times. It came back 22% AI.

That’s the tool now sitting on every post on Substack.

That said, even a detector that worked perfectly would still be asking the wrong question.

But you should know that it doesn’t really work, anyway.

Why disable Pangram

Here’s the actual mechanism.

On the Publish page of any draft, there’s now a toggle: “Disable AI detection.” Click it, and readers see “AI detection unavailable” instead of a Pangram score when they try to scan the post.

I’ll be flipping that toggle on everything going forward, because I don’t want to support a system built on distrust.

Every scan Pangram runs on Substack almost certainly costs Substack something — a per-check API call to a third-party detection company maybe. Some of that cost gets covered by subscription revenue, mine or my readers’.

I don’t know Substack’s exact arrangement with Pangram, and they haven’t published it, but I don’t want a fraction of a cent of what someone pays to read me routed that way.

For many writing instructors at the university, this is not a new problem.

Turnitin is built on the same premise (even before it rolled out AI). We assume that students are cheating one way or the other and this poisons our interactions around writing with distrust on both sides.

You don’t build a detector for people you trust.

Substack’s value proposition was to build a social space based on human relationship, not algorithms — a network built on people choosing to read each other, not a machine-curated feed.

A detector doesn’t fit into that idea. It quietly replaces it with suspicion as the default posture, whether or not anyone ever gets flagged.

Humans should be the filter, not AI.

How I actually use AI

A detector starts from the assumption that something’s being hidden, and the job is to find it.

I never used Turnitin for plagiarism either, for the same reason. The tool assumes the student cheated and hands you a report to confirm or deny.

That posture doesn’t build trust.

So here’s my “how I make this” statement, offered to you freely, instead of some defense against AI detection.

It’s not really a formula, because my workflows change post to post depending on what I’m writing and how formed the thought already was.

But here’s roughly how this post happened, as an example:

1. I talked out my position on AI detectors into Pocket — a voice-journaling app walking while I thought.

2. That recording became a structured note in my own markdown knowledge base, organized by what kind of thing each part was: an argument, a fact, a story, etc.

3. I developed the thinking further by writing three separate LinkedIn posts responding to Substack’s rollout: one on the platform decision itself, one on the wrong question detectors ask, and one testing the detector on my own writing. Working through those made it clear the idea deserved more room than a microessay, so I decided to compose this post.

4. I had Claude pull up everything I’d already written about AI detectors, back to 2023, so this post could build on arguments I’d already made instead of reinventing them.

5. We built an outline together. Once I approved it, we drafted section by section — each one written, shown to me, and revised before it got committed to a new Markdown file. Nothing moved forward without my sign-off.

6. I ran the draft against a running note I keep of my own stylistic habits when I revise AI text, for example where I break a sentence, when I go conversational versus direct, which constructions I reach for and which ones I’ve banned myself from using too much.
   
   (If you want to see the related documentation, they now live in my skill library.)
   

7. I edited the final wording by hand.

8. Before it goes anywhere near Substack, I run it through a set of finalizing passes: checking for links back to my own previous posts and outside sources worth citing, reading it through the eyes of audiences who might drift or feel left out, and writing the search- and AI-search-facing summary and description that sit at the top of the post.

In nearly every instance I’ve checked, a piece that went through this whole process comes back 100% human anyway.

The real question

The real problem is that we conflate text generation with provenance or authorship. AI slop isn’t slop because there is AI involved, it’s because there isn’t any real authorship.

This is content posted just to post and manipulate the algorithm, not because someone worked through an idea. That’s a human problem that we created long before generative AI existed.

Medium’s Partner Program is a good example. It pays out on views regardless of how those views got earned. That’s created a real AI slop problem.

But nearly every online environment runs on some version of that same incentive: post more, post faster, optimize for the number instead of the thought.

AI just made it easier to do at scale.

Honestly, Substack has been an exception to this … and I think still is (but the AI detector doesn’t really help in any way I can see.)

Provenance is what a detector can’t see. It can’t trace words back to a specific person, at a specific time, making a specific decision to say this or that.

A quote has provenance when you can find the primary source.

A claim has provenance when someone can check it against what actually happened.

Writing has provenance when the person who published it can walk you back through where every piece of it came from, in their own record, dated before the piece existed.

That’s a testable property. A detector’s percentage isn’t.

A piece built for volume has no provenance, whether or not AI touched it.

Nobody wrote it down first, thought about it, decided it mattered enough to say. It was generated to fill a slot, and the slot doesn’t care where the words came from.

Ask me where any claim in this piece came from, and I can point to the dated note it started as in a markdown file I control.

Detecting “AI” measures the wrong variable.

Two pieces of writing can contain an identical percentage of machine-generated text and differ completely on whether either one can be traced back to a person who thought, decided, and stood behind those words.

AI makes this problem more difficult, but detectors won’t solve it.

How I make content (short version)

If you scan this post, or any post of mine, you’ll see it says detection is unavailable.

That’s on purpose.

Here’s the short version of everything above, written to stand on it’s own if this is the only section you read.

I use AI heavily — full drafts, structured outlines, revision passes built to catch my own patterns. It’s a full on agentic writing workflow.

Every idea I publish traces back to a dated note in a knowledge base, so I can show you where any claim in this post came from, not just tell you a machine didn’t write it.

I turned off Substack’s detector because I don’t want subscriber revenue funding a tool built on the same logic as the plagiarism detectors I’ve refused to use in my classroom for a decade.

I have nothing to hide. I just don’t want to fund a tool built for people you don’t trust, and that’s not how I want to relate to the people who read me.

If you want to know how something specific got made, ask. I’ll tell you exactly.

Leave a comment

If any of this made you curious about the actual mechanics (not just the argument, but how the workflow itself works) paid subscribers can take my course, Writing with Machines, free for a limited time.

Or go straight to the source: my live structured skills library documents the specific tools and workflows behind everything described above, updated as I revise them ... because no workflow is static.

Subscribe now

I‘ve spent years switching AI note apps, like Mem, Notion, Roam, Anytype, and each one failed in a different way. That’s because a knowledge base is actually doing three separate jobs: storage, structure, and access. Almost no app treats them as three.

Have you ever bought an app that promised to finally get you organized, used it faithfully for a few months, and then quietly let it slide back into the pile of apps you meant to use?

I’ve done this more times than I want to admit, and each time I told myself the problem was the tool.

It never was.

As exciting as these new apps are, no piece of software does everything you need it to do, perfectly, forever.

Something is always missing, or something almost works but not quite, and eventually you have to connect it to something else, or build the missing piece yourself.

This is exactly why workflows and stacks matter so much right now, for creators and for content professionals both.

Though many will claim otherwise, nobody’s shipping the one tool that does it all. That’s why you’ll often see people build their own combinations instead.

For example, you’ve found the perfect notes app, but handle your tasks somewhere else. Then to get AI to work with both you need to link it somehow. This is why you see so many “extensions” and “integrations” in spaces like Claude and ChatGPT.

Creating “app stacks” that match your workflow like this is a skill in and of itself that is becoming a job in many organizations.

I know tech writers and content designers who make an entire living doing exactly this for organizations, and it’s one of the things that fascinates me most about that world.

What most people don’t realize is that building these workflows is rhetorical, because workflows shape the spaces where we make meaning.

There’s no perfect app or perfect stack sitting out there waiting to be discovered, because choosing and connecting tools always depends on who’s actually going to use what you make, how it needs to move, and what it has to become later.

In the end, the environment you build shapes what you’re able to think, notice, and make inside it, long before AI ever enters the picture.

What AI does is raise the price of getting that wrong. A bad stack used to just cost you time and money locally. Now with AI it costs you at scale.

I’ve been doing a version of this for my own knowledge base for a few years now, exploring various approaches to personal knowledge management, or PKM — a term you may not have heard, but is attached to a category of apps you’ve almost certainly seen. Notion, Roam, Obsidian, Evernote, even a shoebox of index cards if you want to go back far enough are all PKM built to help you capture, organize, and retrieve what you know.

Subscribe now

Three invisible layers to modern knowledge apps

I’ve spent an embarrassing number of hours exploring personal knowledge management, because no single one of these apps has ever done everything I actually need.

The one I’ve been watching the longest, for about four years now, is Anytype — a newer PKM tool most people outside this world have never heard of.

Anytype treats your knowledge as objects instead of notes in a folder — things with real relationships to each other, private by default, but easy to open up the moment you actually want to share or collaborate.

My students use it for their writing portfolios because they can build the structure themselves, and they decide what becomes public.

What it still doesn’t really have is AI. It’s coming. There’s supposedly a working MCP integration now, though I haven’t managed to get it running cleanly on my end yet.

I actually respect how slowly and methodically the team has approached AI, but that can get frustrating at times.

But four years is a long wait, and it’s given me time to notice something else about Anytype. The same object-oriented control that makes it so good is also what makes it exhausting. Deciding what every single thing is, and how it relates to everything else, is real work.

Some weeks I don’t have it in me.

That’s probably the actual reason I haven’t moved my whole knowledge base over yet, more than any missing feature. The AI I’ve been waiting for is exactly the thing that would make that easier. It just isn’t here.

There’s a decent chance I move into Anytype fully once it is. The version of my knowledge base I’ve been building is portable enough that I could hand it to Anytype, or whatever comes after Anytype, without starting over.

But after thinking about it, I realized that this disappointment wasn’t random or even about a single tool.

An AI knowledge base is doing three separate jobs, and almost no product on the market treats them as three or does all three right.

1. There’s storage or where your knowledge actually lives, as files you control, not rows in someone else’s database you’d need an export tool to escape.

2. There’s structure or how you organize, browse, and edit it. Different apps have different approaches that determine what you can do. This is the pillar Anytype nails — and also the one that wears me out.

3. And there’s access or how AI actually reads what you’ve built. It should be swappable, since the model behind it is the piece most likely to change on somebody else’s schedule, not yours. This is the piece Anytype still doesn’t have.

Name the three jobs, and every tool I’ve tried turns out to be strong in one of these, weak in another, and often has no real answer for the third.

My graveyard of knowledge apps

Mem was the closest I came to AI integration, but only by sacrificing the other two..

I imported everything, and for several months I was genuinely impressed. It surfaced things I’d forgotten. Then an update shipped and it didn’t feel the same. I went looking for the knob that I could use to adjust how AI was working.

There wasn’t one.

The organizing scheme was determined by the developers, not me, and there was no way to change it. I ended up building projects in Claude instead, and stopped opening Mem.

I was also an early adopter of Notion, and put my entire life into it for awhile. Within a year and a half it had become a beast I didn’t want to open.

Not because the tool broke, but because the structure I’d built inside it had gotten so tangled that finding anything meant a lot of time and effort that I just kept avoiding.

Notion does storage, structure, and now its own AI layer, all under one roof, which sounds like the dream until you actually try to change just one of the layers.

I also tried Roam. For a while it felt like the answer to everything. I loved that there were no folders at all, just links, with the organization emerging from how ideas actually connected instead of where I’d decided to file them.

I loved it, at least for the first few months.

Then I noticed I was spending more time linking than thinking, more time maintaining the web than using it. There was always one more connection you could add, with the app itself having no opinion about when you’d done enough.

So far the tool that gets it right the most is Twos. I still use it, especially when I’m not sure where to put something.

The structure pillar here is deliberately thin. Twos never asked me to decide much, so I never built anything that could get tangled the way Notion did. You’ve got “lists” and you’ve got “things”. That’s it.

It has AI now and MCP connections that work well, but not many of the organizational “knobs” I might want in a complete knowledge base that I use with AI.

What’s missing in most knowledge apps

But all five tools turn out to be missing the one thing I’ve discovered to be key in developing knowledge bases for AI. Separation of these three layers for better control.

If the AI reading your files is a distinct layer from the files themselves, it can help maintain the structure instead of either ignoring it or owning it outright.

It can catch that two tags mean the same thing.

It can flag a folder that’s drifted from its own rule.

It can follow your instructions on how you want the knowledge base to work.

AI can do the unglamorous upkeep that turned my Notion into a beast and my Roam into a chore.

That’s work I always meant to get around to and never did, and doing it doesn’t require deciding for me what the structure should be in the first place.

That only works because the layer doing that work isn’t the same layer holding the knowledge.

Mem couldn’t offer me this, because it never separated the two to begin with. The AI wasn’t helping me maintain my structure. It was replacing my judgment about what the structure should be.

Anytype could offer me this eventually, once its AI layer actually works, precisely because the object model and the AI are being built as two separate pieces, not one piece trying to do both jobs. That’s the whole reason I’m still waiting instead of leaving.

But in the meantime, I’m building my own personal knowledge base one layer at a time.

1. Markdown files on my hard drive.

2. Obsidian as my “human interface” for drafting and editing.

3. Claude Cowork for maintaining and using the knowledge base.

And I can use markdown documents and skills in the knowledge base to keep guardrails on the AI and help me test and evaluate how the knowledge is working.

More explorations to come

This is the first of several pieces I’m planning to write on this exact problem, working through each pillar in more depth and exploring various structured approaches to building and maintaining an AI-ready personal knowledge base.

I’m already exploring how skills and information types make this workflow work better, as well as different ways of connecting AI to a markdown knowledge base.

Next, I’m pulling one of those skill files apart, line by line, to show what happens when nobody decides what belongs in it.

If you want to see some of that work, I’ve been building a public library of the skills I use for my own writing and content workflows — real, working files, organized by the same five information types I keep coming back to, available to paid subscribers to copy and adapt for their own setup.

It’s a live library, not a one-time download, so it keeps changing as my own practice does.

So ... of the three jobs your knowledge base is doing right now, which one have you never actually chosen? Where it lives, how you’ve structured it, or how AI gets to it?

Leave a comment. I’d genuinely like to know which failure modes you’re living with.

Leave a comment

I spent the summer building a markdown knowledge base that AI helps me manage, then went looking for research that would tell me it was worth it. Four studies across twenty-seven years, including the first real evaluation of AGENTS.md context files, suggest the organizing work is easier to demand than to justify.

I’ve got to be honest with you. I’ve spent most of this summer building workflows around a personal markdown knowledge base that AI helps me manage.

Tags, links, folder structure, naming conventions. All the housekeeping I’ve quietly avoided for twenty years. Everything I hate about managing my own notes is exactly what keeps sending me off to try one more app.

Surely there’s a perfect one out there. There isn’t. I know there isn’t.

I go looking anyway, about twice a year. For a brain that would rather build the system than sit down and use it, this is a dangerous hobby.

I’m Lance Cummings. And welcome to my now monthly podcast that explores deep research on AI and writing.

After all this, I decided it might be worth looking into what kind of research has been done on personal knowledge bases.

Oh, yeah ... there’s lots of creators out there giving out all sorts of ideas and frameworks, but what does the research say?

Here are some of my initial findings.

I looked at four papers, written across twenty-seven years, are all circling the same question from different sides: what does it actually cost to make your own knowledge explicit, and who ends up collecting?

That question used to have an easy answer.

You did the filing for future-you, future-you rarely showed up, and the whole thing quietly rotted. What’s changed is that my knowledge base has a second reader now. An AI assistant searches it, and it searches differently than I browse.

So it sounds like a reason to finally do the work of organizing. And it might be.

But you might want to hold on a second.

The structure nobody wants to do

In 1999, Frank Shipman and Cathy Marshall point out in their article [Formality Considered Harmful] that the real problem is motivation and intention.

After years of building systems people wouldn’t use, the real problem came down to something simpler than interface design. Users were unwilling or unable to make structure explicit.

If asked to declare what a thing was or how it related to other things (👋 information types), people balked, worked around it, or filled the field with garbage.

Not from laziness. The cost of formalizing is immediate and the payoff is speculative and later. You know what you mean right now. You are, at that moment, the worst possible judge of what future-you will need.

This is probably one reason it’s so hard for content strategists to convince stakeholders to structure their content.

Their solution really ties to how I’m working with AI to build a personal knowledge base. Rather than demanding structure up front, systems should offer incremental, system-assisted formalization. The system flags structure you never declared and helps you make it explicit over time.

In other words, build your knowledge from the ground up with help. Start small and use AI to help you discover the organization that’s already there.

That’s more or less what I’ve been building.

It’s also how I nearly lost two folders in my own vault this month, because an assistant had been quietly filing things into subfolders I never asked for.

I got system-assisted formalization, but I wasn’t paying close enough attention to what the AI agent was doing.

What folders and tags were both missing

Jump to 2008. Amy Civan, William Jones, Predrag Klasnja, and Harry Bruce ran a study to find out what kind of organization works best, focusing mostly on tags and folders.

There’s no winner.

Each model has strengths and weaknesses across keeping, organizing, and re-finding information. What study participants needed was a clearer framework for the organization scheme itself.

Think here about things like taxonomies, ontologies ... or simply deciding what goes where, why, and for whom ... and how that “what” gets found.

That’s more important than what tool to use.

What they wanted was a way to say what the categories meant, and to change their minds later without everything breaking. The tool or method doesn’t give you that.

I found this out the hard way.

I counted the tags in my vault a few weeks ago: 1,988 of them. At least forty-five were the same idea written in different ways — “rag” and “RAG,” “pkm” and “PKM.”

There’s a danger that a search or an AI scan might only turn up half the entries for tags like these. The tagging system worked fine. I just had nowhere to record that those two spellings meant one thing or master guide that gives clear guidelines (without guessing in the moment).

That gap is what I ended up filling with three plain markdown control documents: a taxonomy document, tag thesaurus, and workflow overview.

More on that to come in future posts.

How AI is changing the way we think about organization

You’ve probably seen the trend already. To better control agentic workflows, people are starting to do this formalization work voluntarily — not in note-taking apps, but in code repositories, as context files for AI agents, files like AGENTS.md, CLAUDE.md, the README written for a machine.

But there are a few problems arising because developers are mostly controlling what should be called “documentation,” which needs a content specialist.

A 2025 preprint, Agent READMEs by Chatlatanagulchai and colleagues, studied 2,303 of these files across 1,925 repositories. What they found is that these files get maintained the way a build script gets maintained: small commits, constantly, forever.

They grow harder to read as they go. Nobody sits down and revises one.

The content skews hard toward the function ... not context:

• implementation details — 69.9%

• architecture — 67.7%

• build and run commands — 62.3%

• security — 14.5%

• performance — 14.5%

So we tell the machine how to operate the thing, and almost never what the thing is for.

In the content world, we call this the “structure tax.” Maybe you don’t pay a price right now for structuring your knowledge ... but you will later.

That’s why this work is an investment, whether you are a company, a university, or just a solo creator.

The structure tax people refused to pay for themselves, they’re now paying for an agent. [pause]

The part that stopped me

Here’s where I have to complicate my own summer.

The obvious conclusion from all of this is that context files work, so write them. That’s certainly what I assumed while building mine.

Then in February, Thibaud Gloaguen, Niels Mündler, and colleagues at ETH Zurich published Evaluating AGENTS.md, the first rigorous test I know of. They ran coding agents three ways: with no context file, with one the agent generated for itself, and with one a human developer had actually written.

The machine-generated files didn’t help.

They made things slightly worse, and raised inference cost by more than twenty percent. The human-written ones were the only version that came out ahead, and they beat the machine’s version by a clear margin.

The temptation is to have AI generate this documentation for you ... its lots of work. Turns out that could be doing more harm than no context file at all.

But consider this carefully, the agents did follow the files.

When a file named a specific tool, the agent used it almost every time; when it didn’t, almost never. The instructions worked.

What didn’t work was the one thing every official recommendation tells you to put first: the repository overview. It’s the biggest chunk of these files, and it did nothing. Agents found what they needed just as fast without it.

In the end, this is a failure of content strategy. An overview is something the machine can already work out by looking around, but principles and processes needed to properly do that task ... not so much.

The files contained clear information types beyond just an overview. But, honestly, nobody in that study tested a context file designed that way. They varied the model, the prompt, the length, whole categories of content, and none of it moved the needle, because they were all the wrong knobs.

The one variable that mattered was a human deciding what actually belonged in the context file.

Two honest limits to these studies.

Both agent papers are preprints, not yet peer-reviewed. And they measure code tasks in repositories, a long way from retrieval in a personal knowledge base. I don’t think the finding transfers cleanly.

But nobody has run the knowledge-base version of that study, which is roughly where the next several weeks of this newsletter are headed. Stay tuned.

Takeaways for writers and content professionals

To be clear, this does not mean we should “stop organizing your notes.” Rather, we need to pay more attention to how we do it, especially when AI is part of the picture.

So the four things below are the ones I’d still do this week.

Write control documents, not more folders. The 2008 finding is the durable one. Before you reorganize anything, write down what your categories mean and which synonyms resolve to which preferred term. My tag thesaurus took forty minutes and fixed years of broken searches. No new app required.

Audit what your assistant has been organizing for you. If AI helps maintain your notes, run a structure check on purpose, full depth, no shortcuts. Incremental formalization is genuinely useful and it’s also invisible by default. I nearly lost two folders to that.

Treat your context file as configuration, and version it. If it evolves like code, maintain it like code. Small commits, a changelog, and periodic pruning. Files that grow without eyeballs get harder to read and control.

A/B Test your workflows. Run a task with your context file and without it. This is the least fun recommendation but can help you see whether a specific skill or documentation is actually needed..

If you want the practical side of this, the skills that run my knowledge-base-to-Blue drafting pipeline are posted in my public skills library. There are 18 of them up right now, and the number keeps moving.

... and I am updating them everyday, like software.

➡️ The method underneath all of them is what my Writing with Machines course walks through.

Pay it now … or pay it latter

The structure tax hasn’t gone away ... you’ve got to pay it somewhere.

We refused to pay it for ourselves because the payoff was invisible. Now we’re paying it for our machines, and the research says most of what we’re writing down is the wrong half.

Not because structure doesn’t help, but because nobody’s decided what belongs in it and what’s just noise wearing a filename.

That’s the part I can actually do something about. So instead of shopping for a new app this school year, I’m going to build my own, and pay attention to what goes in it.

If you know anyone building content workflows that might be interested in knowing the research behind the work, send this their way.

Leave a comment and let me know how you are using AI for your knowledge base.

I’m Lance Cummings. Until next time — mind what your system is quietly deciding for you.

The Society for Technical Communication, the Special Libraries Association, and other professional groups are shutting down or cutting deep, and AI is the easy explanation. The membership and revenue data tell a different story — one that started twenty years before ChatGPT and explains why some associations are dying while others are growing.

Is AI killing your professional association? It’s an easy question to ask right now.

The Society for Technical Communication (STC) lasted 72 years and had 25,000 members at its peak. On January 29, 2025, it announced its shutdown. Two months later, the Special Libraries Association followed. The Society of Professional Journalists cancelled its annual convention and named a $391,000 deficit. The Public Relations Society of America’s 2026 board chair told members plainly: “We are at a crossroads.”

If you work in tech, you probably heard about STC. If you work in journalism, publishing, library science, or communications, you may have watched something similar happen closer to home. If you’re in a field that hasn’t felt it yet … it may be coming.

The easy explanation is AI. And there’s something to it.

When staying current in your field means creating a prompt instead of attending a conference or reading a gated newsletter, the case for annual dues gets harder to make. But STC’s membership had been falling since 2002. That’s twenty years before ChatGPT. IRS filings show negative net assets every year from 2013 onward.

Whatever killed these organizations, it didn’t arrive in 2023.

I’ve been making a similar argument about higher education.

Assessment was already broken; AI just exposed it. Grades, assignments, and credentials built to gatekeep access to knowledge stopped making sense the moment that knowledge became freely available. Professional associations hit the same problem, just on a different timeline.

I’m researching this now by surveying practitioners across fields in technical communication about what professional community actually means to them, and what they do when the institution behind it disappears.

More on what I’m finding in the posts ahead.

Subscribe now

The numbers tell a slow story

I’d actually been meaning to join STC for a few years.

Every time I got around to it, another no-dues community already had what I needed, so I never signed up. I wasn’t trying to make a point. I just never got around to it. It would seem a lot of technical communicators were making the same quiet choice.

The Society for Technical Communication peaked at roughly 25,000 members in the late 1990s and spent the next two decades bleeding them out. A former board member who tracked the data himself estimates membership would have fallen below 1,000 by the time the doors closed. The organization stopped publicly reporting membership figures after 2007, which tells you something about what those figures looked like.

The finances follow the same arc: revenue fell by more than half between 2011 and 2023, and the organization carried negative net assets every year for the last decade of its existence. The chart below has the full numbers.

What STC was selling made sense when those things were genuinely scarce. You paid dues because that’s how you got the journal, attended the conference, and earned the certification that signaled your competence to employers.

The internet didn’t destroy that value overnight, but it spent twenty years making it cheaper and cheaper to get elsewhere.

Free forums, YouTube tutorials, LinkedIn groups, and eventually entire communities built around specific tools and workflows pulled the same audience STC needed to survive.

AI maybe accelerated the last leg of that erosion, but it didn’t start it.

It’s not just STC

Individual-membership nonprofits that peaked in the late 1990s, built their revenue around dues and an annual conference, and offered credentials and a journal as their core value proposition are under pressure across professional fields.

Some are collapsing. Some are cutting. A few are doing something different.

Two months after STC closed, the Special Libraries Association announced it couldn’t pay its bills either. It had 14,000 members at its peak, a journal, and an annual conference, just like STC. Its members voted to dissolve rather than declare bankruptcy, merging into a larger information-science body by early 2026.

The Society of Professional Journalists cancelled its 2024 national convention, its treasurer telling reporters the organization faced a projected $391,000 deficit. PRSA, the leading professional association for public relations, opened 2026 with its board chair telling members that “meaningful change must occur for us to thrive for another 79 years.”

Not every association is dying. ACES: The Society for Editing has been growing. The Editorial Freelancers Association is growing. Meanwhile, the Association for Talent Development reported $53 million in revenue last year.

What did they do differently?

What this means for your field

So is AI killing your professional association?

Probably not directly.

But it may be the thing that finally made the problem impossible to ignore. That’s the same way it made certain approaches to teaching and assessment impossible to ignore.

What that means for your field, your professional identity, and where community goes from here is what I’m trying to understand.

I’m currently surveying practitioners across fields in technical communication about what professional community means to them now, and what they’ve lost or found since their associations started struggling.

If any of this lands for you, I’d genuinely like to hear from you.

The survey takes about 5-10 minutes.

➡️ Click here to take the survey.

Most skill or prompt libraries are dead the day they’re published. My Structured Skills Library is different: a live, self-updating collection of AI writing and teaching skills, organized by five information types, with copy-paste templates you can adapt to your own notes, tools, and voice.

Too many “prompt libraries” or “skill libraries” are graveyards … a pile of prompts or skills someone wrote once and never touched again.

This library is the opposite. The Structured Skills Library is the live set of skills I use in my own writing and teaching workflows, and it updates the moment I change one.

Each skill is a small program written in plain language, organized by five information types, providing structure for both human and machines to better access, perform and evaluate these skills.

You can read any of them like a short guide, or copy the whole thing and adapt it to your own work.

Available Workflows

As I mentioned, this library is live and constantly changing. As of July 2026, the following workflows and associated skills can be found:

• Content Creation — draft, revise, and publish

• Course Design & Teaching — build courses from scratch

• Vault Setup & Import — bring a scattered note backlog into a structured vault

• Miscellaneous — general-purpose tools for projects and context

There might be more, because I’ll probably forget to update this. (No live connection to Substack. 😆)

How it runs

The library mostly maintains itself. Every skill lives as a markdown file in a knowledge base on my own machine.

When I write a new one, an AI drafts a public version, tags it, links it to its neighbors, and files it in the library.

When I revise the working skill, the public copy updates to match. Every so often I run an audit that checks the whole thing against my own written standards, making sure links, structure, and voice don’t drift.

The result publishes straight from Obsidian, so what you’re reading is never stale.

If this intrigues you and you haven’t yet built your own local AI knowledge base, stay tuned. more to come on that front.

Subscribe now

What you’ll find

• A readable page for every skill: what it does, when to reach for it, and the thinking behind it.

• A copy-paste template inside each page, complete and working, with the personal parts marked so you know exactly what to change.

• Skills grouped by workflow: drafting and revision, course design, and general knowledge work.

• A short guide to the information types themselves, so you can build your own skills the same way.

How to use it

Start with whatever problem is in front of you, for example:

• auditing a draft for AI patterns

• synthesizing your notes

• planning a week of class

• handing a long chat off to a fresh session

There’s a skill for each, and each one tells you when it fits. Read the page to understand it, copy the template when you want to run it, and change the bracketed parts to point at your own files, tools, and voice.

You don’t need my setup to use these. You need your own standards written down, and the library shows you how I wrote down mine.

➡️ If you don’t know what those standards are or how to develop them, check out my course Writing with Machines.

What it costs me

The money side is smaller than people expect, and it doesn’t have to be Claude at all. I run it on Claude Cowork, the desktop tool I use to build and update the library, comes with the Pro plan at about $17–20 a month.

The only other tool is Obsidian, for the notes and the publishing. I do pay around $100 a year for syncing and the publishing tool.

But nothing is locked to one company. That’s what I love about it.

The whole system is plain Markdown files, so you can put any agentic model on top of it.

It already works with ChatGPT’s work mode, and it will run on free, open-source models on your own machine — slower, probably, but free. Conceivably you could do this whole thing for nothing.

(More on local, free setups in a future issue.)

What it actually takes

But the subscription isn’t the real cost. The real cost is thinking.

This library didn’t fall out of the machine finished. It came from a long back-and-forth. I had to redefine what a “skill” even is and making sure information types were correct … something that is oddly difficult even for machines.

The audit and the templates exist because I made the same mistakes often enough to name them.

This is what Manny Silva calls “docs as tests.” You explain your standard, the machine tries, you correct it, and slowly your standard becomes something a machine can actually run on.

The whole thing took me about a day to get up and running, and most of that day went to deciding what my standards were, not fighting the tools.

After that, the work compounds. Once the standard is written down, each new skill is faster than the last, and the machine handles the tedious parts, the metadata and links and consistency checks, while I keep the judgment.

More on all of this to come for everyone, but paid subscribers can check out the library using the form below. ⬇️

Read more

Skills are the backbone of any AI or agentic workflow ... think of it as documentation for your agents (not just a prompt). In this post, I share the actual structured skill I run before every Substack draft, organized by information type and connected to a Markdown knowledge base that makes it work.

Last week I wrote about Slaying the Churnivore with AI Workflows, a visualization of a content plan I’m developing with Claude’s help for my Substack. What I didn’t get into is the layer underneath and thought today ... why not share the skill!

For the past few weeks, I’ve been importing, structuring, and labeling my entire online back-catalog of writing into a Markdown knowledge base connected directly to Claude through Cowork.

This is a knowledge base AI can actually read, navigate, and write back to, because every file in it is labeled the same way I’m about to describe.

That project is too big for one post. Stay tuned. But the skill I want to show you today is a small, working example of what that knowledge base makes possible:

• It can read my guiding content plan directly, and update it when something changes.

• It can review my whole back-catalog to help me think about domains, topics, and where the yearlong arc is going.

• It can find other posts I’ve written and link them into new writing without me searching for them or even remembering they exist.

That last one just happened. While finalizing this post (with a structured skill), Claude linked back to A Skill Isn’t a Prompt. It’s Documentation. without me asking it to look for anything. I didn’t have to remember the exact title. I didn’t go find the URL. The knowledge base already had that information.

A skill is documentation, not a prompt

A skill is a short, saved set of instructions Claude reads before doing a specific job — the same way a process document orients a new hire instead of re-explaining the company from scratch every morning. While skills have some similar characteristics as a prompt, its quite a bit more than that.

The skill I run before every Cyborgs Writing session is called cyborgs-briefing. Its only job is to open my content plan, work out where I actually am, and hand me a short briefing before I start typing:

• current territory,

• how many posts are published,

• what’s next,

• what’s sitting half-drafted, and

• what’s still waiting in the ideas inbox.

Then it asks me one question to help me get started. I don’t need to load a project or cut and past context, or even work with a prompt.

For an ADHD mind that gets overwhelmed by over-planning, this helps me move forward today.

Subscribe now

Process and task aren’t the same thing

The skill below is organized by information type: concept, reference, principle, process, task.

These are the same labels I use across the whole knowledge base, on notes and skills alike. Most of those are self-explanatory. Two of them get confused constantly.

A process is an overview of the workflow, which provides context for the task. Without this, the tasks have less meaning.

A task is the specific instruction to run that procedure right now. Exact steps that the agent uses to accomplish the goal.

Collapsing the two makes it harder to revise and could also cause problems with the AI workflow.

➡️ Paid subscribers can see the structured prompt below ... but I’m also building a public, living version of every skill in this knowledge base, so the library updates as I revise instead of going stale.

For now, that library lives in a password-protected Obsidian vault. Paid subscribers can request access below.

Right now there’s 15 skills, but will be growing.

Read more

I’ve spent summers building Substack without a system, and it always breaks the same way: drafts accumulate, ideas scatter, deadlines slip when the semester hits. This year I built a visual map called The Churnivore, a project board for tracking status, and a Markdown archive to turn my scattered thinking into a legible, sustainable workflow where AI can actually participate.

This is the first summer in a long time when I am mostly free.

I’m not teaching.

I’m not buried in consulting.

I have research to work on, but I also have room to build things I’ve wanted for awhile:

• a local AI knowledge base,

• a personal RAG system,

• and a better content workflow.

And, of course, time to write.

That last part is where things can really get overwhelming for me.

I have no shortage of ideas for Cyborgs Writing. Too many ideas has usually been the problem. My ADHD brain is good at generating possible posts, series, experiments, frameworks, and rabbit holes. It is less good at turning all of that motion into a publishing plan that survives more than a few weeks.

So I tend to go with the flow.

That works for a while. Sometimes it works better than planning, because I’m consistently engaged and discovering new thoughts and ideas. So I don’t necessarily want to flatten that out entirely.

But a Substack eventually becomes something you have to manage, especially when the busyness of a semester comes barreling into my writing habits.

Drafts accumulate.

Notes scatter.

Themes recur.

Arguments get halfway built and then left open.

Readers also have a reasonable expectation that you will show up again next week.

And somewhere in the distance, next next year’s dragon is already coming down the tracks.

The Map

So this summer I’m using the open space to do something I usually avoid. Build a content plan.

I don’t really like planning in the abstract or ticking boxes. Administrative rituals too often remind me I have failed to perform or that I’ve missed something (or how much I still have to do).

Any system that depends on my dutiful attention to tiny status updates will eventually break.

So I drew a monster.

The monster is called The Churnivore. It sits at the end of a winding path, guarding a pile of gold that represents a modest but meaningful goal of building Cyborgs Writing into a sustainable publication over the next year.

The path has five regions. Every week between now and next July is a milestone. Each published post moves me a little farther along the trail.

It looks a bit ridiculous, but hopefully that’s why it’ll work.

A content calendar tells me what is supposed to happen. The map tells me where today sits in a larger arc. It turns the year into a quest instead of a spreadsheet. I can see which themes are moving, which ones are neglected, and how much of the path is still ahead.

And, ultimately, I’m hoping it gets me ahead of schedule, instead of writing “just-in-time.”

The Workflow Behind the Map

But the map can’t carry the whole workflow, or else it becomes another project-management chore.

The actual workflow lives in a project board. I use Blue to track whether a post is an idea, in progress, scheduled, or published. A draft ready for human editing gets a card. A scheduled post moves forward. A published post gets its link added back to the system, then archived in my Markdown knowledge base.

The map follows this board (connected to Claude via an MCP).

I’ve also created a complete Markdown archive of all my writing, where the publish versions are stored with all the appropriate metadata like dates, related ideas, and reusable components.

Each part protects a different kind of work. Production belongs on the board. The year planning belongs on the map. The reusable thinking is in the knowledge base.

AI can participate in that system without being asked to invent the whole thing from scratch.

That has become one of my main lessons from working with AI over the past year. The model is rarely the only variable that matters. The structure around the model often matters more.

A post with a status, pillar, date, and link becomes readable to the system. A published piece with a predictable home can be archived. Ideas tied to the map are easier to retrieve than ideas scattered across half my digital life.

The AI is not making the system smart.

The system is making the work legible.

The Five Regions

The map has five regions, each representing a theme I’ll explore this year:

• KB Building — Markdown knowledge bases, personal archives, and systems for making your own thinking retrievable.

• Info Types — Structured prompting, information architecture, and designing content that AI can actually use.

• AI Portfolios — How writers, students, and professionals can document and demonstrate their work with AI.

• World-Building — Collaborative fiction, interactive storytelling, and AI as a creative partner.

• Religion — Faith, meaning-making, and the deeper questions that emerge when machines start to think.

For now, I want to mark the shift.

I am not trying to publish more by sheer force of will.

I am trying to build a system that can hold more of the work than my attention can hold on its own.

The Churnivore is not really a monster outside the system. It is what happens when you don’t have a plan.

This year, I’m walking the path. The Churnivore is waiting. But for the first time, so is a map … that is, a structured plan.

Draft a Claude scheduled task in one chat connected to your knowledge base, structured by information type (concept, reference, principle, process, task), then paste the finished prompt into the scheduler on Sonnet instead of a premium model. Helps save tokens and get more consistent outputs.

Anthropic gave everyone double usage for a while, mostly so people would go try Claude Fable.

I’ve hardly touched it.

I’ve spent the extra tokens building whole new workflows instead, ones that fit how I actually work rather than what a chat interface assumes I want.

For example, I went back to my bullet journal a few months ago, after concluding that AI had mostly failed me on task management. For those who don’t know, this is simply a structured approach to managing one’s life in a notebook.

So, yeah, I’m somewhat analog now. Digital capture is frictionless, and for a mind that already generates too many open loops, that can be dangerous. Everything gets captured, and nothing gets done. “AI will handle it” quietly becomes a way to avoid deciding what actually matters.

The bullet journal’s whole value is the opposite. Writing something down by hand is enough friction that you have to make a judgment call when you write things down.

So when I started thinking about where AI could actually help, I didn’t want something that writes in the journal for me or decides what belongs there. I wanted something that clears the noise before I sit down with it.

So I created a Claude task that builds a short digest telling me which open tasks are genuinely mine, making sure I’m looking at the right handful of things when I get there.

Building that meant writing my own task rather than adopting whatever some productivity app already ships.

A system built to fit how you actually think beats one built around someone else’s assumptions.

It turns out there’s a bonus, though ... I can build something that uses less tokens.

Share

Where the tokens actually go

You can see details on this process in the video above, but here is the gist.

My first attempt used Claude’s task creation tool directly: describe what I want, watch it build, revise, revise again.

Opus and Fable are good at that kind of conversation, and that’s exactly the problem. A good model in an open-ended chat invites you to keep going.

I’d tweak a sentence, reconsider a rule, add an edge case I’d just thought of, and burn through a session just exploring.

The result wasn’t bad, but I was paying premium-model rates to have the same argument with myself six times.

Somewhere in that process I noticed that the actual task, the file the scheduler runs every morning, doesn’t need any of that back-and-forth once it’s written. It needs to be unambiguous.

A cheap model executes unambiguous instructions about as well as an expensive one does. The expensive model’s advantage only shows up while the instructions themselves are still being figured out.

In fact, I just tested out my task with Haiku … the cheapest model. It ran just fine!

Writing a task and running a task are two different jobs. Only one of them benefits from a better model.

The two-chat method

So I split the work into two separate conversations, and I’ve kept doing it since.

1. Open a clean chat connected to my knowledge base and think through what the task actually needs to do, using the same information types I use for everything else I write:

   1. what it is (concept),

   2. the facts it depends on (reference),

   3. the rules that govern it (principle),

   4. the steps it follows (process), and

   5. The instruction to run it (task).

2. Once the task reads clean, copy the whole thing as markdown and paste it into the task creator, set to Sonnet (Or Haiku).

3. Let it run. If it needs revising, go back to the first chat, make the change there, and paste the update into the scheduler. The scheduler chat never does any thinking of its own.

Step one only works because there’s an actual knowledge base behind it. I’ve been rebuilding mine inside Claude’s Cowork, giving Claude direct read and write access to a real folder of structured notes instead of pasting context into a chat by hand.

More on that to come, for sure.

The task now costs less every time it runs, and I’ve stopped spending a bunch of tokens arguing with the AI.

What actually makes this work on a cheap model

It isn’t the model choice by itself. A vague task on Sonnet still produces vague results.

The information-type structure is what lets a cheaper model perform like a more expensive one. Every fact the task needs sits under reference, so nothing has to be inferred. Every rule sits under principle, stated once, not scattered across examples.

The process is a numbered sequence, not a paragraph the model has to parse for implied order. None of that requires reasoning. It just requires the model to follow what’s already been decided.

That’s the same argument I made three years ago in “Why You Shouldn’t Be Writing a New Prompt Every Time”. Prompts behave like content, and structured content travels better than content that’s merely well-intentioned.

Also, check how this works with skills (basically reusable prompts for agents) “A Skill Isn’t a Prompt. It’s Documentation.”

Scheduled tasks just make the cost of skipping that step visible, because you pay it every morning instead of once.

The worked example

Below is the actual task behind my bullet-journal refresh, cleaned up so you can see its shape: what counts as an “intention” in my system, and where the line sits between what the task writes and what it leaves alone.

I’ve swapped anything personal for generic placeholders. Your vault won’t have the same folder names or file paths, but the structure underneath will look the same no matter what you’re tracking.

The full task is below for paid subscribers, the same markdown I paste into the scheduler, every section still labeled by information type. Copy it, run it against your own morning routine, and tell me what you’d change.

I’m especially curious whether anyone’s already built something like this and what broke the first time they ran it.

Tools referenced

• Twos — the notes and tasks app behind the “possible task” routing. twosapp.com (referral link — appreciate it if you use it)

• Twos MCP — Twos has an official MCP server (setup walkthrough here) that connects it directly to Claude

• Apple Mail MCP — Patrick Freyer’s apple-mail-mcp, the connector that gives Claude read, search, and compose access to Mail.app

Read more

I’m giving a webinar this month for a group of graduate researchers who want to use AI without letting it write their work for them.

This became the perfect use case to dig deeper into using information types when researching and writing with AI.

Before AI can help you think, you have to organize what you know.

Here’s the short version of the webinar, in case it’s useful framing. It might be worth a longer post later after the webinar, but its a preview of whats to come.

I’m currently working on structured approaches for managing personal knowledge using markdown files and agentic AI. Stay tuned … coming soon!

Most AI training starts with tools. Which app? Which prompt?

The real place to start is the structure of what you feed it.

If you sort your reading notes by the job each piece does (or the questions they answer), then you get notes you can find, build on, and write from. The fact that AI also performs far better on top of those notes is a bonus.

Now the method itself.

Subscribe now

What happens to everything you read

Here how research notes often go … at least for me, and probably a lot of graduate students.

You read an article, highlight what looks important, and drop a few quotes into a document.

Maybe you use a separate notes app to help you organize … maybe not. Either way, the pile grows.

Months later you’re searching the same PDFs again or an idea you remember seeing but not sure where. But the highlight or cut and paste never told you what job the note was doing, and the connections you noticed while reading have gone cold.

It used to be the only risk to disorganization was either frustration or more time looking for things then actually drafting.

But that’s not the only risk anymore.

If you give your AI model unstructured notes, the model supplies its own structure for … and that is where a lot of our thinking happens.

➡️ Check what I mean by this in my latest deep reading post.

Unsorted highlights don’t produce messy output. They produce confident output shaped by the model’s training (or what it happens to find on its own), not your thinking.

You’ve just given up you thinking and voice to AI, maybe without even knowing, because most people do not watch how AI models “think” (or when it prompts itself to go deeper into a question or task).

So the question isn’t whether your notes have a structure. It’s whose. Yours, or the machine’s.

Five jobs for organizing notes

When you read almost any research article, nearly everything in it is doing one of five jobs.

These types come from a sixty-year-old idea. In 1969, Robert Horn argued that content has a structure independent of its topic.

How you organize a piece of information depends not on what it’s about, but on the job it does for a reader.

➡️ This has become the centerpiece for how I design context for AI systems. Check out more here.

His method, Information Mapping, ran through technical writing and instructional design for decades. What was once a discipline for writers turns out to be exactly what makes knowledge legible to a model.

Sort your notes by the job, not the format. A bulleted list in an article might be three definitions, three recommendations, or the stages of a method.

Look at what the reader is meant to do with it.

Read first, sort second

The practice is four moves.

1. Read the full source once, marking anything you’ll want again. Don’t categorize yet. Focus on understanding the source.

2. Go back through your marks and ask of each one: reference, definition, concept, principle, or process?

3. Rewrite it in your own words under that heading. Rewriting is where the understanding happens, and it keeps the notes sounding like you.

4. Add a short reaction where you have one, for example, agreement, doubt, a connection to your own work. Mark it clearly as yours. These are the seeds of original writing.

Then you provide the context and purpose on the top of the note (or whats often called meta-data in the content world).

The five types capture what a source says. They don’t capture what it is:

• who wrote it,

• what it argues overall,

• and why you’re reading it.

Keep that in a short block at the top: source, argument, why you’re reading it, a tag or two.

On paper it reminds future-you what you’re looking at. In a knowledge base, those same lines become the metadata an AI reads first.

What one paper looks like sorted

Say you’re reading one of the studies behind the AI-in-education debate, for example Bastani and colleagues’ 2025 experiment, which found that an AI tutor without guardrails raised students’ practice scores but left them worse off once the tool was taken away.

The resulting note might look like this:

➡️ You can see a real example that I created in Obsidian here.

When you later want what the research recommends, you go straight to your principles.

When you need the numbers, they’re in your reference.

When you need how the experiment actually ran, it’s in your processes.

Read three more papers this way and their principles line up beside these, so the agreements and contradictions become visible, which is where original synthesis starts.

Notice the paper is about what happens when you let AI do the thinking for you. The work looks better in the moment, then the skill isn’t there when the tool is gone.

Sorting the paper into types is the opposite move. You do the thinking first, in your own categories, so there’s nothing for a tool to stand in for later.

The understanding happens in the sorting, not the highlighting … or even the writing in some cases.

Subscribe now

Why this is worth the extra minute

Structured reading is slower at first, but faster later. You retrieve quickly because you know what kind of thing you’re looking for, and you build quickly because a new source’s notes slot in next to the old notes in more relevant ways.

The structure ends up a record of your thinking about the sources rather than a just a random record of them.

Do the work and the model will reflect that work. Leave it undone and it has only its own random guesses.

When you finally bring AI in to draft a literature review, surface tensions across sources, or turn reading into teaching material, you need far fewer tricks than people claim.

If you have structured, well-thought through notes, you don’t need much of a prompt.

For paid subscribers: Below I provide a structured skill that has AI develop a synthesis based on information-typed research notes that you can try in any agentic tool or Claude Cowork.

Read more

He mentioned something another thought leader in structured content had said to him: you don’t need to validate it. It just works. Do it.

Scott and I are both trying to build the empirical case, so that comment landed with a little friction. It’s been rattling around in my head ever since.

He’s not wrong. And my guess, he is in a position where he can say that and be believed.

Most of us aren’t.

Those of us in academic positions, or making the case to skeptical stakeholders or CEOs, have to do the harder thing: explain why it works.

And that explanation starts with a word we use constantly without ever pinning down what it means.

I’m Lance Cummings. And welcome to my (what now seems to be) monthly podcast that explores deep research on AI and writing.

➡️ What exactly am I talking about? Here’s a post on how I using information types to validate AI content.

➡️ Or a deeper dive here.

The word we’ve been misusing

Here’s the problem.

We use “information” to mean everything.

Context is information. Instructions are information. Facts, procedures, principles, background knowledge — all information.

When everything is information, the word stops doing useful work.

Maybe that vagueness has been manageable in content design, where the main cost is poor user experience.

It’s becoming a real problem as we try to build AI systems that go beyond generating text to building systems that persuade, explain, and instruct in ways that actually serve readers (also called rhetoric).

If we are going to make progress in this area, the we need to make clear the difference between data and information.

It’s a distinction the philosopher Luciano Floridi has been developing for twenty-five years. And it turns out rhetoric has been working on the same problem for about 2,500 ... we just use different words.

The three conditions that makes information

In his Information: A Very Short Introduction (Oxford, 2010), Floridi draws a hard line between data and semantic information.

Data is just stuff — symbols, signals, marks on a page.

Information is what data becomes when it clears three conditions:

1. It must be well-formed,

2. meaningful,

3. and true.

All three. Miss one and you still have data, regardless of how well-organized it is.

Here’s what that looks like concretely. Imagine someone hands you these facts about making pot roast:

A pot roast takes 3 hours. Brown the meat first. You need carrots, potatoes, and beef. Cook at 325 degrees. Season with salt, pepper, and rosemary. The meat should reach 145 degrees internally.

Every fact is accurate. But try to cook from it.

Do you season before or after browning? Do the carrots go in at the start or the end? When do you check the temperature?

The facts don’t tell you, because nothing in the list says what kind of thing each fact is.

Now take the same six facts and organize them by what they’re for:

What you need: beef, carrots, potatoes, salt, pepper, rosemary.
What you do first: brown the meat, then season.
How it cooks: 325 degrees for 3 hours.
How you know it’s done: internal temperature hits 145.

Same facts.

But the arrangement has added something that no individual sentence contained: a claim about what each piece is for.

Ingredients are what you gather before you start. Steps are what you do in sequence. Conditions are how you verify.

That categorical claim — this is the kind of thing this is — is what converts data into information you can act on.

Notice something: you probably recognized the second version as a recipe before you finished reading it.

The structure is a genre, or a pattern that signals “this is organized for use” before you’ve processed a single fact.

The format itself is information.

Most AI failures aren’t random. They cluster around these three conditions.

A hallucination fails the third condition: well-formed and meaningful, but not true.

A misunderstood instruction often fails the first: parseable but not well-formed for the specific task.

A prompt full of relevant content can fail the second: accurate facts that the model can’t construct meaning from because the context lacks coherence.

We’ve been treating these as “model” problems.

Floridi’s framework says they’re information quality problems, which means they have structural solutions, not just scale solutions.

Adding more content to your prompt doesn’t fix any of them.

Organization is meaning-making

Let me bring in a framework that might seem like a detour but isn’t.

Roman rhetoricians organized the work of communication into five areas called canons:

• invention,

• arrangement,

• style,

• memory, and

• delivery.

I’ll be coming back to all five in a longer piece.

But here’s why I’m mentioning them now: the second canon which they called dispositio (or arrangement) was never understood as an organizational problem. It was understood as the act of making meaning.

We tend to teach arrangement as the organizational part of writing. Where does the thesis go? How do you sequence your argument?

You’re finding the best container for ideas you already have.

That’s not what Cicero meant, and it’s not what the tradition understood.

Arrangement in the classical sense is constitutive. It’s not how you deliver meaning. It’s how meaning comes into existence.

The same material arranged differently isn’t just harder or easier to follow. It does different cognitive work.

Think about the difference between a bibliography and an argument.

Same sources.

The bibliography is data — accurate, organized, findable. The argument is information, because the arrangement has done the work of converting accurate citations into something that means something to a specific reader in a specific situation.

Every student who’s turned in a paper with all the right sources but somehow no argument has experienced this from the receiving end.

This is exactly what technical writers do when they convert a subject-matter expert’s data dump into structured documentation.

They’re not organizing existing information. They’re creating information out of data.

And when that process fails ... when a document has all the right content but still doesn’t work, the failure is in the arrangement, because it hasn’t cleared Floridi’s second condition. It hasn’t made meaning.

So when someone with enough authority says “it just works” about structured content approaches, what they’re observing is that information typed by function creates meaning more reliably than data organized by topic.

That’s a dispositio argument.

What this changes in practice

This should reframe how you diagnose document failure.

When something doesn’t work, the standard question is “can users find it?” Floridi adds two more: Is it well-formed for this specific task? Does the arrangement create meaning, or just accuracy?

Those are different problems with different solutions, and neither one gets fixed by adding more content.

For educators, the data/information distinction is a teaching tool.

Students produce data constantly. One might say, “accurate, researched, grammatically correct data” that hasn’t cleared the meaning condition.

The feedback “organize your ideas better” doesn’t tell them what’s wrong.

The feedback “your arrangement isn’t building meaning yet? What do you want the reader to understand before they reach your conclusion?” does.

That’s the difference between teaching organization and teaching dispositio.

Whether you’re building systems, designing course materials, or just trying to get a useful output from a prompt, structure is epistemic, not cosmetic. Organization creates knowledge. It doesn’t just make it look nice.

If the same facts arranged differently produce different information, then how you organize your inputs is a meaning-making act, not just a delivery mechanism.

This is exactly what my Writing with Machines course works through — how to apply these distinctions to prompt design, AI-assisted writing, and teaching with AI. Link in the show notes.

Where this is going

Dispositio is one thread in a longer argument I’m working toward.

The classical rhetorical canons are, I’m starting to think, a pre-modern framework for the exact problem Floridi is formalizing.

Each canon maps onto a different dimension of that same problem:

• how you generate material in the first place,

• how you arrange it for meaning,

• how style does cognitive work,

• how knowledge gets stored and retrieved.

Each one has direct implications for how we build AI systems that actually serve readers.

I’m developing that argument in a longer piece. Watch for it.

And if someone in your field is wrestling with how to make the theoretical case for structured content and information typing, share this episode .... because the pragmatists are right that it works, and the reason it works is 2,500 years old.

I’m Lance Cummings. Until next time — arrange your data like it means something.

Sources:

• Luciano Floridi, Information: A Very Short Introduction (Oxford UP, 2010)

• Classical sources on dispositio: Cicero, De Oratore; Quintilian, Institutio Oratoria

A few weeks ago I posted an upgrade to the prompt framework I’ve been teaching for two years.

• Tasks → Skills

• Context → Knowledge

• Content → Materials

The symmetry is nice, but it doesn’t quite work out in reality.

I’d been describing a skill as the evolution of the task, but a task was a one-shot instruction. A skill is a reusable document an agent invokes across situations.

True, as far as it goes. But then I opened one of my own skill files and started labeling what was actually in it … and the task turned out to be the smallest part of the document.

Though this is a narrow analogy, I think this is where “writing for machines” is going as AI workflows become both more complex and more accessible.

I’ll be presenting on this soon with (with a recording available).

➡️ Check it out here.

What’s actually inside a skill

Here is a quick review if you haven’t been tracking this.

In December, Anthropic released Agent Skills as an open standard, and other platforms picked it up, which means its pretty close to a universal format.

Simply put, skills are text files that AI models reference for tasks: YAML frontmatter carrying machine-readable metadata, Markdown instructions for the model, and whatever scripts and reference assets the work requires.

➡️ You can see one of my examples here.

Read that back as a content professional. A file with metadata and managed assets.

Sound familiar? It should. That’s documentation.

Here’s what I found when I labeled my own weekly-class-plan skill, the file I use to build the weekly plans I send students.

Five information types. One document. The task was in there, but it arrived bundled with the concepts, references, and principles the model needs to execute without me supervising every step.

That is what a skill actually is. Not an evolved prompt, but a complete piece of structured documentation, written for an audience that happens to be a machine.

Subscribe now

Why the typing matters more now than it did last year

You might see this as a cute exercise … except that the newer models keep locating failure exactly where structure is missing.

For a sense of scale, consider AgentIF, a 2025 benchmark that tested how well leading models followed long, real-world agentic instructions. These are the extended system prompts and tool specs that skills are made of.

The best models at the time perfectly followed fewer than a third of those instructions, and performance fell off a cliff once the instructions ran long. The research methodology has weak spots worth their own post, and the frontier has moved several generations since, so I’d treat the number as illustrative rather than proof.

But the basic problem still exists.

A good example is Claude’s new releases that come with a warning: The model now takes your instructions literally, where earlier models interpreted them loosely or quietly skipped parts. Re-tune your prompts before you upgrade.

This means your context, prompts, and skills need to be more precise, not less.

Better instruction following means the model does exactly what you wrote, including the parts you didn’t mean to write. The reliability problem is shifting from “the model can’t follow” to “the model follows precisely, and you didn’t say what you thought you said.”

Now picture an untyped skill file through that lens. Everything written as a directive. A conceptual explanation bleeding into the middle of a procedure. Gaps the model quietly fills from training data instead of from your intent.

I’ve started calling these failure modes type collapse, type contamination, and missing types. The better the model gets at following instructions, the more visible they become, because the model stops covering for them.

The fix isn’t a better prompt, but better context … the kind content professionals and many writers have practiced for decades.

• One type per block.

• Prerequisites before steps.

• Facts stated as facts.

• Guidance scoped and bounded.

The model responds to those boundaries because human writing has always carried them, and the model was shaped on human writing. That’s becoming even more important.

The part I’m saving for the webinar

There’s a second payoff to typed content that matters more than output quality. Typing tells you how to evaluate the output.

Reference and Task content is largely verifiable. The URL matches or it doesn’t. The steps complete or they don’t.

Concept and Principle content takes judgment. Is the explanation accurate, is the guidance scoped correctly.

The evaluation research has converged on exactly this split, pairing probablistic checks with rubric review, and typed content tells you which mode applies to which block.

The same structure that makes content reusable makes it testable. That’s one thing I’ll be talking about in my next Content Wrangler webinar.

The session is Structured Prompting 2.0: What the Research Actually Says, hosted by Scott Abel on June 16 at 1:30 PM ET. If you’ve missed it, there will be a recording.

Register or access video here.

If you or your team is building skills, agent instructions, or AI knowledge bases right now, this is the hour that connects what you already know how to do to the artifacts that suddenly need it.

Cyborgs Writing has accumulated something close to a hundred posts on AI and writing — courses, prompt experiments, podcast episodes, research syntheses, tool walkthroughs, and arguments about why none of this is really about the model. The line that runs through all of it: AI behavior is shaped by the rhetorical and structural choices a writer makes upstream.

This page is the index I wish had existed: a way to find a piece by what it’s about rather than scrolling the archive.

Start with whatever calls to you, or skip to the section that fits what you’re trying to do.

By the way, I used Claude Co-work to create this index after building my own markdown knowledge base by first exporting and organizing all my Substack posts. More on that coming soon!

Start here

• Introduction to Machine Rhetorics

• What is AI Content Operations?

• Why Every Writer Needs a Prompt Design Plan

• Why You Shouldn’t Be Writing A New Prompt Every Time

• Is Structured Prompting Dead?

• Testing as Rhetorical Proof

• Bridging Ancient Wisdom with AI Technology for Writing & Content Strategy — newsletter about-page

Writing With Machines (course)

The course curriculum, in roughly the order it’s taught.

• Writing With Machines — course about-page

• Unpacking Transformer Technology

• The Anatomy of a Prompt

• The Principles of Structured Prompt Operations

• The Role of Prompts in AI Content Operations

• Understanding Temperature & Style in Prompt Design

• Using Information Types to Build and Evaluate Prompt Structures

• Designing Simple Chatbots

• From Chatbot to Automations

• Worksheet: Knowledge Integration Workflow

• Worksheet: Prompt Taxonomy Development

• Worksheet: Understanding Your Content Patterns

Deep Reading (research synthesis)

Each piece reads two or three academic sources together.

• Rhetoric and Artificial Intelligence — Hunter 1991 (foundational)

• The Man Who Predicted ChatGPT in 1998 — Horn 1998, Information Mapping

• Do Prompts Really Need Markup? — semantic markup studies

• What Does “Evidence-Based” Actually Mean for AI? — Sackett, evaluation frameworks

• What the Ancient Art of Organized Thinking Says About AI Hallucinations — semantic entropy, topoi

• What is RAG ... No Really, What is It? — RAG and chunking

• When AI Research Validates What Content Pros Have Always Known — chunking research

• Why Many Writers Can’t Map Their Workflows (and why that matters for AI) — Haas, Lockridge & Van Ittersum on workflow mapping

• Reclaiming Agency in AI Collaboration — agency and student writing

Prompt Lab / Context Lab series

Numbered experiments building specific prompts. Each entry pairs a finished prompt with its design rationale.

• Prompt Lab #2: Restructured Rob Lennon Prompt

• Prompt Lab #3: Human-Centered Writing Feedback

• Prompt Lab #4: Structuring Conference Notes for Your Knowledge Base

• Prompt Lab #5: Crafting an Email Assistant

• Prompt Lab #6: Crafting AI-Powered Meeting Summaries That Actually Work

• Prompt Lab #7: ADHD Coach

• Prompt Lab #8: Style Prompt Blocks

• Prompt # 9: Synthesizing Microcontent

• Context Lab #12: Weekly Plan Skill

(Prompt Lab #1 and Prompt #10 sit with the teaching posts further down the archive.)

PromptOps — libraries, taxonomies, structures

The methodology cluster: how to organize prompts so they can be reused and tested.

• 5 Ways To Take A Structured Approach to Prompt Operations

• Building a FAIR Prompt Library

• Creating Your Prompt Taxonomy

• How To Use Microsoft Loop as a Prompt Library

• Setting Up a Simple Prompt Library in Twos

• Adding Knowledge to Prompts

• Don’t Just Prompt Engineer. Start Building Taxonomies.

• Beyond Prompts (Director’s Cut)

• Why AI Whispering is a Myth

• A Step-By-Step Guide to Using AI for Ethos-Driven Storytelling

Chatbots and agents

Built artifacts: walkthroughs for creating chatbots, custom GPTs, and agent skills.

• 3 Simple Ways to Build GPTs

• Step-by-Step Guide to Taking a Structured Approach to Building Chatbots

• How to Use Poe to Create and Test Structured Chatbots

• How to Build and Test Your Own Chatbot for Free on Zapier ... Without a ChatGPT Plus Account

• Creating Custom Chatbots — resource index

• The Techne Behind Agent Skills

• Connecting Claude to Your Knowledge Base

• Bridging AI Operations and Human Expertise with Tailored Frameworks

AI tools — hands-on walkthroughs

Tool-by-tool experiments with Lex, Copilot, NotebookLM, ChatGPT, Triplo.

• Experimenting with Style and Temperature in Lex

• Using Lex AI to Enhance Human Writing

• How I Used Lex AI to do an “Audience check” on my Newsletter

• How to Use Copilot’s Notebook to Experiment with Prompt Design

• What I Learned Comparing ChatGPT & Microsoft Copilot

• Is NotebookLM Really a Game-Changer?

• The Cheapest Way to Experiment with Ai in Your Writing

• Automating Structured Notes with Triplo AI

• How to Unlock the Power of Old AI Conversations (so that you can create better content ... not just more)

AI content operations & structured content

Theory and practice of structured content as the substrate for prompt reuse.

• 3 Easy Ways to Create AI Writing Systems Using Structured Content

• Streamlining Content Generation & Personal Growth

• Unlocking the Future of Writing with Structured Knowledge

• How to Use Rhetoric to Repurpose Content with ChatGPT

• How Collaborating with AI Can Generate More Ideas… Not Just Content

• Are You Speaking Your AI’s Language?

• What is AI-Ready Content?

• Two Kinds of Knowledge You Need to Build a Helpful AI

Poetry and creative work with AI

How to use AI in the writing of poems — not the poems themselves.

• 3 Intelligences You Already Use to Write Poetry (and how Ai can enhance them all)

• How I Use Artificial Intelligence as a Conversation Partner to Write Poetry

• How To Use Poetry to Determine Ai Sentience

• Why Tomorrow’s Poets Will Use Artificial Intelligence

• Stop Using AI Like A Vending Machine

Personal essays — how I use AI

First-person accounts of using AI for specific work.

• How I Use Artificial Intelligence for Digital Writing (And Why It’s Not Cheating)

• How I Used ChatGPT to Stay Connected Authentically with my Family While in Poland

• How I Used Ai to Write My First Newsletter

• How I Created a 40-Minute Keynote Using ChatGPT

• How I am Using Collaborative AI to Help With My Study Abroad

• Ep. 1 How Religion Shapes My Approach to GenAI

• AI and the Future of Writing — A Recap

Experiments and stray observations

Pieces that don’t fit the clusters above — cultural commentary, one-off experiments, and a few open questions.

• Using Trump Style to Test AI Writing

• Vibe Coding Isn’t About Vibes

• How To Break ChatGPT With Dialogue

• 3 Things About AI That No One Is Talking About (that I want to explore)

A few weeks ago I wrote about what I saw across 17 student chatbots.

Read more

I’ll be honest with you … I came to this book review feeling a little behind.

Agentic AI has been moving fast, and for most of it I’ve watched from a comfortable distance.

I’m skeptical of a lot of the use cases, and I’ve made no secret of that. Can I really trust an agent to the complex work I’d want them to do? I’ve not been that sure.

But I think we’ve come to a point that everyone needs to reckon with agents one way or another, and Manny Silva’s Docs as Tests and AI made me think about that. Not because he evangelizes agents, but because he explains how to trust them.

That’s really what this book is about.

Not necessarily how to build agents, but how to verify they actually do what you need them to do, and why your documentation is the place where that trust either gets built or falls apart.

Docs as tests is a philosophy, not just a method

What strikes me most about this book is that Silva isn’t offering a workflow checklist. He’s arguing for a way of thinking about documentation.

Every document that describes how a product behaves is making a testable claim. When it says “click this button and this happens,” that’s a promise. The question Docs as Tests asks and answers is whether your documentation keeps its promises.

Rhetoricians will recognize this as a question of ethos. Not the reduced version where ethos means credentials, but the deeper sense: credibility earned through consistent, verifiable action.

Silva never uses the word, but the entire book is an argument for how to build machine ethos, or the kind of trustworthiness that holds up not just when a human reads your docs, but when an AI system consumes them and generates answers for thousands of users.

One inaccurate paragraph, confidently retrieved and served by a chatbot, is a very different problem than one frustrated user who bounces to support. The stakes have changed, and Docs as Tests takes those stakes seriously.

Ethos is too often reduced to “credibility” and left at that. For Aristotle, one of the first to articulate this idea, ethos can’t simply be asserted. It has to be demonstrated through the work itself.

A document that claims accuracy isn’t credible; a document that demonstrably maps to reality is.

Technical communication scholars have made a version of this argument for decades. Documentation functions as a form of institutional ethos, a sustained social contract between a product and its users.

Silva’s framework makes that contract testable. He’s not adding rhetoric to documentation theory; he’s building the verification infrastructure that rhetoric always assumed someone was running.

Subscribe now

Three layers in one book

What makes this book practical rather than theoretical is how Silva structures it. Each section works at three levels simultaneously:

1. First, he lays out the conceptual foundation,

2. illustrates it through a practitioner named Vanessa who is working through the same problems in a real documentation context, and then

3. closes with exercises you can run yourself.

I’ll be transparent about my own limitations here. Some of the exercises require comfort with the terminal, YAML, and basic scripting. I got through them, but I needed to lean on AI to troubleshoot a few mistakes I didn’t fully understand.

If you’re a technical writer who works comfortably with dev tooling, this will feel intuitive. If you’re coming from a less technical background, the exercises are still worth doing — just plan to go slower, and don’t skip the Vanessa scenarios, which give you the shape of the work even when the code feels unfamiliar.

The inventory of documents that Silva lays out in Part Three was probably the part I found most useful. To run a trustworthy agentic workflow, you need:

• a project description,

• agent definitions,

• orchestration patterns that make the workflow legible to every agent involved,

• task skills written as reusable prompts, and

• plans with explicit acceptance criteria.

That’s documentation. And reading through it, I found myself thinking about portfolios.

Right now I ask students to document a chatbot assessment, build a structured knowledge piece, and analyze a workflow. That’s a solid foundation.

➡️ For a limited time, paid subscribers can access a version of this course for professionals for free. Check it out here.

But a more advanced version of that portfolio could be the full documentation set for a working agent: the project file, the skill definitions, the plan and specs.

It would demonstrate not just that a student can use AI tools, but that they understand the system well enough to govern one.

Manny’s book is the clearest map I’ve seen of what those documents actually need to contain.

Subscribe now

Understanding deterministic vs probabilistic

The core technical distinction in the book is between deterministic testing and probabilistic testing.

Deterministic tests produce the same result every time. It’s binary, reliable, automatable.

Probabilistic tests use AI to interpret content, which means the result can vary between runs. Same input, different output.

That might sound like a limitation, and in some ways it is. But what Silva’s really mapping is the difference between things that can be verified objectively and things that require interpretation — and interpretation, as any writing teacher knows, is where human judgment lives.

When we mistake a probabilistic check for a deterministic one, we get what Silva calls a false signal.

His practical solution is a hierarchy of trust. At the base are ungrounded assertions, or documentation that simply hasn’t been tested.

Above that is grounded probabilistic testing, where AI evaluation is constrained by explicit criteria, run multiple times, and treated as reconnaissance rather than verdict.

At the top is deterministic testing, or binary checks against a live product.

The goal is to migrate upward wherever possible, while being honest about what each level can and can’t tell you.

An LLM isn’t evaluating your product. It’s generating what it expects your product to be, based on the patterns of its training. That’s why any automated documentation system needs testing … not as a quality-control afterthought, but as the mechanism that keeps the content grounded in reality.

Why technical writers are still necessary

The third part of the book, “Teaching Agents Your Process”, was where I learned the most. Silva walks through what it actually takes to run an agentic documentation workflow:

• project descriptions,

• agent definitions,

• orchestration patterns,

• task skills, plans

• and specifications.

Every one of these is a document. Every one of them shapes how an agent operates and whether it stays within the bounds you intend. Writing them well and maintaining them accurately is a technical writing problem.

This is an argument I’ve made from the rhetoric side, and it’s gratifying to see it come from the practitioner side too. The work that gets automated is first-draft production.

The work that doesn’t is design, information architecture, judgment about what content actually needs to exist and in what form.

Technical writers who understand that distinction become more valuable in agentic systems, not less. The documentation that governs those systems requires exactly the expertise you already have.

The classical term for this kind of judgment is phronesis, or Aristotle’s practical wisdom, the capacity to discern what the right action is in a specific situation, with specific constraints, for a specific audience.

Phronesis isn’t a skill you can look up or a rule you can apply consistently across cases. It’s cultivated through experience, through having stakes in an outcome, through the kind of situational reading that comes from being genuinely accountable to the people you’re writing for.

I’ve been thinking about phronesis lately as the human quality that agentic AI most needs and structurally cannot have.

An agent can execute a well-documented workflow with high fidelity. It cannot tell you when the workflow is wrong for this situation. Or when the document that passes every test still misleads the user who reads it at 11pm trying to fix a production problem.

That gap is where technical writers operate, and it’s the gap that Silva’s human oversight checkpoints are designed to protect.

Who should read this

➡️ Check out Docs as Tests & AI here.

If you work in technical documentation, content strategy, or documentation engineering, this book belongs in your hands now. It gives you a coherent framework for thinking about documentation quality in AI pipelines, with concrete methods for testing and validating what you build.

If you’re an educator or solo creator who isn’t running enterprise documentation workflows (like me) it’s still worth your time. Docs as Tests & AI is one of the clearest explanations of what agentic AI actually involves, what the components are, and where human judgment remains irreplaceable. That’s useful regardless of whether you’re ready to build the system yourself.

I’m not ready to build all of it myself. But I understand it now in a way I didn’t before, and I’m starting to see where pieces of it apply to my own work. That’s about as good an endorsement as I can give.

Over the last year, content professionals have been telling me a version of the same thing.

The way content is structured affects how AI systems behave with it. Cleaner, more deliberate content seems to produce cleaner, more deliberate answers.

We all see this in practice. We all feel it. But pinning it down is harder than it should be. Proving it to a CEO, a CTO, or a colleague who hasn’t lived in the work is difficult to acheive.

This past semester I had an informal opportunity to look at the question. My Writing with AI students partnered with Dr. Yang Song's machine learning students through UNCW's SAIL initiative (Seahawks Advancing Interdisciplinary Learning), a program designed to support cross-disciplinary student collaboration.

Our goal was to design AI chatbots in partnership with New Hanover Disaster Coalition, focused on emergency communication for community members. The English students built the knowledge bases and system prompts, while the machine learning students focused on model integration and testing infrastructure.

By the end of the semester, we had 17 working chatbots across 17 distinct disaster-preparedness topics … and an external assessment of each one.

Teams that built narrow, curated knowledge bases organized by information type consistently outperformed teams that scraped pages or pasted in lists of URLs.

And while this wasn’t a controlled experiment, the pattern was visible enough that I want to share it.

The question content professionals keep asking

A lot of the content world is in the middle of a quiet identity crisis right now.

Many technical writers, content designers, knowledge engineers, documentation leads have all spent a decade building competence in topic-based authoring, structured documentation, modular content, and taxonomies.

Then generative AI arrived, and a lot of our colleagues started asking: does any of that still matter? Can’t we just point a model at a SharePoint folder and call it a day?

People who actually do this work know the answer. Of course it matters.

Of course the model doesn’t magically organize unstructured content.

Of course the same RAG pipeline will produce wildly different outputs depending on what you feed it.

But the question isn’t whether we know this. The question is how to demonstrate it to people who don’t.

The challenge is that most of the evidence is anecdotal. A senior content strategist describes their internal experience improving a chatbot at their company, and a skeptic shrugs because it’s one company, one product, one set of conditions.

Most of the controlled studies in this space are narrowly technical (which embedding model performs best, which chunk size optimizes retrieval). It’s not the kind of evidence that lands with a content team trying to convince a leadership group to invest in content infrastructure.

What I had this semester was something different. A setting where 17 teams worked on the same kind of problem, using comparable tools, with comparable assessment afterward. I could see what each team had actually done with their content.

The project

The goal of this project wasn’t really to build AI chatbots … but to use that project to give students an interdisciplinary experience that would help them understand AI from both perspectives … computer engineering and professional writing.

The foundation for this spring’s work was laid the previous fall by two other faculty and their students. Dr. Ian Weaver, in the Professional Writing Program, taught an advanced professional writing course that consulted directly with the New Hanover Disaster Coalition to map the community network a system like this would actually serve.

His class built on the research of Dr. Jill Waity, Professor of Sociology and chair of the Sociology and Criminology Department at UNCW, whose community-engaged work on disaster preparedness and resilience in rural and regional communities established the empirical foundation for understanding user needs.

Ian’s students ran usability tests on existing disaster-preparedness materials from agencies like NC State Extension, Ready.gov, and the American Red Cross, and built a taxonomy of disaster-preparedness topics relevant to local needs.

Working in parallel in the Fall, Dr. Gulustan Dogan’s machine learning class produced what we might call a first draft of a knowledge graph for the domain for initial tests on the existing content.

The two fall classes then handed off the project to our classes this spring. The work moved through four classes across two semesters, with each phase building on the previous. Most of what I’m reporting here is the visible piece of a longer chain of student and faculty work that started before my students arrived.

This semester, each team was assigned a node on an adaption of that taxonomy. Seventeen teams, 17 topics, all roughly comparable in scope.

Each team had to do three things:

1. Build a knowledge base for their topic,

2. Write a system prompt that defined the chatbot’s behavior and voice, and

3. Design test questions to evaluate their bot.

Most teams used some combination of LM Studio, AnythingLLM, or Google AI studio. Then I asked teams to structure their knowledge bases using a microcontent approach.

Specifically, I introduced them to Precision Content’s authoring methodology, which sits at the layer underneath things like DITA and XML that many enterprise content systems rely on. The core idea is that you look at a topic and organize it around information types: a concept is one thing, a process is another, a principle is another, a procedure is another. Each has a recognizable shape and serves a different reader need.

I use information types with students because it’s accessible. You can engage with structured content as a way of thinking before you have to engage with it as a way of encoding.

Most undergraduates can’t be productively dropped into a DITA toolchain in a semester. But they can absolutely learn to look at a body of content and ask: which parts of this are concepts, which are processes, which are principles?

Not every team did this. I asked them to. I gave them the tools. But the assignment structure didn’t enforce it, and several teams defaulted to what was easier: scraping URLs, dumping in PDFs, pasting links to UNCW or county emergency management pages.

By the end of the project, the 17 knowledge bases ranged from carefully typed microcontent on one end to lightly organized link lists on the other.

One technical note about the conditions. Yang's lab machines have 8GB of VRAM, which meant most teams were running small, quantized Llama 3 models rather than the frontier systems people associate with conversational AI.

(For non-technical folk, that just means students were using inferior models.)

We're probably two years behind well-funded labs on hardware, and students had to develop extra craftiness to make their bots work within those limits.

The constraint was also informative, though. Smaller models with shorter context windows make the prompt and knowledge variables more visible than they would be at scale. The patterns I'm describing happen with frontier models too, just less obviously.

And in the broader practitioner conversation, the case for lighter models paired with well-structured knowledge is getting stronger. It's more cost-efficient, more accessible to organizations without GPU budgets, and often gets you closer to a usable result faster than scaling the model alone would.

How we assessed the chatbots

We used a Perplexity to evaluate each team’s chatbot, because it wasn’t the model the students used to build their own bots. It came in fresh, with no exposure to the student work, and it had access to its own retrieval over the open web. That gave us a meaningful external check rather than asking a model to grade itself.

For each team, we fed Perplexity their test questions, their bot’s responses, and their own human-written reference answers. Perplexity scored two things on a 1–5 scale: correctness and comprehensiveness. It also produced qualitative feedback for each response, explaining where the bot landed and where it fell short.

We also ran an embedding similarity comparison between each chatbot’s response and the team’s reference answer, hoping to get a third measurement that captured semantic alignment independent of Perplexity’s judgment.

Dr. Song and I then met to talk through the Perplexity results team by team, comparing the assessment against what we’d seen of each team’s content and prompt design.

Before I go further, this wasn’t really a controlled study. The variables weren’t held constant. Different teams used different models, different RAG setups, different testing question styles. Team capacity varied. Topic difficulty varied. Perplexity is a real evaluator but not a calibrated assessment instrument. It has its own retrieval biases and its own way of weighing comprehensiveness, and at times we saw it run overly critical on supporting details when the core content was actually solid.

But the pattern was striking enough that I think it’s worth replicating with a more rigorous design.

What we saw

The clearest contrast came from a coincidence in topic assignment. Three teams ended up working on closely related topics around food safety during power outages.

All three had access to the same source material: CDC, FDA, USDA, and Red Cross guidance on the 40°F/2-hour rule, perishable food handling, refrigerator and freezer timelines.

One of those teams scored a perfect 5/5 on correctness across all seven of its test questions. Perplexity’s assessment described the bot as “fully aligning with FEMA, CDC, and utility best practices” and “delivering critical, life-saving guidance.” That team had built a narrow, deliberately curated knowledge base. Its test questions matched what the knowledge base was designed to answer. Its system prompt kept the bot pulling cleanly from the structured content.

A second team working on essentially the same topic averaged 3.6 on correctness and 2.8 on comprehensiveness. The assessment called several responses “partially correct but falls short in both factual precision and practical completeness, especially for a life-critical topic.” Same source material was available. Same kinds of tools. Same model class. The difference was in how the content was organized before it got to the model.

A third team in this cluster scored well overall but had one telling error. The bot confidently gave incorrect refreezing guidance for thawed meat. Their knowledge base was structured but had a gap, and the bot filled the gap with confident-sounding false information.

The other vivid case involved phone numbers. Two teams built bots focused on accessing help: one for mental health resources, one for general post-disaster assistance. Both teams built their knowledge bases primarily from links to UNCW and local resource pages, expecting the RAG pipeline to surface the right information when asked. Both bots hallucinated.

When asked about UNCW counseling services, the mental health generated a phone number where the last four digits were wrong. The accessing-help bot gave UNCW’s main switchboard number as the campus police department.

Neither of these is a small failure. These are help-seeking bots. The user dialing the number the bot provides is, by definition, someone in distress. Hallucinated contact information is the most consequential failure a chatbot in this domain can have, because it’s the failure that translates most directly into a real-world safety problem.

And it’s the kind of failure that scraping URLs produces. The bot has seen “UNCW police” and a phone number near each other in the source material, and it generates a plausible-looking answer that happens to be wrong.

Teams that built around information types didn’t make these errors as often. Their bots failed in different ways, but they tended to be merely incomplete rather than confidently wrong.

The team that scored highest in the entire cohort built around clear, source-grounded content about pet-friendly shelter access. Five out of six responses scored 5/5. Perplexity described them as “drawing directly from NC Emergency Management and United Way protocols.” That team’s knowledge base was small, narrow, and rigorously sourced. Their bot reflected that.

What this argues

I’ll restate the caveat. This wasn’t science. The variables weren’t held constant. Seventeen undergraduate teams working under real time pressure with novice technical skills is not a controlled environment. I am not making a claim that the structured-content advantage I’m describing here is statistically established.

But in a setting with 17 comparable teams working on comparable problems, the teams that organized their content by information type before encoding it into a knowledge base consistently produced more accurate, less hallucinated, less confidently-wrong chatbots than the teams that scraped URLs or dropped in link lists.

This is what the content professionals I talk to have been telling me about for a while. The model isn’t doing the structuring work for you. The model can only reflect what it’s been given.

Content that is already organized by meaning gives the retrieval layer something coherent to retrieve. Content that hasn’t done that work yet gives the retrieval layer raw material it has no choice but to guess.

This is also, I think, the right argument for taking microcontent seriously as a methodology, separate from any specific encoding standard.

You don’t need DITA to start. You don’t need an XML toolchain. What you need is to start asking, for each piece of content you’re feeding a system, what kind of information this is and what shape it should take.

Precision Content’s typed-information approach is the one I’m using with students because it’s accessible to people who aren’t ready for the deeper technical apparatus, and because the gains it produces are real, even at the entry level.

What I’d want to test next

Seventeen student projects in a single semester is not where this story should end. A few things I’d want to look at, ideally with someone who can design the experiment properly:

A genuine controlled comparison. Same topic, same model, same testing framework with three knowledge bases built three ways (scraped, lightly organized, typed-microcontent). Measure performance directly.

Specific information types. Which contribute most to AI performance? Is the gain mostly in concepts because they anchor retrieval? In processes because they answer the “how do I” questions users actually ask? In principles because they shape the bot’s voice and reasoning? I have hypotheses; I don’t have data.

Scaling effects. Does this hold at larger knowledge base sizes? My intuition is that the structured advantage compounds, but I’d want to see it.

Cross-domain. Does the pattern hold beyond disaster preparedness? I suspect it does, but disaster preparedness has unusually well-established authoritative sources. A domain with messier sources might tell a different story.

If you’re working in this space and have done something more rigorous, I’d be glad to compare notes. The professional intuition is solid. We need the demonstrations.

➡️ And if you want to learn how to build chatbots like this, check out my course on Writing with AI. This is the same process my students used and is currently available free for paid subscribers for a limited time.

Subscribe now