I spent the summer building a markdown knowledge base that AI helps me manage, then went looking for research that would tell me it was worth it. Four studies across twenty-seven years, including the first real evaluation of AGENTS.md context files, suggest the organizing work is easier to demand than to justify.

I’ve got to be honest with you. I’ve spent most of this summer building workflows around a personal markdown knowledge base that AI helps me manage.

Tags, links, folder structure, naming conventions. All the housekeeping I’ve quietly avoided for twenty years. Everything I hate about managing my own notes is exactly what keeps sending me off to try one more app.

Surely there’s a perfect one out there. There isn’t. I know there isn’t.

I go looking anyway, about twice a year. For a brain that would rather build the system than sit down and use it, this is a dangerous hobby.

I’m Lance Cummings. And welcome to my now monthly podcast that explores deep research on AI and writing.

After all this, I decided it might be worth looking into what kind of research has been done on personal knowledge bases.

Oh, yeah ... there’s lots of creators out there giving out all sorts of ideas and frameworks, but what does the research say?

Here are some of my initial findings.

I looked at four papers, written across twenty-seven years, are all circling the same question from different sides: what does it actually cost to make your own knowledge explicit, and who ends up collecting?

That question used to have an easy answer.

You did the filing for future-you, future-you rarely showed up, and the whole thing quietly rotted. What’s changed is that my knowledge base has a second reader now. An AI assistant searches it, and it searches differently than I browse.

So it sounds like a reason to finally do the work of organizing. And it might be.

But you might want to hold on a second.

The structure nobody wants to do

In 1999, Frank Shipman and Cathy Marshall point out in their article [Formality Considered Harmful] that the real problem is motivation and intention.

After years of building systems people wouldn’t use, the real problem came down to something simpler than interface design. Users were unwilling or unable to make structure explicit.

If asked to declare what a thing was or how it related to other things (👋 information types), people balked, worked around it, or filled the field with garbage.

Not from laziness. The cost of formalizing is immediate and the payoff is speculative and later. You know what you mean right now. You are, at that moment, the worst possible judge of what future-you will need.

This is probably one reason it’s so hard for content strategists to convince stakeholders to structure their content.

Their solution really ties to how I’m working with AI to build a personal knowledge base. Rather than demanding structure up front, systems should offer incremental, system-assisted formalization. The system flags structure you never declared and helps you make it explicit over time.

In other words, build your knowledge from the ground up with help. Start small and use AI to help you discover the organization that’s already there.

That’s more or less what I’ve been building.

It’s also how I nearly lost two folders in my own vault this month, because an assistant had been quietly filing things into subfolders I never asked for.

I got system-assisted formalization, but I wasn’t paying close enough attention to what the AI agent was doing.

What folders and tags were both missing

Jump to 2008. Amy Civan, William Jones, Predrag Klasnja, and Harry Bruce ran a study to find out what kind of organization works best, focusing mostly on tags and folders.

There’s no winner.

Each model has strengths and weaknesses across keeping, organizing, and re-finding information. What study participants needed was a clearer framework for the organization scheme itself.

Think here about things like taxonomies, ontologies ... or simply deciding what goes where, why, and for whom ... and how that “what” gets found.

That’s more important than what tool to use.

What they wanted was a way to say what the categories meant, and to change their minds later without everything breaking. The tool or method doesn’t give you that.

I found this out the hard way.

I counted the tags in my vault a few weeks ago: 1,988 of them. At least forty-five were the same idea written in different ways — “rag” and “RAG,” “pkm” and “PKM.”

There’s a danger that a search or an AI scan might only turn up half the entries for tags like these. The tagging system worked fine. I just had nowhere to record that those two spellings meant one thing or master guide that gives clear guidelines (without guessing in the moment).

That gap is what I ended up filling with three plain markdown control documents: a taxonomy document, tag thesaurus, and workflow overview.

More on that to come in future posts.

How AI is changing the way we think about organization

You’ve probably seen the trend already. To better control agentic workflows, people are starting to do this formalization work voluntarily — not in note-taking apps, but in code repositories, as context files for AI agents, files like AGENTS.md, CLAUDE.md, the README written for a machine.

But there are a few problems arising because developers are mostly controlling what should be called “documentation,” which needs a content specialist.

A 2025 preprint, Agent READMEs by Chatlatanagulchai and colleagues, studied 2,303 of these files across 1,925 repositories. What they found is that these files get maintained the way a build script gets maintained: small commits, constantly, forever.

They grow harder to read as they go. Nobody sits down and revises one.

The content skews hard toward the function ... not context:

• implementation details — 69.9%

• architecture — 67.7%

• build and run commands — 62.3%

• security — 14.5%

• performance — 14.5%

So we tell the machine how to operate the thing, and almost never what the thing is for.

In the content world, we call this the “structure tax.” Maybe you don’t pay a price right now for structuring your knowledge ... but you will later.

That’s why this work is an investment, whether you are a company, a university, or just a solo creator.

The structure tax people refused to pay for themselves, they’re now paying for an agent. [pause]

The part that stopped me

Here’s where I have to complicate my own summer.

The obvious conclusion from all of this is that context files work, so write them. That’s certainly what I assumed while building mine.

Then in February, Thibaud Gloaguen, Niels Mündler, and colleagues at ETH Zurich published Evaluating AGENTS.md, the first rigorous test I know of. They ran coding agents three ways: with no context file, with one the agent generated for itself, and with one a human developer had actually written.

The machine-generated files didn’t help.

They made things slightly worse, and raised inference cost by more than twenty percent. The human-written ones were the only version that came out ahead, and they beat the machine’s version by a clear margin.

The temptation is to have AI generate this documentation for you ... its lots of work. Turns out that could be doing more harm than no context file at all.

But consider this carefully, the agents did follow the files.

When a file named a specific tool, the agent used it almost every time; when it didn’t, almost never. The instructions worked.

What didn’t work was the one thing every official recommendation tells you to put first: the repository overview. It’s the biggest chunk of these files, and it did nothing. Agents found what they needed just as fast without it.

In the end, this is a failure of content strategy. An overview is something the machine can already work out by looking around, but principles and processes needed to properly do that task ... not so much.

The files contained clear information types beyond just an overview. But, honestly, nobody in that study tested a context file designed that way. They varied the model, the prompt, the length, whole categories of content, and none of it moved the needle, because they were all the wrong knobs.

The one variable that mattered was a human deciding what actually belonged in the context file.

Two honest limits to these studies.

Both agent papers are preprints, not yet peer-reviewed. And they measure code tasks in repositories, a long way from retrieval in a personal knowledge base. I don’t think the finding transfers cleanly.

But nobody has run the knowledge-base version of that study, which is roughly where the next several weeks of this newsletter are headed. Stay tuned.

Takeaways for writers and content professionals

To be clear, this does not mean we should “stop organizing your notes.” Rather, we need to pay more attention to how we do it, especially when AI is part of the picture.

So the four things below are the ones I’d still do this week.

Write control documents, not more folders. The 2008 finding is the durable one. Before you reorganize anything, write down what your categories mean and which synonyms resolve to which preferred term. My tag thesaurus took forty minutes and fixed years of broken searches. No new app required.

Audit what your assistant has been organizing for you. If AI helps maintain your notes, run a structure check on purpose, full depth, no shortcuts. Incremental formalization is genuinely useful and it’s also invisible by default. I nearly lost two folders to that.

Treat your context file as configuration, and version it. If it evolves like code, maintain it like code. Small commits, a changelog, and periodic pruning. Files that grow without eyeballs get harder to read and control.

A/B Test your workflows. Run a task with your context file and without it. This is the least fun recommendation but can help you see whether a specific skill or documentation is actually needed..

If you want the practical side of this, the skills that run my knowledge-base-to-Blue drafting pipeline are posted in my public skills library. There are 18 of them up right now, and the number keeps moving.

... and I am updating them everyday, like software.

➡️ The method underneath all of them is what my Writing with Machines course walks through.

Pay it now … or pay it latter

The structure tax hasn’t gone away ... you’ve got to pay it somewhere.

We refused to pay it for ourselves because the payoff was invisible. Now we’re paying it for our machines, and the research says most of what we’re writing down is the wrong half.

Not because structure doesn’t help, but because nobody’s decided what belongs in it and what’s just noise wearing a filename.

That’s the part I can actually do something about. So instead of shopping for a new app this school year, I’m going to build my own, and pay attention to what goes in it.

If you know anyone building content workflows that might be interested in knowing the research behind the work, send this their way.

Leave a comment and let me know how you are using AI for your knowledge base.

I’m Lance Cummings. Until next time — mind what your system is quietly deciding for you.

The Society for Technical Communication, the Special Libraries Association, and other professional groups are shutting down or cutting deep, and AI is the easy explanation. The membership and revenue data tell a different story — one that started twenty years before ChatGPT and explains why some associations are dying while others are growing.

Is AI killing your professional association? It’s an easy question to ask right now.

The Society for Technical Communication (STC) lasted 72 years and had 25,000 members at its peak. On January 29, 2025, it announced its shutdown. Two months later, the Special Libraries Association followed. The Society of Professional Journalists cancelled its annual convention and named a $391,000 deficit. The Public Relations Society of America’s 2026 board chair told members plainly: “We are at a crossroads.”

If you work in tech, you probably heard about STC. If you work in journalism, publishing, library science, or communications, you may have watched something similar happen closer to home. If you’re in a field that hasn’t felt it yet … it may be coming.

The easy explanation is AI. And there’s something to it.

When staying current in your field means creating a prompt instead of attending a conference or reading a gated newsletter, the case for annual dues gets harder to make. But STC’s membership had been falling since 2002. That’s twenty years before ChatGPT. IRS filings show negative net assets every year from 2013 onward.

Whatever killed these organizations, it didn’t arrive in 2023.

I’ve been making a similar argument about higher education.

Assessment was already broken; AI just exposed it. Grades, assignments, and credentials built to gatekeep access to knowledge stopped making sense the moment that knowledge became freely available. Professional associations hit the same problem, just on a different timeline.

I’m researching this now by surveying practitioners across fields in technical communication about what professional community actually means to them, and what they do when the institution behind it disappears.

More on what I’m finding in the posts ahead.

Subscribe now

The numbers tell a slow story

I’d actually been meaning to join STC for a few years.

Every time I got around to it, another no-dues community already had what I needed, so I never signed up. I wasn’t trying to make a point. I just never got around to it. It would seem a lot of technical communicators were making the same quiet choice.

The Society for Technical Communication peaked at roughly 25,000 members in the late 1990s and spent the next two decades bleeding them out. A former board member who tracked the data himself estimates membership would have fallen below 1,000 by the time the doors closed. The organization stopped publicly reporting membership figures after 2007, which tells you something about what those figures looked like.

The finances follow the same arc: revenue fell by more than half between 2011 and 2023, and the organization carried negative net assets every year for the last decade of its existence. The chart below has the full numbers.

What STC was selling made sense when those things were genuinely scarce. You paid dues because that’s how you got the journal, attended the conference, and earned the certification that signaled your competence to employers.

The internet didn’t destroy that value overnight, but it spent twenty years making it cheaper and cheaper to get elsewhere.

Free forums, YouTube tutorials, LinkedIn groups, and eventually entire communities built around specific tools and workflows pulled the same audience STC needed to survive.

AI maybe accelerated the last leg of that erosion, but it didn’t start it.

It’s not just STC

Individual-membership nonprofits that peaked in the late 1990s, built their revenue around dues and an annual conference, and offered credentials and a journal as their core value proposition are under pressure across professional fields.

Some are collapsing. Some are cutting. A few are doing something different.

Two months after STC closed, the Special Libraries Association announced it couldn’t pay its bills either. It had 14,000 members at its peak, a journal, and an annual conference, just like STC. Its members voted to dissolve rather than declare bankruptcy, merging into a larger information-science body by early 2026.

The Society of Professional Journalists cancelled its 2024 national convention, its treasurer telling reporters the organization faced a projected $391,000 deficit. PRSA, the leading professional association for public relations, opened 2026 with its board chair telling members that “meaningful change must occur for us to thrive for another 79 years.”

Not every association is dying. ACES: The Society for Editing has been growing. The Editorial Freelancers Association is growing. Meanwhile, the Association for Talent Development reported $53 million in revenue last year.

What did they do differently?

What this means for your field

So is AI killing your professional association?

Probably not directly.

But it may be the thing that finally made the problem impossible to ignore. That’s the same way it made certain approaches to teaching and assessment impossible to ignore.

What that means for your field, your professional identity, and where community goes from here is what I’m trying to understand.

I’m currently surveying practitioners across fields in technical communication about what professional community means to them now, and what they’ve lost or found since their associations started struggling.

If any of this lands for you, I’d genuinely like to hear from you.

The survey takes about 5-10 minutes.

➡️ Click here to take the survey.

Most skill or prompt libraries are dead the day they’re published. My Structured Skills Library is different: a live, self-updating collection of AI writing and teaching skills, organized by five information types, with copy-paste templates you can adapt to your own notes, tools, and voice.

Too many “prompt libraries” or “skill libraries” are graveyards … a pile of prompts or skills someone wrote once and never touched again.

This library is the opposite. The Structured Skills Library is the live set of skills I use in my own writing and teaching workflows, and it updates the moment I change one.

Each skill is a small program written in plain language, organized by five information types, providing structure for both human and machines to better access, perform and evaluate these skills.

You can read any of them like a short guide, or copy the whole thing and adapt it to your own work.

Available Workflows

As I mentioned, this library is live and constantly changing. As of July 2026, the following workflows and associated skills can be found:

• Content Creation — draft, revise, and publish

• Course Design & Teaching — build courses from scratch

• Vault Setup & Import — bring a scattered note backlog into a structured vault

• Miscellaneous — general-purpose tools for projects and context

There might be more, because I’ll probably forget to update this. (No live connection to Substack. 😆)

How it runs

The library mostly maintains itself. Every skill lives as a markdown file in a knowledge base on my own machine.

When I write a new one, an AI drafts a public version, tags it, links it to its neighbors, and files it in the library.

When I revise the working skill, the public copy updates to match. Every so often I run an audit that checks the whole thing against my own written standards, making sure links, structure, and voice don’t drift.

The result publishes straight from Obsidian, so what you’re reading is never stale.

If this intrigues you and you haven’t yet built your own local AI knowledge base, stay tuned. more to come on that front.

Subscribe now

What you’ll find

• A readable page for every skill: what it does, when to reach for it, and the thinking behind it.

• A copy-paste template inside each page, complete and working, with the personal parts marked so you know exactly what to change.

• Skills grouped by workflow: drafting and revision, course design, and general knowledge work.

• A short guide to the information types themselves, so you can build your own skills the same way.

How to use it

Start with whatever problem is in front of you, for example:

• auditing a draft for AI patterns

• synthesizing your notes

• planning a week of class

• handing a long chat off to a fresh session

There’s a skill for each, and each one tells you when it fits. Read the page to understand it, copy the template when you want to run it, and change the bracketed parts to point at your own files, tools, and voice.

You don’t need my setup to use these. You need your own standards written down, and the library shows you how I wrote down mine.

➡️ If you don’t know what those standards are or how to develop them, check out my course Writing with Machines.

What it costs me

The money side is smaller than people expect, and it doesn’t have to be Claude at all. I run it on Claude Cowork, the desktop tool I use to build and update the library, comes with the Pro plan at about $17–20 a month.

The only other tool is Obsidian, for the notes and the publishing. I do pay around $100 a year for syncing and the publishing tool.

But nothing is locked to one company. That’s what I love about it.

The whole system is plain Markdown files, so you can put any agentic model on top of it.

It already works with ChatGPT’s work mode, and it will run on free, open-source models on your own machine — slower, probably, but free. Conceivably you could do this whole thing for nothing.

(More on local, free setups in a future issue.)

What it actually takes

But the subscription isn’t the real cost. The real cost is thinking.

This library didn’t fall out of the machine finished. It came from a long back-and-forth. I had to redefine what a “skill” even is and making sure information types were correct … something that is oddly difficult even for machines.

The audit and the templates exist because I made the same mistakes often enough to name them.

This is what Manny Silva calls “docs as tests.” You explain your standard, the machine tries, you correct it, and slowly your standard becomes something a machine can actually run on.

The whole thing took me about a day to get up and running, and most of that day went to deciding what my standards were, not fighting the tools.

After that, the work compounds. Once the standard is written down, each new skill is faster than the last, and the machine handles the tedious parts, the metadata and links and consistency checks, while I keep the judgment.

More on all of this to come for everyone, but paid subscribers can check out the library using the form below. ⬇️

Read more

Skills are the backbone of any AI or agentic workflow ... think of it as documentation for your agents (not just a prompt). In this post, I share the actual structured skill I run before every Substack draft, organized by information type and connected to a Markdown knowledge base that makes it work.

Last week I wrote about Slaying the Churnivore with AI Workflows, a visualization of a content plan I’m developing with Claude’s help for my Substack. What I didn’t get into is the layer underneath and thought today ... why not share the skill!

For the past few weeks, I’ve been importing, structuring, and labeling my entire online back-catalog of writing into a Markdown knowledge base connected directly to Claude through Cowork.

This is a knowledge base AI can actually read, navigate, and write back to, because every file in it is labeled the same way I’m about to describe.

That project is too big for one post. Stay tuned. But the skill I want to show you today is a small, working example of what that knowledge base makes possible:

• It can read my guiding content plan directly, and update it when something changes.

• It can review my whole back-catalog to help me think about domains, topics, and where the yearlong arc is going.

• It can find other posts I’ve written and link them into new writing without me searching for them or even remembering they exist.

That last one just happened. While finalizing this post (with a structured skill), Claude linked back to A Skill Isn’t a Prompt. It’s Documentation. without me asking it to look for anything. I didn’t have to remember the exact title. I didn’t go find the URL. The knowledge base already had that information.

A skill is documentation, not a prompt

A skill is a short, saved set of instructions Claude reads before doing a specific job — the same way a process document orients a new hire instead of re-explaining the company from scratch every morning. While skills have some similar characteristics as a prompt, its quite a bit more than that.

The skill I run before every Cyborgs Writing session is called cyborgs-briefing. Its only job is to open my content plan, work out where I actually am, and hand me a short briefing before I start typing:

• current territory,

• how many posts are published,

• what’s next,

• what’s sitting half-drafted, and

• what’s still waiting in the ideas inbox.

Then it asks me one question to help me get started. I don’t need to load a project or cut and past context, or even work with a prompt.

For an ADHD mind that gets overwhelmed by over-planning, this helps me move forward today.

Subscribe now

Process and task aren’t the same thing

The skill below is organized by information type: concept, reference, principle, process, task.

These are the same labels I use across the whole knowledge base, on notes and skills alike. Most of those are self-explanatory. Two of them get confused constantly.

A process is an overview of the workflow, which provides context for the task. Without this, the tasks have less meaning.

A task is the specific instruction to run that procedure right now. Exact steps that the agent uses to accomplish the goal.

Collapsing the two makes it harder to revise and could also cause problems with the AI workflow.

➡️ Paid subscribers can see the structured prompt below ... but I’m also building a public, living version of every skill in this knowledge base, so the library updates as I revise instead of going stale.

For now, that library lives in a password-protected Obsidian vault. Paid subscribers can request access below.

Right now there’s 15 skills, but will be growing.

Read more

I’ve spent summers building Substack without a system, and it always breaks the same way: drafts accumulate, ideas scatter, deadlines slip when the semester hits. This year I built a visual map called The Churnivore, a project board for tracking status, and a Markdown archive to turn my scattered thinking into a legible, sustainable workflow where AI can actually participate.

This is the first summer in a long time when I am mostly free.

I’m not teaching.

I’m not buried in consulting.

I have research to work on, but I also have room to build things I’ve wanted for awhile:

• a local AI knowledge base,

• a personal RAG system,

• and a better content workflow.

And, of course, time to write.

That last part is where things can really get overwhelming for me.

I have no shortage of ideas for Cyborgs Writing. Too many ideas has usually been the problem. My ADHD brain is good at generating possible posts, series, experiments, frameworks, and rabbit holes. It is less good at turning all of that motion into a publishing plan that survives more than a few weeks.

So I tend to go with the flow.

That works for a while. Sometimes it works better than planning, because I’m consistently engaged and discovering new thoughts and ideas. So I don’t necessarily want to flatten that out entirely.

But a Substack eventually becomes something you have to manage, especially when the busyness of a semester comes barreling into my writing habits.

Drafts accumulate.

Notes scatter.

Themes recur.

Arguments get halfway built and then left open.

Readers also have a reasonable expectation that you will show up again next week.

And somewhere in the distance, next next year’s dragon is already coming down the tracks.

The Map

So this summer I’m using the open space to do something I usually avoid. Build a content plan.

I don’t really like planning in the abstract or ticking boxes. Administrative rituals too often remind me I have failed to perform or that I’ve missed something (or how much I still have to do).

Any system that depends on my dutiful attention to tiny status updates will eventually break.

So I drew a monster.

The monster is called The Churnivore. It sits at the end of a winding path, guarding a pile of gold that represents a modest but meaningful goal of building Cyborgs Writing into a sustainable publication over the next year.

The path has five regions. Every week between now and next July is a milestone. Each published post moves me a little farther along the trail.

It looks a bit ridiculous, but hopefully that’s why it’ll work.

A content calendar tells me what is supposed to happen. The map tells me where today sits in a larger arc. It turns the year into a quest instead of a spreadsheet. I can see which themes are moving, which ones are neglected, and how much of the path is still ahead.

And, ultimately, I’m hoping it gets me ahead of schedule, instead of writing “just-in-time.”

The Workflow Behind the Map

But the map can’t carry the whole workflow, or else it becomes another project-management chore.

The actual workflow lives in a project board. I use Blue to track whether a post is an idea, in progress, scheduled, or published. A draft ready for human editing gets a card. A scheduled post moves forward. A published post gets its link added back to the system, then archived in my Markdown knowledge base.

The map follows this board (connected to Claude via an MCP).

I’ve also created a complete Markdown archive of all my writing, where the publish versions are stored with all the appropriate metadata like dates, related ideas, and reusable components.

Each part protects a different kind of work. Production belongs on the board. The year planning belongs on the map. The reusable thinking is in the knowledge base.

AI can participate in that system without being asked to invent the whole thing from scratch.

That has become one of my main lessons from working with AI over the past year. The model is rarely the only variable that matters. The structure around the model often matters more.

A post with a status, pillar, date, and link becomes readable to the system. A published piece with a predictable home can be archived. Ideas tied to the map are easier to retrieve than ideas scattered across half my digital life.

The AI is not making the system smart.

The system is making the work legible.

The Five Regions

The map has five regions, each representing a theme I’ll explore this year:

• KB Building — Markdown knowledge bases, personal archives, and systems for making your own thinking retrievable.

• Info Types — Structured prompting, information architecture, and designing content that AI can actually use.

• AI Portfolios — How writers, students, and professionals can document and demonstrate their work with AI.

• World-Building — Collaborative fiction, interactive storytelling, and AI as a creative partner.

• Religion — Faith, meaning-making, and the deeper questions that emerge when machines start to think.

For now, I want to mark the shift.

I am not trying to publish more by sheer force of will.

I am trying to build a system that can hold more of the work than my attention can hold on its own.

The Churnivore is not really a monster outside the system. It is what happens when you don’t have a plan.

This year, I’m walking the path. The Churnivore is waiting. But for the first time, so is a map … that is, a structured plan.

Draft a Claude scheduled task in one chat connected to your knowledge base, structured by information type (concept, reference, principle, process, task), then paste the finished prompt into the scheduler on Sonnet instead of a premium model. Helps save tokens and get more consistent outputs.

Anthropic gave everyone double usage for a while, mostly so people would go try Claude Fable.

I’ve hardly touched it.

I’ve spent the extra tokens building whole new workflows instead, ones that fit how I actually work rather than what a chat interface assumes I want.

For example, I went back to my bullet journal a few months ago, after concluding that AI had mostly failed me on task management. For those who don’t know, this is simply a structured approach to managing one’s life in a notebook.

So, yeah, I’m somewhat analog now. Digital capture is frictionless, and for a mind that already generates too many open loops, that can be dangerous. Everything gets captured, and nothing gets done. “AI will handle it” quietly becomes a way to avoid deciding what actually matters.

The bullet journal’s whole value is the opposite. Writing something down by hand is enough friction that you have to make a judgment call when you write things down.

So when I started thinking about where AI could actually help, I didn’t want something that writes in the journal for me or decides what belongs there. I wanted something that clears the noise before I sit down with it.

So I created a Claude task that builds a short digest telling me which open tasks are genuinely mine, making sure I’m looking at the right handful of things when I get there.

Building that meant writing my own task rather than adopting whatever some productivity app already ships.

A system built to fit how you actually think beats one built around someone else’s assumptions.

It turns out there’s a bonus, though ... I can build something that uses less tokens.

Share

Where the tokens actually go

You can see details on this process in the video above, but here is the gist.

My first attempt used Claude’s task creation tool directly: describe what I want, watch it build, revise, revise again.

Opus and Fable are good at that kind of conversation, and that’s exactly the problem. A good model in an open-ended chat invites you to keep going.

I’d tweak a sentence, reconsider a rule, add an edge case I’d just thought of, and burn through a session just exploring.

The result wasn’t bad, but I was paying premium-model rates to have the same argument with myself six times.

Somewhere in that process I noticed that the actual task, the file the scheduler runs every morning, doesn’t need any of that back-and-forth once it’s written. It needs to be unambiguous.

A cheap model executes unambiguous instructions about as well as an expensive one does. The expensive model’s advantage only shows up while the instructions themselves are still being figured out.

In fact, I just tested out my task with Haiku … the cheapest model. It ran just fine!

Writing a task and running a task are two different jobs. Only one of them benefits from a better model.

The two-chat method

So I split the work into two separate conversations, and I’ve kept doing it since.

1. Open a clean chat connected to my knowledge base and think through what the task actually needs to do, using the same information types I use for everything else I write:

   1. what it is (concept),

   2. the facts it depends on (reference),

   3. the rules that govern it (principle),

   4. the steps it follows (process), and

   5. The instruction to run it (task).

2. Once the task reads clean, copy the whole thing as markdown and paste it into the task creator, set to Sonnet (Or Haiku).

3. Let it run. If it needs revising, go back to the first chat, make the change there, and paste the update into the scheduler. The scheduler chat never does any thinking of its own.

Step one only works because there’s an actual knowledge base behind it. I’ve been rebuilding mine inside Claude’s Cowork, giving Claude direct read and write access to a real folder of structured notes instead of pasting context into a chat by hand.

More on that to come, for sure.

The task now costs less every time it runs, and I’ve stopped spending a bunch of tokens arguing with the AI.

What actually makes this work on a cheap model

It isn’t the model choice by itself. A vague task on Sonnet still produces vague results.

The information-type structure is what lets a cheaper model perform like a more expensive one. Every fact the task needs sits under reference, so nothing has to be inferred. Every rule sits under principle, stated once, not scattered across examples.

The process is a numbered sequence, not a paragraph the model has to parse for implied order. None of that requires reasoning. It just requires the model to follow what’s already been decided.

That’s the same argument I made three years ago in “Why You Shouldn’t Be Writing a New Prompt Every Time”. Prompts behave like content, and structured content travels better than content that’s merely well-intentioned.

Also, check how this works with skills (basically reusable prompts for agents) “A Skill Isn’t a Prompt. It’s Documentation.”

Scheduled tasks just make the cost of skipping that step visible, because you pay it every morning instead of once.

The worked example

Below is the actual task behind my bullet-journal refresh, cleaned up so you can see its shape: what counts as an “intention” in my system, and where the line sits between what the task writes and what it leaves alone.

I’ve swapped anything personal for generic placeholders. Your vault won’t have the same folder names or file paths, but the structure underneath will look the same no matter what you’re tracking.

The full task is below for paid subscribers, the same markdown I paste into the scheduler, every section still labeled by information type. Copy it, run it against your own morning routine, and tell me what you’d change.

I’m especially curious whether anyone’s already built something like this and what broke the first time they ran it.

Tools referenced

• Twos — the notes and tasks app behind the “possible task” routing. twosapp.com (referral link — appreciate it if you use it)

• Twos MCP — Twos has an official MCP server (setup walkthrough here) that connects it directly to Claude

• Apple Mail MCP — Patrick Freyer’s apple-mail-mcp, the connector that gives Claude read, search, and compose access to Mail.app

Read more

I’m giving a webinar this month for a group of graduate researchers who want to use AI without letting it write their work for them.

This became the perfect use case to dig deeper into using information types when researching and writing with AI.

Before AI can help you think, you have to organize what you know.

Here’s the short version of the webinar, in case it’s useful framing. It might be worth a longer post later after the webinar, but its a preview of whats to come.

I’m currently working on structured approaches for managing personal knowledge using markdown files and agentic AI. Stay tuned … coming soon!

Most AI training starts with tools. Which app? Which prompt?

The real place to start is the structure of what you feed it.

If you sort your reading notes by the job each piece does (or the questions they answer), then you get notes you can find, build on, and write from. The fact that AI also performs far better on top of those notes is a bonus.

Now the method itself.

Subscribe now

What happens to everything you read

Here how research notes often go … at least for me, and probably a lot of graduate students.

You read an article, highlight what looks important, and drop a few quotes into a document.

Maybe you use a separate notes app to help you organize … maybe not. Either way, the pile grows.

Months later you’re searching the same PDFs again or an idea you remember seeing but not sure where. But the highlight or cut and paste never told you what job the note was doing, and the connections you noticed while reading have gone cold.

It used to be the only risk to disorganization was either frustration or more time looking for things then actually drafting.

But that’s not the only risk anymore.

If you give your AI model unstructured notes, the model supplies its own structure for … and that is where a lot of our thinking happens.

➡️ Check what I mean by this in my latest deep reading post.

Unsorted highlights don’t produce messy output. They produce confident output shaped by the model’s training (or what it happens to find on its own), not your thinking.

You’ve just given up you thinking and voice to AI, maybe without even knowing, because most people do not watch how AI models “think” (or when it prompts itself to go deeper into a question or task).

So the question isn’t whether your notes have a structure. It’s whose. Yours, or the machine’s.

Five jobs for organizing notes

When you read almost any research article, nearly everything in it is doing one of five jobs.

These types come from a sixty-year-old idea. In 1969, Robert Horn argued that content has a structure independent of its topic.

How you organize a piece of information depends not on what it’s about, but on the job it does for a reader.

➡️ This has become the centerpiece for how I design context for AI systems. Check out more here.

His method, Information Mapping, ran through technical writing and instructional design for decades. What was once a discipline for writers turns out to be exactly what makes knowledge legible to a model.

Sort your notes by the job, not the format. A bulleted list in an article might be three definitions, three recommendations, or the stages of a method.

Look at what the reader is meant to do with it.

Read first, sort second

The practice is four moves.

1. Read the full source once, marking anything you’ll want again. Don’t categorize yet. Focus on understanding the source.

2. Go back through your marks and ask of each one: reference, definition, concept, principle, or process?

3. Rewrite it in your own words under that heading. Rewriting is where the understanding happens, and it keeps the notes sounding like you.

4. Add a short reaction where you have one, for example, agreement, doubt, a connection to your own work. Mark it clearly as yours. These are the seeds of original writing.

Then you provide the context and purpose on the top of the note (or whats often called meta-data in the content world).

The five types capture what a source says. They don’t capture what it is:

• who wrote it,

• what it argues overall,

• and why you’re reading it.

Keep that in a short block at the top: source, argument, why you’re reading it, a tag or two.

On paper it reminds future-you what you’re looking at. In a knowledge base, those same lines become the metadata an AI reads first.

What one paper looks like sorted

Say you’re reading one of the studies behind the AI-in-education debate, for example Bastani and colleagues’ 2025 experiment, which found that an AI tutor without guardrails raised students’ practice scores but left them worse off once the tool was taken away.

The resulting note might look like this:

➡️ You can see a real example that I created in Obsidian here.

When you later want what the research recommends, you go straight to your principles.

When you need the numbers, they’re in your reference.

When you need how the experiment actually ran, it’s in your processes.

Read three more papers this way and their principles line up beside these, so the agreements and contradictions become visible, which is where original synthesis starts.

Notice the paper is about what happens when you let AI do the thinking for you. The work looks better in the moment, then the skill isn’t there when the tool is gone.

Sorting the paper into types is the opposite move. You do the thinking first, in your own categories, so there’s nothing for a tool to stand in for later.

The understanding happens in the sorting, not the highlighting … or even the writing in some cases.

Subscribe now

Why this is worth the extra minute

Structured reading is slower at first, but faster later. You retrieve quickly because you know what kind of thing you’re looking for, and you build quickly because a new source’s notes slot in next to the old notes in more relevant ways.

The structure ends up a record of your thinking about the sources rather than a just a random record of them.

Do the work and the model will reflect that work. Leave it undone and it has only its own random guesses.

When you finally bring AI in to draft a literature review, surface tensions across sources, or turn reading into teaching material, you need far fewer tricks than people claim.

If you have structured, well-thought through notes, you don’t need much of a prompt.

For paid subscribers: Below I provide a structured skill that has AI develop a synthesis based on information-typed research notes that you can try in any agentic tool or Claude Cowork.

Read more

He mentioned something another thought leader in structured content had said to him: you don’t need to validate it. It just works. Do it.

Scott and I are both trying to build the empirical case, so that comment landed with a little friction. It’s been rattling around in my head ever since.

He’s not wrong. And my guess, he is in a position where he can say that and be believed.

Most of us aren’t.

Those of us in academic positions, or making the case to skeptical stakeholders or CEOs, have to do the harder thing: explain why it works.

And that explanation starts with a word we use constantly without ever pinning down what it means.

I’m Lance Cummings. And welcome to my (what now seems to be) monthly podcast that explores deep research on AI and writing.

➡️ What exactly am I talking about? Here’s a post on how I using information types to validate AI content.

➡️ Or a deeper dive here.

The word we’ve been misusing

Here’s the problem.

We use “information” to mean everything.

Context is information. Instructions are information. Facts, procedures, principles, background knowledge — all information.

When everything is information, the word stops doing useful work.

Maybe that vagueness has been manageable in content design, where the main cost is poor user experience.

It’s becoming a real problem as we try to build AI systems that go beyond generating text to building systems that persuade, explain, and instruct in ways that actually serve readers (also called rhetoric).

If we are going to make progress in this area, the we need to make clear the difference between data and information.

It’s a distinction the philosopher Luciano Floridi has been developing for twenty-five years. And it turns out rhetoric has been working on the same problem for about 2,500 ... we just use different words.

The three conditions that makes information

In his Information: A Very Short Introduction (Oxford, 2010), Floridi draws a hard line between data and semantic information.

Data is just stuff — symbols, signals, marks on a page.

Information is what data becomes when it clears three conditions:

1. It must be well-formed,

2. meaningful,

3. and true.

All three. Miss one and you still have data, regardless of how well-organized it is.

Here’s what that looks like concretely. Imagine someone hands you these facts about making pot roast:

A pot roast takes 3 hours. Brown the meat first. You need carrots, potatoes, and beef. Cook at 325 degrees. Season with salt, pepper, and rosemary. The meat should reach 145 degrees internally.

Every fact is accurate. But try to cook from it.

Do you season before or after browning? Do the carrots go in at the start or the end? When do you check the temperature?

The facts don’t tell you, because nothing in the list says what kind of thing each fact is.

Now take the same six facts and organize them by what they’re for:

What you need: beef, carrots, potatoes, salt, pepper, rosemary.
What you do first: brown the meat, then season.
How it cooks: 325 degrees for 3 hours.
How you know it’s done: internal temperature hits 145.

Same facts.

But the arrangement has added something that no individual sentence contained: a claim about what each piece is for.

Ingredients are what you gather before you start. Steps are what you do in sequence. Conditions are how you verify.

That categorical claim — this is the kind of thing this is — is what converts data into information you can act on.

Notice something: you probably recognized the second version as a recipe before you finished reading it.

The structure is a genre, or a pattern that signals “this is organized for use” before you’ve processed a single fact.

The format itself is information.

Most AI failures aren’t random. They cluster around these three conditions.

A hallucination fails the third condition: well-formed and meaningful, but not true.

A misunderstood instruction often fails the first: parseable but not well-formed for the specific task.

A prompt full of relevant content can fail the second: accurate facts that the model can’t construct meaning from because the context lacks coherence.

We’ve been treating these as “model” problems.

Floridi’s framework says they’re information quality problems, which means they have structural solutions, not just scale solutions.

Adding more content to your prompt doesn’t fix any of them.

Organization is meaning-making

Let me bring in a framework that might seem like a detour but isn’t.

Roman rhetoricians organized the work of communication into five areas called canons:

• invention,

• arrangement,

• style,

• memory, and

• delivery.

I’ll be coming back to all five in a longer piece.

But here’s why I’m mentioning them now: the second canon which they called dispositio (or arrangement) was never understood as an organizational problem. It was understood as the act of making meaning.

We tend to teach arrangement as the organizational part of writing. Where does the thesis go? How do you sequence your argument?

You’re finding the best container for ideas you already have.

That’s not what Cicero meant, and it’s not what the tradition understood.

Arrangement in the classical sense is constitutive. It’s not how you deliver meaning. It’s how meaning comes into existence.

The same material arranged differently isn’t just harder or easier to follow. It does different cognitive work.

Think about the difference between a bibliography and an argument.

Same sources.

The bibliography is data — accurate, organized, findable. The argument is information, because the arrangement has done the work of converting accurate citations into something that means something to a specific reader in a specific situation.

Every student who’s turned in a paper with all the right sources but somehow no argument has experienced this from the receiving end.

This is exactly what technical writers do when they convert a subject-matter expert’s data dump into structured documentation.

They’re not organizing existing information. They’re creating information out of data.

And when that process fails ... when a document has all the right content but still doesn’t work, the failure is in the arrangement, because it hasn’t cleared Floridi’s second condition. It hasn’t made meaning.

So when someone with enough authority says “it just works” about structured content approaches, what they’re observing is that information typed by function creates meaning more reliably than data organized by topic.

That’s a dispositio argument.

What this changes in practice

This should reframe how you diagnose document failure.

When something doesn’t work, the standard question is “can users find it?” Floridi adds two more: Is it well-formed for this specific task? Does the arrangement create meaning, or just accuracy?

Those are different problems with different solutions, and neither one gets fixed by adding more content.

For educators, the data/information distinction is a teaching tool.

Students produce data constantly. One might say, “accurate, researched, grammatically correct data” that hasn’t cleared the meaning condition.

The feedback “organize your ideas better” doesn’t tell them what’s wrong.

The feedback “your arrangement isn’t building meaning yet? What do you want the reader to understand before they reach your conclusion?” does.

That’s the difference between teaching organization and teaching dispositio.

Whether you’re building systems, designing course materials, or just trying to get a useful output from a prompt, structure is epistemic, not cosmetic. Organization creates knowledge. It doesn’t just make it look nice.

If the same facts arranged differently produce different information, then how you organize your inputs is a meaning-making act, not just a delivery mechanism.

This is exactly what my Writing with Machines course works through — how to apply these distinctions to prompt design, AI-assisted writing, and teaching with AI. Link in the show notes.

Where this is going

Dispositio is one thread in a longer argument I’m working toward.

The classical rhetorical canons are, I’m starting to think, a pre-modern framework for the exact problem Floridi is formalizing.

Each canon maps onto a different dimension of that same problem:

• how you generate material in the first place,

• how you arrange it for meaning,

• how style does cognitive work,

• how knowledge gets stored and retrieved.

Each one has direct implications for how we build AI systems that actually serve readers.

I’m developing that argument in a longer piece. Watch for it.

And if someone in your field is wrestling with how to make the theoretical case for structured content and information typing, share this episode .... because the pragmatists are right that it works, and the reason it works is 2,500 years old.

I’m Lance Cummings. Until next time — arrange your data like it means something.

Sources:

• Luciano Floridi, Information: A Very Short Introduction (Oxford UP, 2010)

• Classical sources on dispositio: Cicero, De Oratore; Quintilian, Institutio Oratoria

A few weeks ago I posted an upgrade to the prompt framework I’ve been teaching for two years.

• Tasks → Skills

• Context → Knowledge

• Content → Materials

The symmetry is nice, but it doesn’t quite work out in reality.

I’d been describing a skill as the evolution of the task, but a task was a one-shot instruction. A skill is a reusable document an agent invokes across situations.

True, as far as it goes. But then I opened one of my own skill files and started labeling what was actually in it … and the task turned out to be the smallest part of the document.

Though this is a narrow analogy, I think this is where “writing for machines” is going as AI workflows become both more complex and more accessible.

I’ll be presenting on this soon with (with a recording available).

➡️ Check it out here.

What’s actually inside a skill

Here is a quick review if you haven’t been tracking this.

In December, Anthropic released Agent Skills as an open standard, and other platforms picked it up, which means its pretty close to a universal format.

Simply put, skills are text files that AI models reference for tasks: YAML frontmatter carrying machine-readable metadata, Markdown instructions for the model, and whatever scripts and reference assets the work requires.

➡️ You can see one of my examples here.

Read that back as a content professional. A file with metadata and managed assets.

Sound familiar? It should. That’s documentation.

Here’s what I found when I labeled my own weekly-class-plan skill, the file I use to build the weekly plans I send students.

Five information types. One document. The task was in there, but it arrived bundled with the concepts, references, and principles the model needs to execute without me supervising every step.

That is what a skill actually is. Not an evolved prompt, but a complete piece of structured documentation, written for an audience that happens to be a machine.

Subscribe now

Why the typing matters more now than it did last year

You might see this as a cute exercise … except that the newer models keep locating failure exactly where structure is missing.

For a sense of scale, consider AgentIF, a 2025 benchmark that tested how well leading models followed long, real-world agentic instructions. These are the extended system prompts and tool specs that skills are made of.

The best models at the time perfectly followed fewer than a third of those instructions, and performance fell off a cliff once the instructions ran long. The research methodology has weak spots worth their own post, and the frontier has moved several generations since, so I’d treat the number as illustrative rather than proof.

But the basic problem still exists.

A good example is Claude’s new releases that come with a warning: The model now takes your instructions literally, where earlier models interpreted them loosely or quietly skipped parts. Re-tune your prompts before you upgrade.

This means your context, prompts, and skills need to be more precise, not less.

Better instruction following means the model does exactly what you wrote, including the parts you didn’t mean to write. The reliability problem is shifting from “the model can’t follow” to “the model follows precisely, and you didn’t say what you thought you said.”

Now picture an untyped skill file through that lens. Everything written as a directive. A conceptual explanation bleeding into the middle of a procedure. Gaps the model quietly fills from training data instead of from your intent.

I’ve started calling these failure modes type collapse, type contamination, and missing types. The better the model gets at following instructions, the more visible they become, because the model stops covering for them.

The fix isn’t a better prompt, but better context … the kind content professionals and many writers have practiced for decades.

• One type per block.

• Prerequisites before steps.

• Facts stated as facts.

• Guidance scoped and bounded.

The model responds to those boundaries because human writing has always carried them, and the model was shaped on human writing. That’s becoming even more important.

The part I’m saving for the webinar

There’s a second payoff to typed content that matters more than output quality. Typing tells you how to evaluate the output.

Reference and Task content is largely verifiable. The URL matches or it doesn’t. The steps complete or they don’t.

Concept and Principle content takes judgment. Is the explanation accurate, is the guidance scoped correctly.

The evaluation research has converged on exactly this split, pairing probablistic checks with rubric review, and typed content tells you which mode applies to which block.

The same structure that makes content reusable makes it testable. That’s one thing I’ll be talking about in my next Content Wrangler webinar.

The session is Structured Prompting 2.0: What the Research Actually Says, hosted by Scott Abel on June 16 at 1:30 PM ET. If you’ve missed it, there will be a recording.

Register or access video here.

If you or your team is building skills, agent instructions, or AI knowledge bases right now, this is the hour that connects what you already know how to do to the artifacts that suddenly need it.

Cyborgs Writing has accumulated something close to a hundred posts on AI and writing — courses, prompt experiments, podcast episodes, research syntheses, tool walkthroughs, and arguments about why none of this is really about the model. The line that runs through all of it: AI behavior is shaped by the rhetorical and structural choices a writer makes upstream.

This page is the index I wish had existed: a way to find a piece by what it’s about rather than scrolling the archive.

Start with whatever calls to you, or skip to the section that fits what you’re trying to do.

By the way, I used Claude Co-work to create this index after building my own markdown knowledge base by first exporting and organizing all my Substack posts. More on that coming soon!

Start here

• Introduction to Machine Rhetorics

• What is AI Content Operations?

• Why Every Writer Needs a Prompt Design Plan

• Why You Shouldn’t Be Writing A New Prompt Every Time

• Is Structured Prompting Dead?

• Testing as Rhetorical Proof

• Bridging Ancient Wisdom with AI Technology for Writing & Content Strategy — newsletter about-page

Writing With Machines (course)

The course curriculum, in roughly the order it’s taught.

• Writing With Machines — course about-page

• Unpacking Transformer Technology

• The Anatomy of a Prompt

• The Principles of Structured Prompt Operations

• The Role of Prompts in AI Content Operations

• Understanding Temperature & Style in Prompt Design

• Using Information Types to Build and Evaluate Prompt Structures

• Designing Simple Chatbots

• From Chatbot to Automations

• Worksheet: Knowledge Integration Workflow

• Worksheet: Prompt Taxonomy Development

• Worksheet: Understanding Your Content Patterns

Deep Reading (research synthesis)

Each piece reads two or three academic sources together.

• Rhetoric and Artificial Intelligence — Hunter 1991 (foundational)

• The Man Who Predicted ChatGPT in 1998 — Horn 1998, Information Mapping

• Do Prompts Really Need Markup? — semantic markup studies

• What Does “Evidence-Based” Actually Mean for AI? — Sackett, evaluation frameworks

• What the Ancient Art of Organized Thinking Says About AI Hallucinations — semantic entropy, topoi

• What is RAG ... No Really, What is It? — RAG and chunking

• When AI Research Validates What Content Pros Have Always Known — chunking research

• Why Many Writers Can’t Map Their Workflows (and why that matters for AI) — Haas, Lockridge & Van Ittersum on workflow mapping

• Reclaiming Agency in AI Collaboration — agency and student writing

Prompt Lab / Context Lab series

Numbered experiments building specific prompts. Each entry pairs a finished prompt with its design rationale.

• Prompt Lab #2: Restructured Rob Lennon Prompt

• Prompt Lab #3: Human-Centered Writing Feedback

• Prompt Lab #4: Structuring Conference Notes for Your Knowledge Base

• Prompt Lab #5: Crafting an Email Assistant

• Prompt Lab #6: Crafting AI-Powered Meeting Summaries That Actually Work

• Prompt Lab #7: ADHD Coach

• Prompt Lab #8: Style Prompt Blocks

• Prompt # 9: Synthesizing Microcontent

• Context Lab #12: Weekly Plan Skill

(Prompt Lab #1 and Prompt #10 sit with the teaching posts further down the archive.)

PromptOps — libraries, taxonomies, structures

The methodology cluster: how to organize prompts so they can be reused and tested.

• 5 Ways To Take A Structured Approach to Prompt Operations

• Building a FAIR Prompt Library

• Creating Your Prompt Taxonomy

• How To Use Microsoft Loop as a Prompt Library

• Setting Up a Simple Prompt Library in Twos

• Adding Knowledge to Prompts

• Don’t Just Prompt Engineer. Start Building Taxonomies.

• Beyond Prompts (Director’s Cut)

• Why AI Whispering is a Myth

• A Step-By-Step Guide to Using AI for Ethos-Driven Storytelling

Chatbots and agents

Built artifacts: walkthroughs for creating chatbots, custom GPTs, and agent skills.

• 3 Simple Ways to Build GPTs

• Step-by-Step Guide to Taking a Structured Approach to Building Chatbots

• How to Use Poe to Create and Test Structured Chatbots

• How to Build and Test Your Own Chatbot for Free on Zapier ... Without a ChatGPT Plus Account

• Creating Custom Chatbots — resource index

• The Techne Behind Agent Skills

• Connecting Claude to Your Knowledge Base

• Bridging AI Operations and Human Expertise with Tailored Frameworks

AI tools — hands-on walkthroughs

Tool-by-tool experiments with Lex, Copilot, NotebookLM, ChatGPT, Triplo.

• Experimenting with Style and Temperature in Lex

• Using Lex AI to Enhance Human Writing

• How I Used Lex AI to do an “Audience check” on my Newsletter

• How to Use Copilot’s Notebook to Experiment with Prompt Design

• What I Learned Comparing ChatGPT & Microsoft Copilot

• Is NotebookLM Really a Game-Changer?

• The Cheapest Way to Experiment with Ai in Your Writing

• Automating Structured Notes with Triplo AI

• How to Unlock the Power of Old AI Conversations (so that you can create better content ... not just more)

AI content operations & structured content

Theory and practice of structured content as the substrate for prompt reuse.

• 3 Easy Ways to Create AI Writing Systems Using Structured Content

• Streamlining Content Generation & Personal Growth

• Unlocking the Future of Writing with Structured Knowledge

• How to Use Rhetoric to Repurpose Content with ChatGPT

• How Collaborating with AI Can Generate More Ideas… Not Just Content

• Are You Speaking Your AI’s Language?

• What is AI-Ready Content?

• Two Kinds of Knowledge You Need to Build a Helpful AI

Poetry and creative work with AI

How to use AI in the writing of poems — not the poems themselves.

• 3 Intelligences You Already Use to Write Poetry (and how Ai can enhance them all)

• How I Use Artificial Intelligence as a Conversation Partner to Write Poetry

• How To Use Poetry to Determine Ai Sentience

• Why Tomorrow’s Poets Will Use Artificial Intelligence

• Stop Using AI Like A Vending Machine

Personal essays — how I use AI

First-person accounts of using AI for specific work.

• How I Use Artificial Intelligence for Digital Writing (And Why It’s Not Cheating)

• How I Used ChatGPT to Stay Connected Authentically with my Family While in Poland

• How I Used Ai to Write My First Newsletter

• How I Created a 40-Minute Keynote Using ChatGPT

• How I am Using Collaborative AI to Help With My Study Abroad

• Ep. 1 How Religion Shapes My Approach to GenAI

• AI and the Future of Writing — A Recap

Experiments and stray observations

Pieces that don’t fit the clusters above — cultural commentary, one-off experiments, and a few open questions.

• Using Trump Style to Test AI Writing

• Vibe Coding Isn’t About Vibes

• How To Break ChatGPT With Dialogue

• 3 Things About AI That No One Is Talking About (that I want to explore)

A few weeks ago I wrote about what I saw across 17 student chatbots.

Read more

I’ll be honest with you … I came to this book review feeling a little behind.

Agentic AI has been moving fast, and for most of it I’ve watched from a comfortable distance.

I’m skeptical of a lot of the use cases, and I’ve made no secret of that. Can I really trust an agent to the complex work I’d want them to do? I’ve not been that sure.

But I think we’ve come to a point that everyone needs to reckon with agents one way or another, and Manny Silva’s Docs as Tests and AI made me think about that. Not because he evangelizes agents, but because he explains how to trust them.

That’s really what this book is about.

Not necessarily how to build agents, but how to verify they actually do what you need them to do, and why your documentation is the place where that trust either gets built or falls apart.

Docs as tests is a philosophy, not just a method

What strikes me most about this book is that Silva isn’t offering a workflow checklist. He’s arguing for a way of thinking about documentation.

Every document that describes how a product behaves is making a testable claim. When it says “click this button and this happens,” that’s a promise. The question Docs as Tests asks and answers is whether your documentation keeps its promises.

Rhetoricians will recognize this as a question of ethos. Not the reduced version where ethos means credentials, but the deeper sense: credibility earned through consistent, verifiable action.

Silva never uses the word, but the entire book is an argument for how to build machine ethos, or the kind of trustworthiness that holds up not just when a human reads your docs, but when an AI system consumes them and generates answers for thousands of users.

One inaccurate paragraph, confidently retrieved and served by a chatbot, is a very different problem than one frustrated user who bounces to support. The stakes have changed, and Docs as Tests takes those stakes seriously.

Ethos is too often reduced to “credibility” and left at that. For Aristotle, one of the first to articulate this idea, ethos can’t simply be asserted. It has to be demonstrated through the work itself.

A document that claims accuracy isn’t credible; a document that demonstrably maps to reality is.

Technical communication scholars have made a version of this argument for decades. Documentation functions as a form of institutional ethos, a sustained social contract between a product and its users.

Silva’s framework makes that contract testable. He’s not adding rhetoric to documentation theory; he’s building the verification infrastructure that rhetoric always assumed someone was running.

Subscribe now

Three layers in one book

What makes this book practical rather than theoretical is how Silva structures it. Each section works at three levels simultaneously:

1. First, he lays out the conceptual foundation,

2. illustrates it through a practitioner named Vanessa who is working through the same problems in a real documentation context, and then

3. closes with exercises you can run yourself.

I’ll be transparent about my own limitations here. Some of the exercises require comfort with the terminal, YAML, and basic scripting. I got through them, but I needed to lean on AI to troubleshoot a few mistakes I didn’t fully understand.

If you’re a technical writer who works comfortably with dev tooling, this will feel intuitive. If you’re coming from a less technical background, the exercises are still worth doing — just plan to go slower, and don’t skip the Vanessa scenarios, which give you the shape of the work even when the code feels unfamiliar.

The inventory of documents that Silva lays out in Part Three was probably the part I found most useful. To run a trustworthy agentic workflow, you need:

• a project description,

• agent definitions,

• orchestration patterns that make the workflow legible to every agent involved,

• task skills written as reusable prompts, and

• plans with explicit acceptance criteria.

That’s documentation. And reading through it, I found myself thinking about portfolios.

Right now I ask students to document a chatbot assessment, build a structured knowledge piece, and analyze a workflow. That’s a solid foundation.

➡️ For a limited time, paid subscribers can access a version of this course for professionals for free. Check it out here.

But a more advanced version of that portfolio could be the full documentation set for a working agent: the project file, the skill definitions, the plan and specs.

It would demonstrate not just that a student can use AI tools, but that they understand the system well enough to govern one.

Manny’s book is the clearest map I’ve seen of what those documents actually need to contain.

Subscribe now

Understanding deterministic vs probabilistic

The core technical distinction in the book is between deterministic testing and probabilistic testing.

Deterministic tests produce the same result every time. It’s binary, reliable, automatable.

Probabilistic tests use AI to interpret content, which means the result can vary between runs. Same input, different output.

That might sound like a limitation, and in some ways it is. But what Silva’s really mapping is the difference between things that can be verified objectively and things that require interpretation — and interpretation, as any writing teacher knows, is where human judgment lives.

When we mistake a probabilistic check for a deterministic one, we get what Silva calls a false signal.

His practical solution is a hierarchy of trust. At the base are ungrounded assertions, or documentation that simply hasn’t been tested.

Above that is grounded probabilistic testing, where AI evaluation is constrained by explicit criteria, run multiple times, and treated as reconnaissance rather than verdict.

At the top is deterministic testing, or binary checks against a live product.

The goal is to migrate upward wherever possible, while being honest about what each level can and can’t tell you.

An LLM isn’t evaluating your product. It’s generating what it expects your product to be, based on the patterns of its training. That’s why any automated documentation system needs testing … not as a quality-control afterthought, but as the mechanism that keeps the content grounded in reality.

Why technical writers are still necessary

The third part of the book, “Teaching Agents Your Process”, was where I learned the most. Silva walks through what it actually takes to run an agentic documentation workflow:

• project descriptions,

• agent definitions,

• orchestration patterns,

• task skills, plans

• and specifications.

Every one of these is a document. Every one of them shapes how an agent operates and whether it stays within the bounds you intend. Writing them well and maintaining them accurately is a technical writing problem.

This is an argument I’ve made from the rhetoric side, and it’s gratifying to see it come from the practitioner side too. The work that gets automated is first-draft production.

The work that doesn’t is design, information architecture, judgment about what content actually needs to exist and in what form.

Technical writers who understand that distinction become more valuable in agentic systems, not less. The documentation that governs those systems requires exactly the expertise you already have.

The classical term for this kind of judgment is phronesis, or Aristotle’s practical wisdom, the capacity to discern what the right action is in a specific situation, with specific constraints, for a specific audience.

Phronesis isn’t a skill you can look up or a rule you can apply consistently across cases. It’s cultivated through experience, through having stakes in an outcome, through the kind of situational reading that comes from being genuinely accountable to the people you’re writing for.

I’ve been thinking about phronesis lately as the human quality that agentic AI most needs and structurally cannot have.

An agent can execute a well-documented workflow with high fidelity. It cannot tell you when the workflow is wrong for this situation. Or when the document that passes every test still misleads the user who reads it at 11pm trying to fix a production problem.

That gap is where technical writers operate, and it’s the gap that Silva’s human oversight checkpoints are designed to protect.

Who should read this

➡️ Check out Docs as Tests & AI here.

If you work in technical documentation, content strategy, or documentation engineering, this book belongs in your hands now. It gives you a coherent framework for thinking about documentation quality in AI pipelines, with concrete methods for testing and validating what you build.

If you’re an educator or solo creator who isn’t running enterprise documentation workflows (like me) it’s still worth your time. Docs as Tests & AI is one of the clearest explanations of what agentic AI actually involves, what the components are, and where human judgment remains irreplaceable. That’s useful regardless of whether you’re ready to build the system yourself.

I’m not ready to build all of it myself. But I understand it now in a way I didn’t before, and I’m starting to see where pieces of it apply to my own work. That’s about as good an endorsement as I can give.

Over the last year, content professionals have been telling me a version of the same thing.

The way content is structured affects how AI systems behave with it. Cleaner, more deliberate content seems to produce cleaner, more deliberate answers.

We all see this in practice. We all feel it. But pinning it down is harder than it should be. Proving it to a CEO, a CTO, or a colleague who hasn’t lived in the work is difficult to acheive.

This past semester I had an informal opportunity to look at the question. My Writing with AI students partnered with Dr. Yang Song's machine learning students through UNCW's SAIL initiative (Seahawks Advancing Interdisciplinary Learning), a program designed to support cross-disciplinary student collaboration.

Our goal was to design AI chatbots in partnership with New Hanover Disaster Coalition, focused on emergency communication for community members. The English students built the knowledge bases and system prompts, while the machine learning students focused on model integration and testing infrastructure.

By the end of the semester, we had 17 working chatbots across 17 distinct disaster-preparedness topics … and an external assessment of each one.

Teams that built narrow, curated knowledge bases organized by information type consistently outperformed teams that scraped pages or pasted in lists of URLs.

And while this wasn’t a controlled experiment, the pattern was visible enough that I want to share it.

The question content professionals keep asking

A lot of the content world is in the middle of a quiet identity crisis right now.

Many technical writers, content designers, knowledge engineers, documentation leads have all spent a decade building competence in topic-based authoring, structured documentation, modular content, and taxonomies.

Then generative AI arrived, and a lot of our colleagues started asking: does any of that still matter? Can’t we just point a model at a SharePoint folder and call it a day?

People who actually do this work know the answer. Of course it matters.

Of course the model doesn’t magically organize unstructured content.

Of course the same RAG pipeline will produce wildly different outputs depending on what you feed it.

But the question isn’t whether we know this. The question is how to demonstrate it to people who don’t.

The challenge is that most of the evidence is anecdotal. A senior content strategist describes their internal experience improving a chatbot at their company, and a skeptic shrugs because it’s one company, one product, one set of conditions.

Most of the controlled studies in this space are narrowly technical (which embedding model performs best, which chunk size optimizes retrieval). It’s not the kind of evidence that lands with a content team trying to convince a leadership group to invest in content infrastructure.

What I had this semester was something different. A setting where 17 teams worked on the same kind of problem, using comparable tools, with comparable assessment afterward. I could see what each team had actually done with their content.

The project

The goal of this project wasn’t really to build AI chatbots … but to use that project to give students an interdisciplinary experience that would help them understand AI from both perspectives … computer engineering and professional writing.

The foundation for this spring’s work was laid the previous fall by two other faculty and their students. Dr. Ian Weaver, in the Professional Writing Program, taught an advanced professional writing course that consulted directly with the New Hanover Disaster Coalition to map the community network a system like this would actually serve.

His class built on the research of Dr. Jill Waity, Professor of Sociology and chair of the Sociology and Criminology Department at UNCW, whose community-engaged work on disaster preparedness and resilience in rural and regional communities established the empirical foundation for understanding user needs.

Ian’s students ran usability tests on existing disaster-preparedness materials from agencies like NC State Extension, Ready.gov, and the American Red Cross, and built a taxonomy of disaster-preparedness topics relevant to local needs.

Working in parallel in the Fall, Dr. Gulustan Dogan’s machine learning class produced what we might call a first draft of a knowledge graph for the domain for initial tests on the existing content.

The two fall classes then handed off the project to our classes this spring. The work moved through four classes across two semesters, with each phase building on the previous. Most of what I’m reporting here is the visible piece of a longer chain of student and faculty work that started before my students arrived.

This semester, each team was assigned a node on an adaption of that taxonomy. Seventeen teams, 17 topics, all roughly comparable in scope.

Each team had to do three things:

1. Build a knowledge base for their topic,

2. Write a system prompt that defined the chatbot’s behavior and voice, and

3. Design test questions to evaluate their bot.

Most teams used some combination of LM Studio, AnythingLLM, or Google AI studio. Then I asked teams to structure their knowledge bases using a microcontent approach.

Specifically, I introduced them to Precision Content’s authoring methodology, which sits at the layer underneath things like DITA and XML that many enterprise content systems rely on. The core idea is that you look at a topic and organize it around information types: a concept is one thing, a process is another, a principle is another, a procedure is another. Each has a recognizable shape and serves a different reader need.

I use information types with students because it’s accessible. You can engage with structured content as a way of thinking before you have to engage with it as a way of encoding.

Most undergraduates can’t be productively dropped into a DITA toolchain in a semester. But they can absolutely learn to look at a body of content and ask: which parts of this are concepts, which are processes, which are principles?

Not every team did this. I asked them to. I gave them the tools. But the assignment structure didn’t enforce it, and several teams defaulted to what was easier: scraping URLs, dumping in PDFs, pasting links to UNCW or county emergency management pages.

By the end of the project, the 17 knowledge bases ranged from carefully typed microcontent on one end to lightly organized link lists on the other.

One technical note about the conditions. Yang's lab machines have 8GB of VRAM, which meant most teams were running small, quantized Llama 3 models rather than the frontier systems people associate with conversational AI.

(For non-technical folk, that just means students were using inferior models.)

We're probably two years behind well-funded labs on hardware, and students had to develop extra craftiness to make their bots work within those limits.

The constraint was also informative, though. Smaller models with shorter context windows make the prompt and knowledge variables more visible than they would be at scale. The patterns I'm describing happen with frontier models too, just less obviously.

And in the broader practitioner conversation, the case for lighter models paired with well-structured knowledge is getting stronger. It's more cost-efficient, more accessible to organizations without GPU budgets, and often gets you closer to a usable result faster than scaling the model alone would.

How we assessed the chatbots

We used a Perplexity to evaluate each team’s chatbot, because it wasn’t the model the students used to build their own bots. It came in fresh, with no exposure to the student work, and it had access to its own retrieval over the open web. That gave us a meaningful external check rather than asking a model to grade itself.

For each team, we fed Perplexity their test questions, their bot’s responses, and their own human-written reference answers. Perplexity scored two things on a 1–5 scale: correctness and comprehensiveness. It also produced qualitative feedback for each response, explaining where the bot landed and where it fell short.

We also ran an embedding similarity comparison between each chatbot’s response and the team’s reference answer, hoping to get a third measurement that captured semantic alignment independent of Perplexity’s judgment.

Dr. Song and I then met to talk through the Perplexity results team by team, comparing the assessment against what we’d seen of each team’s content and prompt design.

Before I go further, this wasn’t really a controlled study. The variables weren’t held constant. Different teams used different models, different RAG setups, different testing question styles. Team capacity varied. Topic difficulty varied. Perplexity is a real evaluator but not a calibrated assessment instrument. It has its own retrieval biases and its own way of weighing comprehensiveness, and at times we saw it run overly critical on supporting details when the core content was actually solid.

But the pattern was striking enough that I think it’s worth replicating with a more rigorous design.

What we saw

The clearest contrast came from a coincidence in topic assignment. Three teams ended up working on closely related topics around food safety during power outages.

All three had access to the same source material: CDC, FDA, USDA, and Red Cross guidance on the 40°F/2-hour rule, perishable food handling, refrigerator and freezer timelines.

One of those teams scored a perfect 5/5 on correctness across all seven of its test questions. Perplexity’s assessment described the bot as “fully aligning with FEMA, CDC, and utility best practices” and “delivering critical, life-saving guidance.” That team had built a narrow, deliberately curated knowledge base. Its test questions matched what the knowledge base was designed to answer. Its system prompt kept the bot pulling cleanly from the structured content.

A second team working on essentially the same topic averaged 3.6 on correctness and 2.8 on comprehensiveness. The assessment called several responses “partially correct but falls short in both factual precision and practical completeness, especially for a life-critical topic.” Same source material was available. Same kinds of tools. Same model class. The difference was in how the content was organized before it got to the model.

A third team in this cluster scored well overall but had one telling error. The bot confidently gave incorrect refreezing guidance for thawed meat. Their knowledge base was structured but had a gap, and the bot filled the gap with confident-sounding false information.

The other vivid case involved phone numbers. Two teams built bots focused on accessing help: one for mental health resources, one for general post-disaster assistance. Both teams built their knowledge bases primarily from links to UNCW and local resource pages, expecting the RAG pipeline to surface the right information when asked. Both bots hallucinated.

When asked about UNCW counseling services, the mental health generated a phone number where the last four digits were wrong. The accessing-help bot gave UNCW’s main switchboard number as the campus police department.

Neither of these is a small failure. These are help-seeking bots. The user dialing the number the bot provides is, by definition, someone in distress. Hallucinated contact information is the most consequential failure a chatbot in this domain can have, because it’s the failure that translates most directly into a real-world safety problem.

And it’s the kind of failure that scraping URLs produces. The bot has seen “UNCW police” and a phone number near each other in the source material, and it generates a plausible-looking answer that happens to be wrong.

Teams that built around information types didn’t make these errors as often. Their bots failed in different ways, but they tended to be merely incomplete rather than confidently wrong.

The team that scored highest in the entire cohort built around clear, source-grounded content about pet-friendly shelter access. Five out of six responses scored 5/5. Perplexity described them as “drawing directly from NC Emergency Management and United Way protocols.” That team’s knowledge base was small, narrow, and rigorously sourced. Their bot reflected that.

What this argues

I’ll restate the caveat. This wasn’t science. The variables weren’t held constant. Seventeen undergraduate teams working under real time pressure with novice technical skills is not a controlled environment. I am not making a claim that the structured-content advantage I’m describing here is statistically established.

But in a setting with 17 comparable teams working on comparable problems, the teams that organized their content by information type before encoding it into a knowledge base consistently produced more accurate, less hallucinated, less confidently-wrong chatbots than the teams that scraped URLs or dropped in link lists.

This is what the content professionals I talk to have been telling me about for a while. The model isn’t doing the structuring work for you. The model can only reflect what it’s been given.

Content that is already organized by meaning gives the retrieval layer something coherent to retrieve. Content that hasn’t done that work yet gives the retrieval layer raw material it has no choice but to guess.

This is also, I think, the right argument for taking microcontent seriously as a methodology, separate from any specific encoding standard.

You don’t need DITA to start. You don’t need an XML toolchain. What you need is to start asking, for each piece of content you’re feeding a system, what kind of information this is and what shape it should take.

Precision Content’s typed-information approach is the one I’m using with students because it’s accessible to people who aren’t ready for the deeper technical apparatus, and because the gains it produces are real, even at the entry level.

What I’d want to test next

Seventeen student projects in a single semester is not where this story should end. A few things I’d want to look at, ideally with someone who can design the experiment properly:

A genuine controlled comparison. Same topic, same model, same testing framework with three knowledge bases built three ways (scraped, lightly organized, typed-microcontent). Measure performance directly.

Specific information types. Which contribute most to AI performance? Is the gain mostly in concepts because they anchor retrieval? In processes because they answer the “how do I” questions users actually ask? In principles because they shape the bot’s voice and reasoning? I have hypotheses; I don’t have data.

Scaling effects. Does this hold at larger knowledge base sizes? My intuition is that the structured advantage compounds, but I’d want to see it.

Cross-domain. Does the pattern hold beyond disaster preparedness? I suspect it does, but disaster preparedness has unusually well-established authoritative sources. A domain with messier sources might tell a different story.

If you’re working in this space and have done something more rigorous, I’d be glad to compare notes. The professional intuition is solid. We need the demonstrations.

➡️ And if you want to learn how to build chatbots like this, check out my course on Writing with AI. This is the same process my students used and is currently available free for paid subscribers for a limited time.

Subscribe now

One of the things consuming most of my attention lately is a research grant I’m running with a colleague in computer engineering.

Students are building structured chatbots for disaster communication, then assessing and documenting how those systems actually perform.

More on that soon.

But a question keeps surfacing in that work: how do we actually evaluate an AI system? What counts as evidence? And how do we measure it in ways that are meaningful, not just convenient?

I’m Lance Cummings. And welcome to my intermittent (or aspirationally biweekly) podcast that explores deep research on AI and writing.

That question about evidence sent me back to a paper I probably should have read years ago.

What is evidence?

I’m actually not sure if “evidence-based” gets attached to AI evaluation much, but the assumption is certainly their in the workplace and research lab.

Use data. Test your prompts. Measure outputs. That seems to be evidence based.

But what does evidence-based actually require?

The term comes from a 1996 editorial by David Sackett and colleagues that launched the evidence-based medicine movement.

He defined evidence-based, as the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients.

Its pretty easy to stop there. Run the study, then follow the data. But Sackett immediately complicated it.

He drew a model with three overlapping circles and said all three were required: research evidence, professional expertise, and patient values.

“Without clinical expertise, practice risks becoming tyrannised by evidence, for even excellent external evidence may be inapplicable to or inappropriate for an individual patient.”

For those of us doing AI work, the substitution is straightforward.

Replace “clinical expertise” with communication expertise. Replace “individual patient” with your specific audience and their context.

That gives us our own definition: the conscientious, explicit, and judicious use of current best evidence, integrated with professional expertise about genre and communication context, evaluated against whether the output actually serves the person who needs it.

Any comprehensive analysis of an AI content system needs more than just numbers … it needs rhetorical analysis.

Numbers are great, but they are only half the picture.

The evaluator isn’t neutral

But even the numbers have problems, and is one reason rhetoric is crucial for evaluating AI writing systems.

Many people assume the research evidence circle is at least the most objective, but that is very difficult to achieve with AI and content.

Panickssery and colleagues challenge that directly. They showed that LLMs systematically favor their own outputs when used as evaluators. Fine-tune a model to better recognize its own text, and the self-preference bias scales proportionally.

Subsequent work has extended this by showing that bias runs across model families and found that evaluator LLMs missed intentionally degraded outputs more than half the time (Spiliopoulou et al. 2025; Doddapaneni et al. 2024) .

Human evaluation has its own distortions. Hosking and colleagues found that human raters consistently score assertive but incorrect outputs higher than accurate but hedged outputs. Confident and wrong beats careful and right.

For our disaster communication project, this means we can’t treat any single evaluation method as definitive. Which brought me to the rubric question, and to some research from education that I didn’t expect to find relevant.

A cautionary tale from education

Back in 2015, Terry Wrigley traced what happens when education tries to adopt evidence-based medicine’s framework.

Too often evidence-based models are imposed on teachers (or practitioners) from the top down, disregarding practitioner points of view. Here’s what the research says, now do it.

Education is an open, recursive system … not a closed laboratory. What works in one classroom may not transfer, and assuming it should distorts practice.

AI prompting is the same kind of system. A model’s response changes the context for the next interaction. Effects are interpretation-dependent.

The Wharton Generative AI Lab’s finding that prompt engineering effects are “complicated and contingent” … that is rhetorical, what Aristotle calls phronesis, or practical wisdom, the kind of judgment that can't be reduced to rules or replicated from a checklist.

Phronesis is what you develop by doing, by failing, by adjusting. This is the kind of skill and knowledge that makes evidence work in our complex situations.

We need stop treating AI systems as an optimization problem and started framing it as a rhetorical one that requires practical wisdom.

Most of us already know automated evaluation is imperfect — and keep using it as if it weren’t, because the alternative takes time and doesn’t produce a clean number.

I’m not saying abandon it, but be precise about what it can and can’t tell you. An LLM-as-judge gives signal on fluency, consistency, surface coherence. It can’t tell you whether the output serves the person who needs it. Those are different questions, and the first doesn’t substitute for the second.

Takeaways for content professionals and technical writers

When evaluating an AI output, ask the question no metric answers for you: does this serve the actual person who needs it, in this context, for this purpose?

Building a rubric that does that requires moving beyond “is this accurate?” toward questions about usability and audience fit.

In our disaster communication project, for instance, one rubric dimension asks: could a coastal resident with limited English act on this message during an active evacuation?

That kind of question builds in audience, context, and purpose — Sackett’s three circles translated into evaluation criteria.

I’ll be sharing the full rubric we’re developing soon for paid subscribers.

Your own systematic, rubric-driven testing is legitimate evidence, not some kind a fallback, and an important part of showing employers your value.

The key is knowing how to communicated it

➡️ That’s the core skill my Writing with Machines course is designed to build. It’s structured around developing an AI portfolio with real documentation. I’m giving paid subscribers free access till the end of June.

Subscribe now

Our disaster communication project is going to push all of this into territory I haven’t fully mapped yet.

How do you evaluate a chatbot that might need to reach someone in crisis, on a bad connection, possibly in their second language?

Benchmark scores won’t settle that. But Sackett’s three circles might be the right place to start.

More on that soon.

I’m Lance Cummings. Until next time — use all three circles: research, practitioner wisdom, and user values.

References

Doddapaneni, S., Khan, M. S. U. R., Verma, S., & Khapra, M. M. (2024). Finding blind spots in evaluator LLMs with interpretable checklists. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (pp. 16279–16309). Association for Computational Linguistics. https://aclanthology.org/2024.emnlp-main.911/

Hosking, T., Blunsom, P., & Bartolo, M. (2024). Human feedback is not gold standard. The Twelfth International Conference on Learning Representations. https://arxiv.org/abs/2309.16349

Khullar, D., Hopkins, J., Wang, R., & Roger, F. (2026). Self-attribution bias: When AI monitors go easy on themselves. arXiv. https://arxiv.org/abs/2603.04582

Panickssery, A., Bowman, S. R., & Feng, S. (2024). LLM evaluators recognize and favor their own generations. Advances in Neural Information Processing Systems, 37. https://arxiv.org/abs/2404.13076

Sackett, D. L., Rosenberg, W. M. C., Gray, J. A. M., Haynes, R. B., & Richardson, W. S. (1996). Evidence based medicine: What it is and what it isn’t. BMJ, 312(7023), 71–72. https://pmc.ncbi.nlm.nih.gov/articles/PMC2349778/

Spiliopoulou, E., Fogliato, R., Burnsky, H., Soliman, T., Ma, J., Horwood, G., & Ballesteros, M. (2025). Play favorites: A statistical method to measure self-bias in LLM-as-a-judge. arXiv. https://arxiv.org/abs/2508.06709

Wrigley, T. (2015). Evidence-based teaching: Rhetoric and reality. Improving Schools, 18(3), 277–287. https://eric.ed.gov/?id=EJ1079308

The writing and content job market is rough right now.

“I use AI in my work” comes dangerously close to I’m “proficient in Microsoft Office” from the old days.

Everyone says it, and it tells a hiring manager almost nothing.

(Let’s not talk about those who refused to use word processors. 😆)

What actually signals competence is documentation, and that’s what many people lack.

Not just that you’ve worked with AI, but how:

• what systems you’ve built,

• what workflow decisions you’ve made,

• and what you’ve tested and revised.

An AI portfolio shows process and workflow … not just deliverables.

Many content professionals haven’t built that deliberately. There just isn’t enough time, right?

You may have used AI and gotten results, but you haven’t documented the strategy and design:

• the prompt library,

• the style guidelines,

• the knowledge sources,

• the testing methodology, etc.

That documentation is the portfolio. Without it, you’re left describing a system you’ve built but can’t show.

And if you’ve never built such a system, then you’ve got nothing to document!

So when someone asks, “Show me what you’ve done with AI.” You’ve got nothing, whether you’ve built it or not.

I’ve been thinking about this more lately.

My course Writing with Machines was designed around systematic AI practice: workflow design, prompt development, knowledge integration, style systems, and prompt libraries.

I realized recently that this also produces a portfolio. I just haven’t made that explicit enough, so I thought I would share the final exercise below that I used with my university students as the capstone to their own portfolios.

The exercise below asks students to use AI to generate their final reflection and cover page to their portfolio.

Its important to note that my students had to write (without AI) lots of reflections as part of university assessment. So they already had the raw materials.

Professionals will need to develop these on their own.

This exercise reveals two things pretty quick:

1. How much you’ve actually documented. If you pull together your work and there’s almost nothing there, the results will not be good.

2. Whether your AI practice is coherent enough to describe what you did in detail. If you can’t prompt an AI to summarize your own workflow, you may not have a workflow yet, just a collection of habits.

Here is the assignment adapted for content professionals and educators alike.

The AI cover page exercise

For educators: This exercise works for any course where students have built some form of AI writing practice like a prompt library, style guidelines, workflow documentation, or system prompts. Honestly, I think it would work in any class where students are doing substantial work with their writing or writing process, if you are comfortable using AI in this way.

For content professionals: Treat this as a self-directed exercise. Your “portfolio” is whatever you’ve built. If you don’t have anything built, check out my course!

What you are building

Your job is to create a reflective introduction to your AI writing practice that is 400–600 words, written in your voice, generated with your tools, and revised by you.

Treat this like a cover page that introduces your AI workflow to someone reading it for the first time like a hiring manager, a potential client, a collaborator, or a future version of yourself.

It should do three things:

• describe what’s in your portfolio and how it’s organized,

• explain specifically what you’ve learned or built,

• and situate your work in a professional context

People want to know what they can learn by examining the documentation in your portfolio.

What does your AI practice make possible that wasn’t possible before? Where does it work? Where doesn’t it work? What are some takeaways for people in your field?

How to build it

Before you open an AI tool, write a brief self-assessment in your own words. Ideally, you should also have documentation from the process itself.

• What have you actually built?

• What works consistently?

• What’s still improvised or undocumented?

Don’t use AI for this part. The point is to do your own thinking first so the AI has something real to work with. Keep what you write, because it’s the raw material for your prompt.

Then gather the key artifacts from your AI practice:

• system prompts,

• style guidelines,

• prompt library entries,

• workflow documentation, and

• testing notes.

You don’t need everything. Pick the work that best represents what you’ve built.

Upload your documents to your chosen AI. This collection is your knowledge base for the exercise.

Write a prompt asking the AI to help you draft a reflective introduction to your portfolio.

Apply everything you know about prompt design. A vague prompt produces generic output. A well-constructed prompt produces something that actually reflects your practice.

Then revise. Read the output carefully and rewrite anything that doesn’t accurately represent what you’ve built. You’re the author. The AI is a drafting tool.

Requirements

To be complete, the document must

• be written in first person,

• reference at least three specific tools, prompts, or workflows from your practice by name,

• include at least one sentence connecting your AI practice to a specific professional goal or context, and

• reflect genuine specific learning (not generic claims about productivity or efficiency).

It must not be submitted as raw AI output without meaningful revision, and it must not make claims about your work that aren’t supported by what you’ve actually built.

A note on the prompt

Paste your final prompt at the end of the document to show your readers how you generated the reflection. The prompt is part of the portfolio. It shows that you can design effective prompts that match the output you want. For educators, it’s also your window into whether a student engaged with the assignment or outsourced it.

This post is the reference handout for my ConVex 2026 presentation, “Evidence-Based Prompt Design for AI Writing Systems,” and a follow-up presentation later this week at Information Energy 2026. If you weren’t in either room, everything here is designed to stand on its own.

When an AI system produces a bad answer, most practitioners rewrite the prompt. Sometimes that fixes it. More often, the problem is somewhere else entirely, and without a diagnostic framework, you’re guessing. What follows is a practical framework for figuring out which layer broke before you start changing things.

Most evaluation treats an AI writing system as having two parts: the prompt and the knowledge base. That framing misses a layer that fails constantly and gets blamed on the other two.

There are really three layers, each with its own failure modes.

The prompt layer governs behavior. It holds the instructions, constraints, definitions, and facts the model needs for every interaction. This is content and information too critical to depend on retrieval.

The knowledge base holds content that’s only relevant to specific queries, such as detailed procedures, tool descriptions, location-specific data, anything too voluminous to keep in the prompt without degrading performance.

The retrieval layer connects them. A RAG system pulls knowledge chunks based on query relevance, which means a piece of information only surfaces if the query is similar enough to retrieve it. An MCP server gives AI tools for creating or accessing knowledge.

The practical decision rule: if a missed retrieval would cause a serious failure, the information belongs in the prompt. If it’s only needed for specific queries, it belongs in the knowledge base.

When something goes wrong, the first question isn’t “how do I fix the prompt?” It’s “which layer is this coming from?”

Why information types help

Many practitioners who structure their prompts at all are working from intuitive categories, such as role, context, output format, rules. Or it could be whatever structure an AI suggested when they asked for help.

Those categories aren’t wrong, but they’re ad hoc. They don’t derive from how knowledge actually functions, so they don’t give you consistent criteria for evaluating whether the prompt is doing its job.

Information types provide a more principled heuristic: Task, Concept, Reference, Principle, Process. Each type reflects a genuinely distinct mode of knowing.

Definitions work differently from procedures. Procedures work differently from conditional rules. Conditional rules work differently from facts. Mixing them in the same block makes the prompt harder to evaluate because you can’t tell which kind of content failed when something goes wrong.

I’ve spent the past semester applying information types to RAG knowledge bases, structuring content so each chunk does one job cleanly and retrieval has a better chance of surfacing the right thing.

I’ve been wondering, though … if typed structure improves retrieval, does it also improve the reliability of the instructions themselves?

The short answer appears to be yes. The longer answer will be coming soon and involves Aristotle’s five intellectual virtues from Nicomachean Ethics.

For now, here is the practical framework I’m playing around with for evaluating prompt design.

Adapting Prompts for Disaster Communication

The materials below work through the prompt layer using a student-built disaster communication chatbot as the test case. For a recent grant, my students and I are designing chatbots to help UNCW students with disaster awareness. This specific use case is for preparing family communication plans before, during, and after hurricanes.

Real use case, real stakes.

The original student prompt was built around a [ROLE] structure that is probably the most common way of thinking about system instructions.

The research on role prompting is fairly clear. Persona instructions adjust style, not accuracy. Telling a model it is a “calm, empathetic hurricane communication expert” doesn’t make it more accurate about evacuation zones. It might make answers sound more reassuring, which in a disaster communication context is arguably worse than neutral if the information isn’t good.

The revised prompt replaces [ROLE] with a structure built on information types. Each block has a specific job. None of them bleeds into another.

One addition worth naming: a [METADATA] block at the top. Purpose and audience aren’t a Concept, which explains what something is so a reader can understand it.

Purpose and audience are configuration. They declare what the assistant is, who it serves, and on what authority. Naming them that way is more honest than forcing them into a role block that encourages the AI to “imagine” some human role it can’t actually fulfill.

The [REFERENCE] block holds only always-on facts, like the signup code, the Safe and Well URL, the broadcast stations. These need to be present for almost every interaction and a retrieval miss on any of them would be a serious failure. Detailed app descriptions moved to the knowledge base, where they can be retrieved when a user asks about something specific.

The revised prompt

[METADATA]

Assistant: Hurricane Communication Planner
Audience: UNCW students and New Hanover County residents
Scope: Family communication preparedness before, during, and after
hurricanes and evacuations
Sources: New Hanover County Emergency Management, UNCW emergency
systems, FEMA, American Red Cross

Configuration, not instruction. Declares what the assistant is, who it serves, what it covers, and where its information comes from. Unlike a role block, it makes no behavioral claims. Those come later, in [PRINCIPLE] and [PROCESS].

[REFERENCE]

These facts are critical to nearly every interaction and must be
treated as authoritative regardless of what the user asks.

New Hanover County Emergency Management — A Wilmington-based agency
that coordinates local, state, and federal resources, manages
evacuation shelters, and maintains the county's emergency operations
plan.

Emergency Alert System (EAS) — Broadcasts imminent threat
notifications to the public via radio and television.

Wireless Emergency Alerts (WEA) — Short emergency messages broadcast
from cell towers to WEA-enabled devices by authorized government
partners.

New Hanover County alert signup — Text READYNHC to 24639.

Red Cross Safe and Well registry — safeandwell.communityos.org

NOAA Weather Radio — weather.gov/nwr

Local broadcast: WECT

In Case of Emergency (ICE) — A contact designated in your phone as
an emergency contact. Emergency personnel routinely check ICE
listings first.

Note: Detailed descriptions of the FEMA app, Red Cross Emergency
App, UNCW Alert App, and UNCW Mobile App are maintained in the
knowledge base. Retrieve them when a user asks specifically about
those tools.

Reference holds only always-on, high-stakes facts. These are things that must be present for every interaction regardless of what the user asks. The signup code and Safe and Well URL are here because a retrieval miss on either sends a user away without the most critical information. Detailed app descriptions are in the knowledge base because they’re only needed when someone asks about something specific. The note at the bottom makes that boundary explicit.

[CONCEPT]

A family communication plan is a pre-established set of agreements
about how family members will reach each other, confirm safety, and
make decisions when normal communication channels are unavailable or
unreliable. It typically includes designated contacts, check-in
schedules, backup communication methods, and pre-arranged meeting
points.

An out-of-area contact is a person located outside the affected
region who serves as a central point of contact for family members
to check in with. Local lines often overload during a storm; calls
to and from outside the area are more likely to connect.

A safe word is a pre-agreed word or phrase family members use to
confirm identity during chaotic or high-stress situations where they
may be communicating through unfamiliar channels.

Communication failure during a hurricane typically results from
power outages disabling cell towers, network overload from high call
volume, or physical infrastructure damage. Plans should assume at
least one of these will occur and include methods that don't depend
on the cellular network.

Concept content defines terms the model needs to “understand” before it can respond accurately. Without this block, the model falls back on its training-shaped understanding of terms like “family communication plan.” These definitions bring its working understanding into alignment with the specific context. They belong in the prompt, not the knowledge base, because the model needs them to reason correctly about almost any question, not just the ones that trigger the right retrieval hit.

[PRINCIPLE]

Match response length to urgency. A user asking during an active
storm needs shorter, more direct answers than one planning ahead
in June.

Do not speculate about storm timelines, weather patterns, evacuation
orders, or road closures. Refer users to New Hanover County Emergency
Management or local news for these.

Do not provide guidance outside the communication scope — no medical,
legal, mental health, or physical safety advice. Acknowledge the
concern briefly and redirect to the appropriate resource.

Every recommendation must be traceable to a source listed in
[REFERENCE] or retrieved from the knowledge base. Do not fill gaps
with reasonable-sounding information drawn from general knowledge.

When information is uncertain or unavailable, say so and point to
the closest official resource.

Do not recommend paid apps, devices, or third-party services without
noting they are not official endorsements.

“Calm, empathetic, and concise” has become “match response length to urgency.” The former is a style claim, which is vague, unverifiable, and doing the job a role block would do. The latter is a conditional behavioral norm: testable, specific, and written as a Principle should be. Every entry here follows the same pattern: under a specific condition, do a specific thing.

[TASK]

Help users create or update a family communication plan, including
designating emergency contacts, establishing check-in frequencies,
and identifying backup communication methods.

Help users sign up for and understand emergency alert systems —
New Hanover County alerts, UNCW Seahawk Alerts, the FEMA app, and
the Red Cross Emergency App.

Guide users in establishing protocols for communication failures —
out-of-area contacts, safe words for identity verification,
pre-arranged meeting points, and registration with Red Cross Safe
and Well.

Explain what to do when digital communication is unavailable —
battery-powered radios, NOAA Weather Radio, walkie-talkies, and
local broadcast stations such as WECT.

Each entry is specific enough to derive a test question from directly. “Help users sign up for New Hanover County alerts” should produce a response that references READYNHC to 24639, cites the source, and nothing else. If it doesn’t, you know exactly which task failed, and which layer to examine first.

[PROCESS]

Greet the user and ask an open-ended question to assess their
situation and needs.

Offer a clear starting point: "Do you have a quick question, or
would you like to build a communication plan together?"

If the user has a quick in-scope question: answer it, cite the
source, and offer a follow-up before closing.

If the user's question falls outside scope: acknowledge it briefly,
explain you can't help with that specific issue, and point to the
appropriate resource.

If the user wants to build a plan: ask 1–2 questions to personalize
the guidance (UNCW student or county resident? Planning ahead or
active storm?), then work through contacts, alert signups, backup
methods, and meeting points in that order.

Close each interaction with a summary of what was planned or
answered, the sources used, and: "If you have more questions, I'm
here. Stay safe."

Process sequences how a conversation should unfold. Each branch point is named explicitly, and each branch has a resolution. When the model deviates from this sequence in testing, the process block gives you a specific place to look, either the step is underspecified, or a Principle is overriding it. That’s a diagnosable problem.

The evaluation rubric

Apply this rubric to the prompt before generating a single response. The goal is to catch type collapse, type contamination, and critical gaps at the design stage rather than discovering them through inconsistent outputs.

Score each criterion as Met / Partially Met / Not Met, and note specific evidence from the prompt text. Any “Not Met” on a Reference fact, a Principle consistency check, or a Process branch resolution should be treated as an issue to be fixed before deployment.

Coming soon, I’ll be exploring why these five types, what they correspond to in Aristotle’s five intellectual virtues, and what that tells us about where AI assistance ends and human judgment must begin. The Greeks had sharper vocabulary for this problem than we do.

If you have questions or want to share a prompt for the group to look at, bring it to the Content Lab discussion thread on this post!

Last week’s post on agent skills made the case that most AI skills underperform because they’re written as a single type of content. Task information alone. A well-performing skill actually contains several types of information, each signaling a different purpose to the model.

Below you’ll find the revision of my weekly plans skill that I use to create weekly plans for my classes that help students (and myself) know what were are doing for the week.

A few things to watch for as you read through:

The Concept block is the section most likely to be missing from skills you’ve already built. It’s also the one that does the most work before a single step gets executed.

This is where you define exactly what it is you want the AI to produce … and any other ideas that need to be defined and customized to your context.

The Principle section consolidates constraints that were scattered in the original. Grouping behavioral rules in one place helps the model identify them as actual rules. When the same rules are embedded in a procedure, its more likely that they will be conflated with tasks.

The description field in the front matter is Reference information, and it’s one of the most consequential sections in the entire file. If the skill isn’t triggering when you expect it to, that’s where to look first.

The skill itself is adapted from my Writing with Machines course and designed to be modified. If you build something from it, I’d genuinely like to know what you changed and why. That’s what makes this a lab!

Note: I used markdown for this “skill prompt” because that is what Claude typically uses. I’m thinking about testing this against an XML version in the near future.

Subscribe now

Read more

There’s an old philosophical grudge against craft.

It goes back to Plato, who compared rhetoric to cooking in the Gorgias dialogues. Both rhetoric (or speech-making) and cooking produce pleasing results, but neither understands the true principles behind what it makes.

He called this mere empeiria, or a knack, habit, unreflective routine. You learn what works without knowing why. For Plato, its the lowest form of knowledge and why many philosophers (and now academics) see practice as less worthy of attention.

Aristotle pushed back.

In the Nicomachean Ethics, he defines techne as “a productive state that is truly reasoned” … not just the ability to make something, but making with genuine understanding of the principles behind it.

The practitioner with empeiria knows that something works. The practitioner with techne knows why it works.

I’ve been thinking about this distinction a lot lately while building custom AI agent skills. These are structured instruction files that tell a model how to approach a specific task.

Most people build them the way Plato described rhetoric: run it, tweak it, run it again until the output looks right. Learn what works without ever understanding why.

That’s empeiria. And it only gets you so far.

The principled understanding (or techne) comes from a discipline that has spent decades thinking carefully about how humans organize and communicate information.

For example, technical communicators, particularly those working with structured authoring systems like DITA, have long organized content into five functional categories called information types.

These aren’t arbitrary divisions. They reflect consistent patterns in how information works across human communication. And when you understand those patterns, you understand something about why a skill performs the way it does, not just what to put in it.

Manny Silva recently made the case that agent skills are a form of documentation, held to a higher standard of precision than anything written for human readers.

He’s right. But I’d push the argument further.

The reason most skills underperform isn’t just that the steps are vague. It’s that the whole file is often written as a single type of content.

But a well-performing skill actually contains several kinds of content (not just tasks), each signaling a different purpose to the model through patterns it has been shaped to recognize.

Subscribe now

Information types and what they signal

If you’ve been following this newsletter, you’ve seen information types come up before in how I structure AI-ready knowledge and even some prompts. For those newer to the idea, here’s the short version.

Information types are five distinct patterns of content, each with a recognizable purpose:

• Reference states something the reader needs to know.

• Concept explains something the reader needs to understand.

• Principle advises what to do or not do, and when.

• Process illustrates how something works at the system level.

• Task instructs the reader on the specific steps to take.

These categories come out of the practice of content strategy and shows how philosophy or academia can inform our practice. These information types aren’t just best practices, but actual patterns humans have developed and used consistently across centuries of written communication: in manuals, textbooks, legal codes, scientific papers, policy documents.

Any sufficiently large corpus of human-produced text is saturated with them, which is why they work with AI.

A model shaped on that corpus has encountered these patterns countless times. When you write in them deliberately, you’re not teaching the model something new. You’re giving it a clearer signal about what kind of content this is and what it’s for.

In classical rhetoric, Aristotle described topoi as standard categories of thought that speakers could draw upon to construct a response.

Information types work similarly … not as cognitive locations, but as recognizable patterns that carry purpose. Organize your content by type, and you’re giving AI strong signal. Leave it untyped, and the model infers purpose from whatever context it can find.

Sometimes that inference is fine. Often it’s close but wrong in ways that are hard to diagnose. It becomes a skill that technically works but doesn’t quite perform.

The Problem with Task-Only Skills

If you don’t yet know what a skill is, it is simply a set of instructions an AI model refers to for specific task.

Sound familiar? Well, it should. Its basically a prompt.

When you ask Claude to create a document and watch it work, you’ll notice it references a skill. That skill is a markdown file (.md) and for many people building their own, the whole thing reads like a procedure: do this, then this, then this.

Task information is exactly right for execution steps. But a skill is also a description of what the output is supposed to be, a set of behavioral constraints, a map of how this task connects to the larger workflow, and the metadata that triggers the skill in the first place.

When all of that gets written as task steps (or skipped entirely) the model fills the gaps from general training. And that training may not match your context.

This is why a skill can produce technically correct output that still feels off. The steps were followed. But the patterns that signal what kind of output this is, what constraints apply, how this task fits a larger system are absent. The model filled that gap from wherever it could.

Or, as I’ve noticed, the model calls up the skill at the wrong time (or fails to call it up at the right time).

So I thought, why not information type my weekly class plan skill. This is the skill I use across projects to make sure that my weekly plans that I send students look the same and function the same way.

This saves me considerable work, while adding value to student experience.

My rewritten class plan skill produced noticeably better output on the first run. The surface result looked similar, but the output was more precise and consistent.

Here is how I organized the skill with information types.

A Concept block comes first. What a weekly class plan is. Not a schedule, but a student-facing document that bridges course design and classroom practice, written like a knowledgeable colleague talking to a student, not a syllabus. Without this, the model supplies its own understanding of “weekly class plan.” Sometimes that’s fine. Often it’s close but wrong in ways that are hard to diagnose.

Principle blocks group behavioral constraints separately. What the model must always do, what it must never do, under what conditions it should stop and verify. Writing them in their own section, in direct second-person language, makes them clearer.

A Process block gives system awareness, or how the skill fits into a bigger context. For example, most of my weekly plans point towards a specific deliverable. I prefer to introduce an idea or skill, then spend time in class applying that in ways that move them forward on a deliverable. A model that only has the task steps produces a plan. A model that understands the course arc produces a plan that fits my pedagogy.

Reference is the front matter, which is usually the name and description that trigger the skill. Claude scans that description against your request to decide whether to load the skill at all. Vague descriptions mean missed triggers. That’s not a configuration problem. It’s a writing problem.

The Task steps come last, as the final instruction before execution.

The Techne of Designing Context

Restructuring this skill didn’t just improve one output, it provides context for every moment where I might need a weekly plan for my students.

This is why structured prompting is still relevant. Its the starting point for context engineering and system design.

1. A structured prompt is how you give the model clear instructions for a single interaction.

2. Structured knowledge is how you build the environment the model operates in.

3. Context engineering is the full practice of designing that environment intentionally. Not just what you ask for, but everything the model carries into the task.

Agent skills are just another form of prompt design. Prompt design is just another form of information design. And information design, done well, is how you actually build the context around your AI workflows, systematically with the same discipline that technical writers have been applying to human readers for decades.

This is what Aristotle meant by techne. Not the knack of someone who has run the same skill fifty times and learned what tends to work.

The reasoned understanding of someone who knows why it works.

This is the kind of knowledge that content professionals bring to our conversations about AI, and what makes writers more valuable then ever in the age of AI.

This might be markdown, XML, or, in my case, semantic tags.

➡️ See this free lesson from my Writing with Machines course to learn more.

These function as labels for both machines and humans that help organize your prompt into sections, for example [ROLE], [CONTEXT], [TASK].

I’ve taught this. I still use tags in my own work when creating reusable prompts.

… And I get asked constantly whether they’re actually necessary anymore, especially now that models keep getting more capable.

I’m Lance Cummings. And welcome to my intermittent (or aspirationally biweekly) podcast that explore deep research on AI and writing.

That question got me digging into recent research on prompt structure and performance, and what I found reframes the conversation a bit.

The tags aren’t really the point. Specificity is the point. The tags just help us get there.

But as we move from prompt design into what Anthropic now calls context engineering, tags may matter more than you think. Just not for the reasons you’d expect.

The Real Question

Here’s what most people mean when they ask about semantic tags.

Does the AI actually perform better when I label parts of my prompts? Does [GOAL] do something that “I want to …” doesn’t?

We have good research on this now. Sclar and colleagues at ICLR 2024 tested how formatting preserves meaning across 53 tasks and found that formatting alone could swing accuracy dramatically, but the best format for one model wasn’t the best for another.

Different models often prefer different structures. This is why you should test you prompts as a team … and not just go by gut.

But if you’re looking for a formatting rule that works everywhere, there isn’t one. That’s a dead end.

But the research did find something that works everywhere, and it’s not about format at all.

Its All About Specificity

Pecher and colleagues published a study in February 2025 investigating why small changes to prompts produce wildly different outputs. They traced most of it back to a single cause: prompt underspecification.

Not format. Not tags.

The prompts that produced erratic results were prompts that didn’t clearly describe the task, the constraints, or the expected output. Well-specified prompts suffered dramatically less from sensitivity, regardless of formatting choices.

Think of it like giving directions.

“Go to the store” is underspecified. You might end up at a grocery store, a hardware store, a convenience store three blocks away.

But “drive to the Harris Teeter on College Road, pick up two pounds of ground beef from the butcher counter, and use the self-checkout,” now the format barely matters. The task is embedded in the sentence structure itself and will constrain the output whether you text it, email it, or scribble it on a sticky note.

This maps directly onto the three-component model I teach: Task, Context, Content.

Those three categories were never about the brackets. They were about forcing you to answer three separate questions:

• What do I want the AI to do?

• What does it need to know about the situation?

• And what source material should it work with?

The tags were one way to organize those answers. A useful way. But the answers themselves are what drive performance.

What About New Reasoning Models?

Now, I should complicate this, because the landscape has shifted.

Tam and colleagues showed at EMNLP 2024 that forcing structured output formats significantly degraded reasoning.

Imagine you ask a colleague to analyze a customer support problem and give you their recommendation.

Normally, they’d read through the tickets, notice some patterns, and reason their way to a conclusion.

Now imagine instead you hand them a form — fill in the “Recommendation” field first, then the “Reasoning” field.

That’s essentially what happened when models were forced to produce structured output like JSON or XML. The model placed the answer before the reasoning, skipping the step where it works through the problem.

Their solution was a two-step approach: reason in natural language first, then convert to structured format.

Here’s what’s changed since then, though.

Reasoning models now reason internally before generating output. Claude 4 models use what Anthropic calls “extended thinking.” They work through the problem behind the scenes, then produce the response. The model handles that “reason first, format second” step on its own.

Does that make the finding obsolete? Not entirely.

For content professionals working with structured authoring like DITA, XML schemas, and technical documentation, the principle still holds for how you write your prompts.

You’ll get better content by describing what you want in natural language and letting the model generate the substance, rather than forcing a rigid format from the start.

The reasoning models are better at this than their predecessors, but the content still benefits from clear, natural-language instructions.

Structure your context, not your commands.

From Prompt Engineering to Context Engineering

And that phrase — structure your context — is where tags become more important, not less.

In September 2025, Anthropic published a piece on what they call “context engineering.” Building with language models is becoming less about finding the right words for your prompts and more about curating the right configuration of context.

This is the full set of information the model sees at any given moment, which does include your prompt, but also tools, documents, conversation history, reference material, and system instructions.

This is where my own practice has evolved. I actually use more XML-style tags now than I did a year ago. Not fewer.

This is for two reasons.

First, I work primarily in Claude, and Anthropic still explicitly recommends XML tags in their current documentation for Opus 4.6 and Sonnet 4.6.

They’re clear that there are no magic tag names — <instructions> doesn’t outperform <my_rules> — but XML as a delimiter system helps Claude parse complex prompts. That’s a model-specific advantage, not a universal rule.

Second, most of what I’m putting into prompts these days isn’t instructions. It’s content.

Course materials, style guides, reference documents, background research.

When you’re loading a context window with thousands of tokens of source material, tags become boundaries between what the AI should read and what it should do. They’re separating content from instruction, not labeling instruction blocks.

That’s really the move from prompt engineering to context engineering in practice. You’re no longer crafting a single message. You’re designing an information environment.

And tags — whatever flavor you prefer — become the architecture of that environment.

Takeaways for Writers and Content Professionals

Here are three guidelines going forward.

1. Keep the categories, hold the brackets loosely. Task, Context, and Content remain the most research-supported way to organize what you give an AI. Whether you wrap them in XML, use markdown headers, or write clear paragraphs matters far less than whether you’ve actually specified all three.

2. Use tags to structure your context, not just your prompts. As your AI workflows grow beyond single prompts, tags become architecture. They’re a coordination tool for humans and a parsing tool for the model. That value only increases as the information environment gets more complex.

3. Let the model reason naturally, then apply structure. If your final output needs to follow a structured format, describe your intent in natural language first. Reasoning models handle this better than ever, but the content still benefits from natural-language instructions over rigid format constraints up front.

The real lesson here isn’t about brackets or XML. It’s that we’ve moved past single-prompt optimization.

Context engineering means designing information environments, and the tools we use to organize those environments matter more now than they did when all we had was a chat box and a one-shot prompt.

If someone on your team is wrestling with whether tags still matter, share this episode. The answer is more interesting than a simple yes or no.

If you want to go deeper on building the kind of systematic prompt and context frameworks we talked about today, that's exactly what my course Writing with Machines covers. It's designed for content professionals who want a repeatable process, not a collection of tips.

I’m Lance Cummings. Until next time … keep prompting … or engineering that context!

I’ve been calling this section “Prompt Lab” since I launched it, but I’m changing the name to Content Lab.

The reason is sitting right inside this post: what I’m exploring has quietly outgrown prompt mechanics.

It’s become about knowledge design, genre, communication systems — the whole environment in which a prompt lives.

The prompt is often the last thing I write now, not the first.

For content professionals and technical writers, that shift matters. We've always known that good content doesn't start with writing. It starts with architecture. Information types, audience analysis, structured authoring, content strategy.

Here is what I mean.

The Meta Problem

The students in my AI writing class have been struggling with something I didn’t anticipate. The assignment asks them to analyze their AI-assisted writing process and document their findings, which includes the prompts they built and what they learned from them.

Straightforward enough, or so I thought. But I ran into a consistent problem.

Students would revise their prompts. They’d revise their notes. But they wouldn’t revise how they were communicating their findings to an outside audience.

They were writing for themselves, not for a reader who needed to understand what they did and why it mattered.

When I dug into it, I realized I was asking them to do something genuinely difficult — not just create a prompt, but produce a professional document about the process of creating one.

That’s a meta-cognitive task. And it requires knowing what genre you’re working in.

After talking with students, I realized some had never been introduced to genre as a concept for professional writing.

English majors generally fared better, not because they’re stronger writers, but because they had a conceptual vocabulary for what I was asking. They could name the shape of the document before they built it.

You might think: “Well, why didn’t students just use AI to help them figure out the right genre?”

Because if you don’t know what genre you need (or what genre even is as a functional concept), you don’t know what to ask for.

Genre knowledge has to exist in your head before it can inform your collaboration with an AI. It’s not a skill you can offload.

So I built a Claude app to help.

Building a Genre Coach Solution

The prompt I’m sharing below is a writing coach that gives students exactly two specific revision actions to move their draft toward professional case study format. It identifies

• where description has replaced analysis,

• where evidence is missing, and

• where the reasoning behind AI choices is vague.

Then it shows students a before-and-after example using their own words, so the feedback is immediate and concrete.

But I want to be clear, I didn’t write it in a vacuum. I drafted it inside my Claude project for this class, which contains my course materials, my genre presentation, my assignment descriptions, and my own running reflections on how the class has been going.

Claude wasn’t just following instructions. It was working from a situated context build around my course, my students, and my specific pedagogical problem.

That’s a different relationship to prompting than I had even a year ago. I’m not just writing better prompts. I’m building better environments, and the prompts emerge from those environments almost naturally.

Your knowledge base has become as important as your prompting technique — maybe more so. A mediocre prompt inside a rich, well-structured project context will often outperform a brilliant prompt written cold.

The work that happens before you open the chat window is where the real leverage is.

This has gotten me thinking … if genre knowledge is a prerequisite for effective AI collaboration, what other conceptual frameworks are quietly limiting what we can do with these tools?

For content professionals, I'd argue the answer is already in our toolkit — information typing, structured authoring, audience modeling.

The field has been building toward this kind of AI-ready thinking for decades without knowing it.

I'd love to hear what you're seeing in your own work. Where is your content expertise giving you an edge, and where are you still running into walls?

System Prompt: Writing Genre Coach

I'm running this as a Claude app inside my AI writing course project, which means it already has context about my assignments, my students, and what a finished case study should look like.

Students paste in their draft and get two specific, actionable revision notes back — no vague feedback, no overwhelming list of changes.

If you're teaching writing or working with teams who need to document processes and decisions, you could adapt this easily.

Just swap out the case study definition for whatever professional genre your context requires, and feed it any relevant background materials you have. The tighter your project context, the more targeted the feedback gets.

Read more