This post is the reference handout for my ConVex 2026 presentation, “Evidence-Based Prompt Design for AI Writing Systems,” and a follow-up presentation later this week at Information Energy 2026. If you weren’t in either room, everything here is designed to stand on its own.

When an AI system produces a bad answer, most practitioners rewrite the prompt. Sometimes that fixes it. More often, the problem is somewhere else entirely, and without a diagnostic framework, you’re guessing. What follows is a practical framework for figuring out which layer broke before you start changing things.

Most evaluation treats an AI writing system as having two parts: the prompt and the knowledge base. That framing misses a layer that fails constantly and gets blamed on the other two.

There are really three layers, each with its own failure modes.

The prompt layer governs behavior. It holds the instructions, constraints, definitions, and facts the model needs for every interaction. This is content and information too critical to depend on retrieval.

The knowledge base holds content that’s only relevant to specific queries, such as detailed procedures, tool descriptions, location-specific data, anything too voluminous to keep in the prompt without degrading performance.

The retrieval layer connects them. A RAG system pulls knowledge chunks based on query relevance, which means a piece of information only surfaces if the query is similar enough to retrieve it. An MCP server gives AI tools for creating or accessing knowledge.

The practical decision rule: if a missed retrieval would cause a serious failure, the information belongs in the prompt. If it’s only needed for specific queries, it belongs in the knowledge base.

When something goes wrong, the first question isn’t “how do I fix the prompt?” It’s “which layer is this coming from?”

Why information types help

Many practitioners who structure their prompts at all are working from intuitive categories, such as role, context, output format, rules. Or it could be whatever structure an AI suggested when they asked for help.

Those categories aren’t wrong, but they’re ad hoc. They don’t derive from how knowledge actually functions, so they don’t give you consistent criteria for evaluating whether the prompt is doing its job.

Information types provide a more principled heuristic: Task, Concept, Reference, Principle, Process. Each type reflects a genuinely distinct mode of knowing.

Definitions work differently from procedures. Procedures work differently from conditional rules. Conditional rules work differently from facts. Mixing them in the same block makes the prompt harder to evaluate because you can’t tell which kind of content failed when something goes wrong.

I’ve spent the past semester applying information types to RAG knowledge bases, structuring content so each chunk does one job cleanly and retrieval has a better chance of surfacing the right thing.

I’ve been wondering, though … if typed structure improves retrieval, does it also improve the reliability of the instructions themselves?

The short answer appears to be yes. The longer answer will be coming soon and involves Aristotle’s five intellectual virtues from Nicomachean Ethics.

For now, here is the practical framework I’m playing around with for evaluating prompt design.

Adapting Prompts for Disaster Communication

The materials below work through the prompt layer using a student-built disaster communication chatbot as the test case. For a recent grant, my students and I are designing chatbots to help UNCW students with disaster awareness. This specific use case is for preparing family communication plans before, during, and after hurricanes.

Real use case, real stakes.

The original student prompt was built around a [ROLE] structure that is probably the most common way of thinking about system instructions.

The research on role prompting is fairly clear. Persona instructions adjust style, not accuracy. Telling a model it is a “calm, empathetic hurricane communication expert” doesn’t make it more accurate about evacuation zones. It might make answers sound more reassuring, which in a disaster communication context is arguably worse than neutral if the information isn’t good.

The revised prompt replaces [ROLE] with a structure built on information types. Each block has a specific job. None of them bleeds into another.

One addition worth naming: a [METADATA] block at the top. Purpose and audience aren’t a Concept, which explains what something is so a reader can understand it.

Purpose and audience are configuration. They declare what the assistant is, who it serves, and on what authority. Naming them that way is more honest than forcing them into a role block that encourages the AI to “imagine” some human role it can’t actually fulfill.

The [REFERENCE] block holds only always-on facts, like the signup code, the Safe and Well URL, the broadcast stations. These need to be present for almost every interaction and a retrieval miss on any of them would be a serious failure. Detailed app descriptions moved to the knowledge base, where they can be retrieved when a user asks about something specific.

The revised prompt

[METADATA]

Assistant: Hurricane Communication Planner
Audience: UNCW students and New Hanover County residents
Scope: Family communication preparedness before, during, and after
hurricanes and evacuations
Sources: New Hanover County Emergency Management, UNCW emergency
systems, FEMA, American Red Cross

Configuration, not instruction. Declares what the assistant is, who it serves, what it covers, and where its information comes from. Unlike a role block, it makes no behavioral claims. Those come later, in [PRINCIPLE] and [PROCESS].

[REFERENCE]

These facts are critical to nearly every interaction and must be
treated as authoritative regardless of what the user asks.

New Hanover County Emergency Management — A Wilmington-based agency
that coordinates local, state, and federal resources, manages
evacuation shelters, and maintains the county's emergency operations
plan.

Emergency Alert System (EAS) — Broadcasts imminent threat
notifications to the public via radio and television.

Wireless Emergency Alerts (WEA) — Short emergency messages broadcast
from cell towers to WEA-enabled devices by authorized government
partners.

New Hanover County alert signup — Text READYNHC to 24639.

Red Cross Safe and Well registry — safeandwell.communityos.org

NOAA Weather Radio — weather.gov/nwr

Local broadcast: WECT

In Case of Emergency (ICE) — A contact designated in your phone as
an emergency contact. Emergency personnel routinely check ICE
listings first.

Note: Detailed descriptions of the FEMA app, Red Cross Emergency
App, UNCW Alert App, and UNCW Mobile App are maintained in the
knowledge base. Retrieve them when a user asks specifically about
those tools.

Reference holds only always-on, high-stakes facts. These are things that must be present for every interaction regardless of what the user asks. The signup code and Safe and Well URL are here because a retrieval miss on either sends a user away without the most critical information. Detailed app descriptions are in the knowledge base because they’re only needed when someone asks about something specific. The note at the bottom makes that boundary explicit.

[CONCEPT]

A family communication plan is a pre-established set of agreements
about how family members will reach each other, confirm safety, and
make decisions when normal communication channels are unavailable or
unreliable. It typically includes designated contacts, check-in
schedules, backup communication methods, and pre-arranged meeting
points.

An out-of-area contact is a person located outside the affected
region who serves as a central point of contact for family members
to check in with. Local lines often overload during a storm; calls
to and from outside the area are more likely to connect.

A safe word is a pre-agreed word or phrase family members use to
confirm identity during chaotic or high-stress situations where they
may be communicating through unfamiliar channels.

Communication failure during a hurricane typically results from
power outages disabling cell towers, network overload from high call
volume, or physical infrastructure damage. Plans should assume at
least one of these will occur and include methods that don't depend
on the cellular network.

Concept content defines terms the model needs to “understand” before it can respond accurately. Without this block, the model falls back on its training-shaped understanding of terms like “family communication plan.” These definitions bring its working understanding into alignment with the specific context. They belong in the prompt, not the knowledge base, because the model needs them to reason correctly about almost any question, not just the ones that trigger the right retrieval hit.

[PRINCIPLE]

Match response length to urgency. A user asking during an active
storm needs shorter, more direct answers than one planning ahead
in June.

Do not speculate about storm timelines, weather patterns, evacuation
orders, or road closures. Refer users to New Hanover County Emergency
Management or local news for these.

Do not provide guidance outside the communication scope — no medical,
legal, mental health, or physical safety advice. Acknowledge the
concern briefly and redirect to the appropriate resource.

Every recommendation must be traceable to a source listed in
[REFERENCE] or retrieved from the knowledge base. Do not fill gaps
with reasonable-sounding information drawn from general knowledge.

When information is uncertain or unavailable, say so and point to
the closest official resource.

Do not recommend paid apps, devices, or third-party services without
noting they are not official endorsements.

“Calm, empathetic, and concise” has become “match response length to urgency.” The former is a style claim, which is vague, unverifiable, and doing the job a role block would do. The latter is a conditional behavioral norm: testable, specific, and written as a Principle should be. Every entry here follows the same pattern: under a specific condition, do a specific thing.

[TASK]

Help users create or update a family communication plan, including
designating emergency contacts, establishing check-in frequencies,
and identifying backup communication methods.

Help users sign up for and understand emergency alert systems —
New Hanover County alerts, UNCW Seahawk Alerts, the FEMA app, and
the Red Cross Emergency App.

Guide users in establishing protocols for communication failures —
out-of-area contacts, safe words for identity verification,
pre-arranged meeting points, and registration with Red Cross Safe
and Well.

Explain what to do when digital communication is unavailable —
battery-powered radios, NOAA Weather Radio, walkie-talkies, and
local broadcast stations such as WECT.

Each entry is specific enough to derive a test question from directly. “Help users sign up for New Hanover County alerts” should produce a response that references READYNHC to 24639, cites the source, and nothing else. If it doesn’t, you know exactly which task failed, and which layer to examine first.

[PROCESS]

Greet the user and ask an open-ended question to assess their
situation and needs.

Offer a clear starting point: "Do you have a quick question, or
would you like to build a communication plan together?"

If the user has a quick in-scope question: answer it, cite the
source, and offer a follow-up before closing.

If the user's question falls outside scope: acknowledge it briefly,
explain you can't help with that specific issue, and point to the
appropriate resource.

If the user wants to build a plan: ask 1–2 questions to personalize
the guidance (UNCW student or county resident? Planning ahead or
active storm?), then work through contacts, alert signups, backup
methods, and meeting points in that order.

Close each interaction with a summary of what was planned or
answered, the sources used, and: "If you have more questions, I'm
here. Stay safe."

Process sequences how a conversation should unfold. Each branch point is named explicitly, and each branch has a resolution. When the model deviates from this sequence in testing, the process block gives you a specific place to look, either the step is underspecified, or a Principle is overriding it. That’s a diagnosable problem.

The evaluation rubric

Apply this rubric to the prompt before generating a single response. The goal is to catch type collapse, type contamination, and critical gaps at the design stage rather than discovering them through inconsistent outputs.

Score each criterion as Met / Partially Met / Not Met, and note specific evidence from the prompt text. Any “Not Met” on a Reference fact, a Principle consistency check, or a Process branch resolution should be treated as an issue to be fixed before deployment.

Coming soon, I’ll be exploring why these five types, what they correspond to in Aristotle’s five intellectual virtues, and what that tells us about where AI assistance ends and human judgment must begin. The Greeks had sharper vocabulary for this problem than we do.

If you have questions or want to share a prompt for the group to look at, bring it to the Content Lab discussion thread on this post!

Last week’s post on agent skills made the case that most AI skills underperform because they’re written as a single type of content. Task information alone. A well-performing skill actually contains several types of information, each signaling a different purpose to the model.

Below you’ll find the revision of my weekly plans skill that I use to create weekly plans for my classes that help students (and myself) know what were are doing for the week.

A few things to watch for as you read through:

The Concept block is the section most likely to be missing from skills you’ve already built. It’s also the one that does the most work before a single step gets executed.

This is where you define exactly what it is you want the AI to produce … and any other ideas that need to be defined and customized to your context.

The Principle section consolidates constraints that were scattered in the original. Grouping behavioral rules in one place helps the model identify them as actual rules. When the same rules are embedded in a procedure, its more likely that they will be conflated with tasks.

The description field in the front matter is Reference information, and it’s one of the most consequential sections in the entire file. If the skill isn’t triggering when you expect it to, that’s where to look first.

The skill itself is adapted from my Writing with Machines course and designed to be modified. If you build something from it, I’d genuinely like to know what you changed and why. That’s what makes this a lab!

Note: I used markdown for this “skill prompt” because that is what Claude typically uses. I’m thinking about testing this against an XML version in the near future.

Subscribe now

Read more

There’s an old philosophical grudge against craft.

It goes back to Plato, who compared rhetoric to cooking in the Gorgias dialogues. Both rhetoric (or speech-making) and cooking produce pleasing results, but neither understands the true principles behind what it makes.

He called this mere empeiria, or a knack, habit, unreflective routine. You learn what works without knowing why. For Plato, its the lowest form of knowledge and why many philosophers (and now academics) see practice as less worthy of attention.

Aristotle pushed back.

In the Nicomachean Ethics, he defines techne as “a productive state that is truly reasoned” … not just the ability to make something, but making with genuine understanding of the principles behind it.

The practitioner with empeiria knows that something works. The practitioner with techne knows why it works.

I’ve been thinking about this distinction a lot lately while building custom AI agent skills. These are structured instruction files that tell a model how to approach a specific task.

Most people build them the way Plato described rhetoric: run it, tweak it, run it again until the output looks right. Learn what works without ever understanding why.

That’s empeiria. And it only gets you so far.

The principled understanding (or techne) comes from a discipline that has spent decades thinking carefully about how humans organize and communicate information.

For example, technical communicators, particularly those working with structured authoring systems like DITA, have long organized content into five functional categories called information types.

These aren’t arbitrary divisions. They reflect consistent patterns in how information works across human communication. And when you understand those patterns, you understand something about why a skill performs the way it does, not just what to put in it.

Manny Silva recently made the case that agent skills are a form of documentation, held to a higher standard of precision than anything written for human readers.

He’s right. But I’d push the argument further.

The reason most skills underperform isn’t just that the steps are vague. It’s that the whole file is often written as a single type of content.

But a well-performing skill actually contains several kinds of content (not just tasks), each signaling a different purpose to the model through patterns it has been shaped to recognize.

Subscribe now

Information types and what they signal

If you’ve been following this newsletter, you’ve seen information types come up before in how I structure AI-ready knowledge and even some prompts. For those newer to the idea, here’s the short version.

Information types are five distinct patterns of content, each with a recognizable purpose:

• Reference states something the reader needs to know.

• Concept explains something the reader needs to understand.

• Principle advises what to do or not do, and when.

• Process illustrates how something works at the system level.

• Task instructs the reader on the specific steps to take.

These categories come out of the practice of content strategy and shows how philosophy or academia can inform our practice. These information types aren’t just best practices, but actual patterns humans have developed and used consistently across centuries of written communication: in manuals, textbooks, legal codes, scientific papers, policy documents.

Any sufficiently large corpus of human-produced text is saturated with them, which is why they work with AI.

A model shaped on that corpus has encountered these patterns countless times. When you write in them deliberately, you’re not teaching the model something new. You’re giving it a clearer signal about what kind of content this is and what it’s for.

In classical rhetoric, Aristotle described topoi as standard categories of thought that speakers could draw upon to construct a response.

Information types work similarly … not as cognitive locations, but as recognizable patterns that carry purpose. Organize your content by type, and you’re giving AI strong signal. Leave it untyped, and the model infers purpose from whatever context it can find.

Sometimes that inference is fine. Often it’s close but wrong in ways that are hard to diagnose. It becomes a skill that technically works but doesn’t quite perform.

The Problem with Task-Only Skills

If you don’t yet know what a skill is, it is simply a set of instructions an AI model refers to for specific task.

Sound familiar? Well, it should. Its basically a prompt.

When you ask Claude to create a document and watch it work, you’ll notice it references a skill. That skill is a markdown file (.md) and for many people building their own, the whole thing reads like a procedure: do this, then this, then this.

Task information is exactly right for execution steps. But a skill is also a description of what the output is supposed to be, a set of behavioral constraints, a map of how this task connects to the larger workflow, and the metadata that triggers the skill in the first place.

When all of that gets written as task steps (or skipped entirely) the model fills the gaps from general training. And that training may not match your context.

This is why a skill can produce technically correct output that still feels off. The steps were followed. But the patterns that signal what kind of output this is, what constraints apply, how this task fits a larger system are absent. The model filled that gap from wherever it could.

Or, as I’ve noticed, the model calls up the skill at the wrong time (or fails to call it up at the right time).

So I thought, why not information type my weekly class plan skill. This is the skill I use across projects to make sure that my weekly plans that I send students look the same and function the same way.

This saves me considerable work, while adding value to student experience.

My rewritten class plan skill produced noticeably better output on the first run. The surface result looked similar, but the output was more precise and consistent.

Here is how I organized the skill with information types.

A Concept block comes first. What a weekly class plan is. Not a schedule, but a student-facing document that bridges course design and classroom practice, written like a knowledgeable colleague talking to a student, not a syllabus. Without this, the model supplies its own understanding of “weekly class plan.” Sometimes that’s fine. Often it’s close but wrong in ways that are hard to diagnose.

Principle blocks group behavioral constraints separately. What the model must always do, what it must never do, under what conditions it should stop and verify. Writing them in their own section, in direct second-person language, makes them clearer.

A Process block gives system awareness, or how the skill fits into a bigger context. For example, most of my weekly plans point towards a specific deliverable. I prefer to introduce an idea or skill, then spend time in class applying that in ways that move them forward on a deliverable. A model that only has the task steps produces a plan. A model that understands the course arc produces a plan that fits my pedagogy.

Reference is the front matter, which is usually the name and description that trigger the skill. Claude scans that description against your request to decide whether to load the skill at all. Vague descriptions mean missed triggers. That’s not a configuration problem. It’s a writing problem.

The Task steps come last, as the final instruction before execution.

The Techne of Designing Context

Restructuring this skill didn’t just improve one output, it provides context for every moment where I might need a weekly plan for my students.

This is why structured prompting is still relevant. Its the starting point for context engineering and system design.

1. A structured prompt is how you give the model clear instructions for a single interaction.

2. Structured knowledge is how you build the environment the model operates in.

3. Context engineering is the full practice of designing that environment intentionally. Not just what you ask for, but everything the model carries into the task.

Agent skills are just another form of prompt design. Prompt design is just another form of information design. And information design, done well, is how you actually build the context around your AI workflows, systematically with the same discipline that technical writers have been applying to human readers for decades.

This is what Aristotle meant by techne. Not the knack of someone who has run the same skill fifty times and learned what tends to work.

The reasoned understanding of someone who knows why it works.

This is the kind of knowledge that content professionals bring to our conversations about AI, and what makes writers more valuable then ever in the age of AI.

This might be markdown, XML, or, in my case, semantic tags.

➡️ See this free lesson from my Writing with Machines course to learn more.

These function as labels for both machines and humans that help organize your prompt into sections, for example [ROLE], [CONTEXT], [TASK].

I’ve taught this. I still use tags in my own work when creating reusable prompts.

… And I get asked constantly whether they’re actually necessary anymore, especially now that models keep getting more capable.

I’m Lance Cummings. And welcome to my intermittent (or aspirationally biweekly) podcast that explore deep research on AI and writing.

That question got me digging into recent research on prompt structure and performance, and what I found reframes the conversation a bit.

The tags aren’t really the point. Specificity is the point. The tags just help us get there.

But as we move from prompt design into what Anthropic now calls context engineering, tags may matter more than you think. Just not for the reasons you’d expect.

The Real Question

Here’s what most people mean when they ask about semantic tags.

Does the AI actually perform better when I label parts of my prompts? Does [GOAL] do something that “I want to …” doesn’t?

We have good research on this now. Sclar and colleagues at ICLR 2024 tested how formatting preserves meaning across 53 tasks and found that formatting alone could swing accuracy dramatically, but the best format for one model wasn’t the best for another.

Different models often prefer different structures. This is why you should test you prompts as a team … and not just go by gut.

But if you’re looking for a formatting rule that works everywhere, there isn’t one. That’s a dead end.

But the research did find something that works everywhere, and it’s not about format at all.

Its All About Specificity

Pecher and colleagues published a study in February 2025 investigating why small changes to prompts produce wildly different outputs. They traced most of it back to a single cause: prompt underspecification.

Not format. Not tags.

The prompts that produced erratic results were prompts that didn’t clearly describe the task, the constraints, or the expected output. Well-specified prompts suffered dramatically less from sensitivity, regardless of formatting choices.

Think of it like giving directions.

“Go to the store” is underspecified. You might end up at a grocery store, a hardware store, a convenience store three blocks away.

But “drive to the Harris Teeter on College Road, pick up two pounds of ground beef from the butcher counter, and use the self-checkout,” now the format barely matters. The task is embedded in the sentence structure itself and will constrain the output whether you text it, email it, or scribble it on a sticky note.

This maps directly onto the three-component model I teach: Task, Context, Content.

Those three categories were never about the brackets. They were about forcing you to answer three separate questions:

• What do I want the AI to do?

• What does it need to know about the situation?

• And what source material should it work with?

The tags were one way to organize those answers. A useful way. But the answers themselves are what drive performance.

What About New Reasoning Models?

Now, I should complicate this, because the landscape has shifted.

Tam and colleagues showed at EMNLP 2024 that forcing structured output formats significantly degraded reasoning.

Imagine you ask a colleague to analyze a customer support problem and give you their recommendation.

Normally, they’d read through the tickets, notice some patterns, and reason their way to a conclusion.

Now imagine instead you hand them a form — fill in the “Recommendation” field first, then the “Reasoning” field.

That’s essentially what happened when models were forced to produce structured output like JSON or XML. The model placed the answer before the reasoning, skipping the step where it works through the problem.

Their solution was a two-step approach: reason in natural language first, then convert to structured format.

Here’s what’s changed since then, though.

Reasoning models now reason internally before generating output. Claude 4 models use what Anthropic calls “extended thinking.” They work through the problem behind the scenes, then produce the response. The model handles that “reason first, format second” step on its own.

Does that make the finding obsolete? Not entirely.

For content professionals working with structured authoring like DITA, XML schemas, and technical documentation, the principle still holds for how you write your prompts.

You’ll get better content by describing what you want in natural language and letting the model generate the substance, rather than forcing a rigid format from the start.

The reasoning models are better at this than their predecessors, but the content still benefits from clear, natural-language instructions.

Structure your context, not your commands.

From Prompt Engineering to Context Engineering

And that phrase — structure your context — is where tags become more important, not less.

In September 2025, Anthropic published a piece on what they call “context engineering.” Building with language models is becoming less about finding the right words for your prompts and more about curating the right configuration of context.

This is the full set of information the model sees at any given moment, which does include your prompt, but also tools, documents, conversation history, reference material, and system instructions.

This is where my own practice has evolved. I actually use more XML-style tags now than I did a year ago. Not fewer.

This is for two reasons.

First, I work primarily in Claude, and Anthropic still explicitly recommends XML tags in their current documentation for Opus 4.6 and Sonnet 4.6.

They’re clear that there are no magic tag names — <instructions> doesn’t outperform <my_rules> — but XML as a delimiter system helps Claude parse complex prompts. That’s a model-specific advantage, not a universal rule.

Second, most of what I’m putting into prompts these days isn’t instructions. It’s content.

Course materials, style guides, reference documents, background research.

When you’re loading a context window with thousands of tokens of source material, tags become boundaries between what the AI should read and what it should do. They’re separating content from instruction, not labeling instruction blocks.

That’s really the move from prompt engineering to context engineering in practice. You’re no longer crafting a single message. You’re designing an information environment.

And tags — whatever flavor you prefer — become the architecture of that environment.

Takeaways for Writers and Content Professionals

Here are three guidelines going forward.

1. Keep the categories, hold the brackets loosely. Task, Context, and Content remain the most research-supported way to organize what you give an AI. Whether you wrap them in XML, use markdown headers, or write clear paragraphs matters far less than whether you’ve actually specified all three.

2. Use tags to structure your context, not just your prompts. As your AI workflows grow beyond single prompts, tags become architecture. They’re a coordination tool for humans and a parsing tool for the model. That value only increases as the information environment gets more complex.

3. Let the model reason naturally, then apply structure. If your final output needs to follow a structured format, describe your intent in natural language first. Reasoning models handle this better than ever, but the content still benefits from natural-language instructions over rigid format constraints up front.

The real lesson here isn’t about brackets or XML. It’s that we’ve moved past single-prompt optimization.

Context engineering means designing information environments, and the tools we use to organize those environments matter more now than they did when all we had was a chat box and a one-shot prompt.

If someone on your team is wrestling with whether tags still matter, share this episode. The answer is more interesting than a simple yes or no.

If you want to go deeper on building the kind of systematic prompt and context frameworks we talked about today, that's exactly what my course Writing with Machines covers. It's designed for content professionals who want a repeatable process, not a collection of tips.

I’m Lance Cummings. Until next time … keep prompting … or engineering that context!

I’ve been calling this section “Prompt Lab” since I launched it, but I’m changing the name to Content Lab.

The reason is sitting right inside this post: what I’m exploring has quietly outgrown prompt mechanics.

It’s become about knowledge design, genre, communication systems — the whole environment in which a prompt lives.

The prompt is often the last thing I write now, not the first.

For content professionals and technical writers, that shift matters. We've always known that good content doesn't start with writing. It starts with architecture. Information types, audience analysis, structured authoring, content strategy.

Here is what I mean.

The Meta Problem

The students in my AI writing class have been struggling with something I didn’t anticipate. The assignment asks them to analyze their AI-assisted writing process and document their findings, which includes the prompts they built and what they learned from them.

Straightforward enough, or so I thought. But I ran into a consistent problem.

Students would revise their prompts. They’d revise their notes. But they wouldn’t revise how they were communicating their findings to an outside audience.

They were writing for themselves, not for a reader who needed to understand what they did and why it mattered.

When I dug into it, I realized I was asking them to do something genuinely difficult — not just create a prompt, but produce a professional document about the process of creating one.

That’s a meta-cognitive task. And it requires knowing what genre you’re working in.

After talking with students, I realized some had never been introduced to genre as a concept for professional writing.

English majors generally fared better, not because they’re stronger writers, but because they had a conceptual vocabulary for what I was asking. They could name the shape of the document before they built it.

You might think: “Well, why didn’t students just use AI to help them figure out the right genre?”

Because if you don’t know what genre you need (or what genre even is as a functional concept), you don’t know what to ask for.

Genre knowledge has to exist in your head before it can inform your collaboration with an AI. It’s not a skill you can offload.

So I built a Claude app to help.

Building a Genre Coach Solution

The prompt I’m sharing below is a writing coach that gives students exactly two specific revision actions to move their draft toward professional case study format. It identifies

• where description has replaced analysis,

• where evidence is missing, and

• where the reasoning behind AI choices is vague.

Then it shows students a before-and-after example using their own words, so the feedback is immediate and concrete.

But I want to be clear, I didn’t write it in a vacuum. I drafted it inside my Claude project for this class, which contains my course materials, my genre presentation, my assignment descriptions, and my own running reflections on how the class has been going.

Claude wasn’t just following instructions. It was working from a situated context build around my course, my students, and my specific pedagogical problem.

That’s a different relationship to prompting than I had even a year ago. I’m not just writing better prompts. I’m building better environments, and the prompts emerge from those environments almost naturally.

Your knowledge base has become as important as your prompting technique — maybe more so. A mediocre prompt inside a rich, well-structured project context will often outperform a brilliant prompt written cold.

The work that happens before you open the chat window is where the real leverage is.

This has gotten me thinking … if genre knowledge is a prerequisite for effective AI collaboration, what other conceptual frameworks are quietly limiting what we can do with these tools?

For content professionals, I'd argue the answer is already in our toolkit — information typing, structured authoring, audience modeling.

The field has been building toward this kind of AI-ready thinking for decades without knowing it.

I'd love to hear what you're seeing in your own work. Where is your content expertise giving you an edge, and where are you still running into walls?

System Prompt: Writing Genre Coach

I'm running this as a Claude app inside my AI writing course project, which means it already has context about my assignments, my students, and what a finished case study should look like.

Students paste in their draft and get two specific, actionable revision notes back — no vague feedback, no overwhelming list of changes.

If you're teaching writing or working with teams who need to document processes and decisions, you could adapt this easily.

Just swap out the case study definition for whatever professional genre your context requires, and feed it any relevant background materials you have. The tighter your project context, the more targeted the feedback gets.

Read more

Here’s a test case that reveals why casual AI use too often fails: disaster communication.

Imagine building a chatbot to help communities understand food scarcity risks during emergencies. You feed it information about food security, emergency preparedness, community resources.

People ask questions. AI generates responses.

When you test it with real life scenarios, it might seem to work well enough … but then breaks down under scrutiny:

“Should I eat meat that’s been in a refrigerator for one day without power?”

“How do I know if my community is at high risk?”

“What should I do if I can’t afford to prepare?”

The AI may generate perfectly grammatical sentences and sound authoritative. It might even cite statistics. But the responses can be generic, often irrelevant, and occasionally dangerous—exactly what you can’t afford in crisis communication.

This isn’t a hypothetical problem. I’m working on this with my students this semester. But the reason it matters for content professionals and writers generally has nothing to do with disaster communication specifically.

It matters because when AI absolutely has to be reliable, the same fundamental limitations appear every time. And too many people (and maybe organizations) are skipping the systematic work needed to address them.

Combining a machine rhetorics framework with user research is key to improving AI performance in real life situations.

Subscribe now

“Reasoning Models” Still Don’t Reason

AI doesn’t reason. It pattern-matches.

You might be thinking: “But what about the new reasoning models? Aren’t they moving beyond pattern-matching?”

It’s true that newer models show impressive capabilities that can break down complex problems, check their own work, and produce more consistent outputs. Some people argue this means we’re past the need for more systematic approaches to AI collaboration.

I’m skeptical.

Even with extended reasoning capabilities, these systems still operate on statistical patterns, not genuine causal understanding.

They’re better at appearing to reason, which makes their failures less obvious but not less consequential.

Even if reasoning models reduce some failure rates, they don’t eliminate the need for structured approaches when accuracy matters.

A more sophisticated pattern-matcher still benefits from well-structured prompts, organized content, and explicit knowledge mapping … just like a more powerful search engine still needs well-structured information architecture.

The machine rhetorics framework becomes more valuable, not less, as AI capabilities improve. Because the stakes get higher when people trust AI more.

AI doesn’t reason. It pattern-matches.

When you ask ChatGPT about food scarcity during disasters, it’s recognizing patterns from millions of texts where those words appeared together. It’s predicting what words typically follow other words in similar contexts.

What it’s not doing?

• Understanding cause-and-effect relationships

• Making logical connections between concepts

• Recognizing when general advice doesn’t apply to specific situations

• Knowing why certain information matters more than other information in emergency contexts.

This is why AI often produces responses that sound right but fall apart under scrutiny. The patterns are there. The reasoning isn’t.

From a practical standpoint, you can’t just “talk to AI” and expect sophisticated results, any more than you could walk into a library and expect books to organize themselves for your research project.

AI needs architecture. It needs structure. It needs humans to provide the logical scaffolding that helps it move from pattern recognition toward contextually appropriate responses.

This is where most casual AI use breaks down, and it’s why content professionals and other writers have an advantage most people don’t realize yet.

The Three-Part Framework

The disaster communication problem reveals three distinct challenges that too many people treat as one problem. Understanding machine rhetorics helps break down these problems and build solutions that are human-based.

1. AI doesn’t know what you actually need from a response (versus what it can plausibly generate).

2. AI can’t distinguish what’s relevant when everything’s mixed together.

3. AI can’t reason about logical relationships—it can only recognize patterns.

Each limitation requires a different solution. Each solution builds on the previous one. And you can’t skip steps.

Machine rhetorics means applying rhetorical principles to AI systems the same way content professionals apply them to any information experience:

• start with users,

• understand their needs through research, then

• structure systems around those insights.

Understanding machine rhetorics helps break down these problems and build human-centered solutions through strategic prompt design, structured content development. and rhetorical knowledge mapping.

This is what I’m exploring with students this spring, but the framework applies wherever content professionals need AI to be genuinely reliable.

Let me show you what each component does using disaster communication as the test case.

Subscribe now

1. Strategic Prompt Frameworks

The instinct is to build a chatbot by dumping information into it and hoping for the best.

Someone asks, “What should I eat if my power’s been out for a day?” and the AI generates a response pulled from its training data.

The problem? That response might be generically accurate but miss everything that matters about the person asking.

This is a usability problem and needs usability frameworks to solve. When content professionals design information experiences, its more than just getting users to click the right buttons.

Designing information experiences is about whether information is findable, understandable, and actionable for the right people, in the right place, at the right time.

Ultimately, an AI chatbot is an information interface, and it can fail for the same reasons any interface fails. It wasn’t designed around how real users actually encounter and act on information.

That’s why we started with user research in our disaster communication project. Last semester, our students conducted interviews, usability tests, and card-sorting exercises with the target audience for a disaster food security chatbot.

This semester, a new group of students is using those findings to develop test user questions and structured content for the AI system. The usability insights from that research are what will shape how we build the prompts.

Generic prompts produce generic answers. But when you structure the system around what usability research has revealed about real users, the AI has something meaningful to work with.

Here’s the difference. A basic system instruction might say:

You are a helpful assistant that answers questions about food safety during disasters.

A system prompt grounded in usability research looks more like this:

Task: Answer questions about food safety during power outages, prioritizing actionable steps the user can take right now with what they have available.

Here is some context from research that might be added to the system prompt:

• Users are often college students with limited storage, limited transportation, and tight budgets. They need to know what to do with the food they already have, not what they should have bought in advance.

• Users skip overly official language. Usability testing showed they abandoned documents that read like government pamphlets. They responded to conversational, direct guidance.

• A common misconception from card-sorting data: users grouped “frozen food” and “refrigerated food” together, not realizing they follow different safety timelines during outages. Responses should proactively clarify this distinction when relevant.

Now when someone asks “Is the chicken in my fridge still safe?” the AI can respond with specific guidance calibrated to the user’s actual situation — not a generic food safety lecture.

And the reason it can do that isn’t because someone wrote a clever prompt. It’s because usability research identified what users need, how they process information, and where their mental models diverge from expert knowledge.

The difference isn’t prompt length. It’s that the prompt is structured around a usability framework content professionals already work with:

• audience analysis drawn from real research,

• purpose defined by actual user needs, and

• content organized to match how people actually encounter problems rather than how experts categorize them.

This same rhetorical approach needs to be applied when building the information and content AI draws on in its responses.

2. Structured Content Development

Even well-structured prompts fall short when you feed AI disorganized information or rely on its black boxed training.

You might have:

• Research about food scarcity

• Statistics about emergency food systems, and

• Interview data from community organizations.

But if it’s all jumbled together, AI can’t tell what matters when, or how different pieces of information relate to each other, or what readers need to know before they can understand something else.

The solution is organizing materials using what content professionals call information types:

Reference information: Basic facts readers need to understand the situation. What counts as food insecurity? What do terms like “food desert” actually mean? What are current statistics?

Concept information: Frameworks for understanding the problem. How do sociologists think about food systems? What models help explain why communities experience food scarcity? How do emergency management professionals analyze risk?

Principle information: Cause-and-effect relationships that explain how things work. Why do food supply chains fail during disasters? What factors determine community resilience? How does economic stress affect food access?

Process information: How things unfold over time. What’s the typical progression of food scarcity during emergencies? How do communities respond? What happens when intervention comes too late?

Task information: Specific actions people can take. How should individuals prepare? What should community organizations do first? When should people seek additional resources?

When you organize information this way, two things happen.

First, you understand your own materials better. You can see gaps in your knowledge and recognize which types of information you’re missing entirely.

Second, when you structure this organized content into AI interactions, responses become more targeted, more appropriate to specific situations, and actually useful for different kinds of questions.

This is the second component: organizing information systematically so both humans and AI can understand how different pieces relate to each other and serve different purposes.

That’s what my students will be working on soon.

You can get a sneak peak at the taxonomy we are using in my Content Lab

3. Rhetorical Knowledge Mapping

Strategic prompts and structured content get you far, but they still leave a gap: AI can’t reason about relationships between concepts unless you map them explicitly.

When someone asks “Is the chicken in my fridge still safe?”, the answer depends on understanding how factors connect: How long has the power been out? Was the chicken frozen or refrigerated? Has the fridge stayed closed? What’s the safe timeline for this food type?

These aren’t isolated facts. They’re part of a reasoning structure where one decision point leads to another. AI can’t extract these logical relationships reliably from unstructured text alone.

This is where knowledge graphs come in—not as a technical database concept, but as a systematic way to map the reasoning structure that underlies domain expertise.

Our project taxonomy (developed from Fall 2025 user research) organizes disaster food security content into domains and topics: Food & Hydration, Water Safety, Power & Utilities, each with specific subtopics like “Refrigerator/freezer safety during outages.”

A taxonomy tells you what exists in the domain. A knowledge graph tells you how those things relate.

You identify the discrete concepts (refrigerated food, power outage, time thresholds, student constraints), then map the relationships between them: what’s safe for how long under which conditions, what constraints affect which options, what triggers what timeline.

The graph also captures context from user research, for example students confuse frozen and refrigerated timelines, trust action-oriented guidance, and face transportation and storage constraints. These rhetorical insights shape how relationships get structured.

With this mapped, AI can trace a path from the user question → refrigerated food entity → time without power → safety threshold → a more specific response about food.

Building a knowledge graph is about choosing which distinctions matter for this audience, which relationships drive decisions, which concepts need disambiguation, and which context shapes interpretation.

It’s rhetorical.

Content professionals already make these decisions when organizing information. The knowledge graph makes those decisions explicit and machine-readable so AI systems can reason with them.

How They Build On Each Other

You can’t skip steps in this progression, and disaster communication shows why with unusual clarity.

Without strategic prompt frameworks, you’re hoping AI will guess what constitutes appropriate crisis communication. Sometimes it guesses right. Maybe even most of the time. But what if it doesn’t? The consequences matter.

Without structured content, even well-framed prompts produce generic responses because AI can’t distinguish what’s relevant when facts, theories, procedures, and definitions are jumbled together. In crisis communication, generic responses can cause harm.

Without knowledge maps, AI can’t understand logical relationships, cause-and-effect connections, or contextual constraints that determine when information applies versus when it doesn’t. It will confidently generate plausible-sounding advice that violates the actual constraints governing emergency response.

But when you address all three challenges systematically, something shifts. You’re not just “using AI.” You’re designing information systems that help AI perform better while developing your own analytical capabilities.

The disaster communication scenario makes failures visible because stakes are high. But the same three limitations show up everywhere content professionals need AI to be reliable:

• technical documentation where incorrect instructions cause problems,

• compliance content where errors create liability, and

• strategic communication where off-brand messaging damages reputation.

The framework is the same. Only the domain changes.

What This Means for Content Professionals

True AI collaboration develops systematic approaches to these three distinct challenges—strategic framing, content organization, and knowledge architecture.

That’s why machine rhetorics is more than just prompt engineering.

It is a way of thinking that focuses on audience and context beyond the interface or immediate chat. Its a way of thinking that most content professionals already do.

This is why the workflow mapping advantage I wrote about last week matters. You can see how your work actually gets done.

Can you design systematic approaches to AI collaboration that leverage your rhetorical expertise while developing capabilities AI doesn’t possess? That’s the real question.

The Writing with Machines course teaches this framework—not just the three components, but how to apply them across different professional contexts where AI needs to be reliable rather than just plausible.

I’m testing it this spring in one of the most demanding contexts possible: crisis communication where generic responses can cause real harm. But the framework works anywhere content professionals need systematic AI integration—documentation, strategic communication, research synthesis, content operations.

If you’re interested in learning to apply this in your work, the beta course is opening up next week for paid subscribers!

Subscribe now

This taxonomy emerged from Fall 2025 user research conducted by students in our disaster communication project, and illustrates how I’m using taxonomies to organize and structure AI collaboration. Working with the New Hanover Disaster Coalition, students interviewed UNCW students about food security needs during disasters, conducted usability testing on existing emergency documents, and ran card-sorting exercises to understand how people naturally categorize disaster preparation information.

The taxonomy you see below builds directly on student work but has been adapted and refined using an MCP data modeling tool and AI to demonstrate how research-based categorization translates into structured content organization—which then becomes the foundation for knowledge graphs that AI systems can use reliably.

Interested in exploring this in your own work? Check out my course, Writing with Machines. Now available for paid subscribers.

Subscribe now

This taxonomy synthesizes work from Fall 2025, when students in ENG 404 (Advanced Professional Writing) and CSC 302 (Intro to AI) collaborated on the foundation for a disaster relief chatbot. ENG students conducted user research with UNCW students about food security during disasters, created taxonomies based on that research, and prepared source documents. CSC students built initial knowledge graphs using Neo4j tools.

This semester, ENG 326 and CSC 322 continue that work by developing functional chatbots. This taxonomy organizes the project so teams can divide the work systematically.

What is a taxonomy and why does it matter?

A taxonomy is a classification system that organizes information into categories and subcategories. You encounter taxonomies constantly—the folder structure on your computer, the way a grocery store organizes aisles, the categories on a news website.

For AI systems, taxonomies matter because they determine how information gets structured, stored, and retrieved. A chatbot answering questions about disaster preparedness needs to “know” that generator safety relates to power outages, which relates to food spoilage, which relates to health risks. Without an organizing structure, the chatbot has no map for navigating these connections.

The taxonomy below divides disaster preparedness into domains and specific topics. Each team will own one topic, becoming the experts responsible for gathering authoritative information, structuring it for AI use, and building a chatbot that handles questions in that area.

Taxonomy: Broad Disaster Preparedness Coverage

How teams will use this taxonomy

Each team receives one topic from the taxonomy. Over the semester, teams will:

Identify and collect source material. Find authoritative sources (CDC, FEMA, Red Cross, etc.) that address your topic. The Fall 2025 classes prepared some text files already—check there first, then supplement as needed.

Structure the content. ENG students will extract key information and organize it into a consistent format—concepts, tasks, rules, warnings—that works well for AI retrieval. This structured content feeds both your team’s chatbot and a shared repository.

Build and test chatbots. CSC students will build two versions: one without a knowledge graph (baseline) and one with a knowledge graph built from the structured content. Teams will compare performance to see how structured knowledge affects accuracy.

Contribute to the larger project. Your structured content joins a shared repository. Even though each team builds a focused chatbot, the combined work creates a foundation for a more comprehensive system—whether integrated this semester or built upon in future classes.

Authoritative sources by domain

You’re getting this because you’re a paid subscriber, and I want to make sure you didn’t miss this.

I just released a Deep Reading episode on the transparent technology myth which explains why most people can’t see their own workflows, and why that matters for AI integration.

If you haven’t listened yet, it’s the foundation for what I’ve building.

Writing…

Read more

You work with engineers every day. Smart people. They build complex systems. They understand workflows, pipelines, dependencies.

But ask them to map how they actually write documentation, and most of them can’t do it.

They’ll describe the end result. They’ll tell you what the doc should contain.

But the actual workflow? How information moves from subject matter expert interview to final publication? The tools, the handoffs, the format conversions, where things get stuck?

Blank stare.

Meanwhile, you can map that workflow in your sleep.

You know exactly where the bottleneck is—usually in review cycles, right? Or maybe it’s getting engineers to actually respond to edit queries. Or it’s that one legacy system that doesn’t integrate with anything.

You see the system because your job depends on seeing the system.

But here’s what I realized recently: Most people were literally taught NOT to see what you see.

And that’s not a small thing. That’s the difference between people who are prepared for AI integration and people who are scrambling.

I’m Lance Cummings, and you’re listening to Deep Reading, where we look at research that changes how we think about writing and AI.

Today I want to show you a piece of research from 1996 that explains why you’re positioned to lead AI integration in your organization—even though nobody’s probably told you that yet.

And why the engineers who are supposed to be “tech people” are actually starting from behind.

Subscribe now

The Invisible Tool Myth

In 1996, a researcher named Christina Haas published a study called “Writing Technology: Studies on the Materiality of Literacy.”

She was investigating something she called the transparent technology myth.

Here’s what she found: Most people are taught to treat writing tools as neutral instruments that don’t change the writing itself.

The tool is just a conduit. The thinking happens inside your head, and the tool just captures it.

This is how most of us were taught. “It doesn’t matter if you write by hand or on a computer—the writing is the same.”

Except that’s not true at all.

Haas showed that tools fundamentally shape not just the final product, but the process of composing itself.

Writing in Microsoft Word feels different than writing in MadCap Flare. Not because one is better, but because they organize information differently, which changes how you think about structure.

Managing documentation in Confluence creates different workflows than managing it in SharePoint.

Authoring in DITA with Oxygen XML forces you to think modularly in ways that a traditional word processor doesn’t.

The tool isn’t transparent. It’s active. It shapes the work.

But if you’ve been taught the transparent technology myth—that tools don’t matter, only ideas matter—then you literally can’t see how tools shape your process.

In the Classroom

I saw this play out exactly as the research predicts in a classroom last week.

I asked twenty students—computer engineers, English majors, cybersecurity students—to map their writing workflows.

The engineers were the most striking. These are people who live in version control systems. They can map a deployment pipeline in their sleep.

But their writing workflow? “I just... write it until it’s done.”

The English majors could describe intellectual moves—brainstorming, researching, drafting—because that’s what they’d been taught to name.

But the operational reality? The actual tools, formats, handoffs, file management, version control? That was supposed to be background noise. Irrelevant to “real” writing.

They couldn’t map their workflows because they’d been taught not to see the tools.

And this matters now because you can’t integrate AI into workflows you can’t see.

From Process to Workflow

In 2020, two researchers named Tim Lockridge and Derek Van Ittersum published a framework specifically about writing workflows.

They defined a workflow as “the tools and the process used for a writing task.”

Not just the cognitive process—brainstorm, draft, revise.

But the tool sequences. How work actually flows through systems.

They argued that you can’t understand contemporary writing without examining these tool sequences rather than treating technology as transparent.

Then in 2024, a researcher named Alan Knowles extended this to AI specifically.

He merged workflow thinking with something called Human-in-the-Loop principles.

The question isn’t “what can AI do?”

The question is “where does AI fit within existing work practices?”

Does it reduce friction? Does it open up new relationships with tools and tasks?

You can only answer that if you can see the workflow first.

Here’s where it gets interesting for content professionals.

In 2025, three researchers—Getto, Kelley, and Vance—applied this specifically to technical communication.

They pointed out something crucial: Technical communicators don’t operate under the transparent technology myth.

They never could.

Because technical communicators routinely attend to how tools shape content.

Style guide enforcement software. Content management systems. Structured authoring environments. XML editors. Publication pipelines.

Technical editors and content professionals have always had to think about where human judgment enters a production sequence and where automation can handle routine operations.

That’s the job.

You can’t manage content through production systems while pretending tools are transparent.

So when AI shows up, technical communicators are already prepared.

They already ask: Which tasks, at which stages, under what oversight conditions?

That’s exactly what Human-in-the-Loop AI collaboration requires.

Understanding AI Workflows

When most people try to integrate AI, they treat it like the transparent technology myth: just another neutral tool that captures thinking.

“Help me write this.”

But AI isn’t transparent. It’s deeply shaped by how you structure information, how you sequence prompts, what format you give it, what stage of the workflow it enters.

Getto, Kelley, and Vance describe the Human-in-the-Loop role as “manager of the process, validating outputs for whatever criteria they are aiming for.”

That’s familiar work if you’re a technical editor.

You already define tasks. Evaluate outputs against rhetorical criteria. Iterate based on results.

You already manage content through production systems.

AI is just applying the same analytical framework you’ve been using for structured authoring, content management, and publication workflows.

The question isn’t whether AI changes writing. Of course it does—tools always do.

The question is: Can you see well enough to manage that change strategically?

Here’s what this research tells us:

Most writers can’t map their own workflows because they were taught not to see tools as shaping work.

But content professionals—especially technical communicators and editors—were never trained that way.

You’ve always had to see the systems.

You understand that tools aren’t neutral. They shape how information flows, where friction happens, what’s easy and what’s hard.

You know where human judgment matters and where automation helps because you’ve been making those decisions for style guides, CMSs, and structured content for years.

That’s not a nice-to-have skill anymore. It’s the essential skill for AI integration.

Because you can’t integrate AI into workflows you can’t see.

And you can already see them.

The Value of Operational Thinking

The panic narrative says AI replaces writers.

But the research suggests something different: AI integration requires exactly the operational thinking that content professionals have been developing all along.

You’re not behind. You’re prepared.

You just might not have recognized workflow thinking as the strategic advantage it actually is. Because now that you can see the workflow, we need to understand what AI can actually do within it.

I’m Lance Cummings. Thanks for listening to Deep Reading.

If this changed how you think about AI and writing, share it with someone who needs to hear it.

And if you want the practical framework for mapping your workflows and integrating AI systematically, check out my newsletter Cyborgs Writing at http://www.isophist.com.

I’m also releasing a beta version of my Writing with Machines course soon for paid subscribers. For more information, click here.

Oftentimes, people are surprised that I teach writing … and AI. Isn’t that some kind of contradiction?

I often get this question: “What’s left for writers when AI can generate adequate text?”

The question itself reveals a misunderstanding about what makes writers valuable.

Last week I ran a workflow mapping exercise with twenty students from different disciplines—computer engineers, English majors, cybersecurity students, elementary education majors. I wanted them to map their actual writing workflows before we started integrating AI.

What happened revealed something important about the difference between describing a process and understanding a workflow.

The computer engineers struggled. They could explain their coding workflow in extraordinary detail. But their writing workflow?

“I just... do it until it’s done.”

The English majors did better initially. Brainstorming with clustering diagrams. Research with annotated bibliographies. Thesis development. Zero drafts for idea generation. Revision for argument structure.

But it became clear as we dug deeper that they were describing an idealized process or the intellectual steps they’d been taught. When I asked about the operational reality, they struggled too.

“I write in Google Docs, I guess?”

“My notes are... everywhere. Notebook, phone, scattered documents.”

“I don’t really have a system for organizing research.”

They could articulate the intellectual process. But the operational workflow remained largely invisible.

That distinction matters enormously for AI integration.

This pattern held across disciplines. Students from humanities backgrounds could articulate intellectual process, or the moves writers make.

Technical students could describe technical workflows, or the systems code moves through. But almost no one could clearly map their writing workflow, or the operational system through which their intellectual work actually happens.

In a world where AI supposedly makes writers obsolete, the ability to map operational workflows might be writers’ most valuable professional skill. Not just describing what you do, but understanding the entire system through which information flows.

This year I’m releasing beta access to my course, Writing with Machines, which helps writers and content professionals take a workflow approach to AI. Click below for more info.

Get Beta Access

The Irony of Technical Education

Those computer engineers already think in workflows for their technical work. They understand systems, information flow, version control, deployment pipelines, etc. They can map how code moves through their development environment.

But they haven’t applied that systems thinking to their intellectual work.

Meanwhile, English majors have been trained to articulate intellectual process, but typically in isolated academic contexts. They haven’t learned to map workflows in operational terms like tools, formats, handoffs, and information design.

Content professionals (or most professional writers) bridge both worlds. We understand intellectual process AND operational workflow.

We know that “writing” isn’t just thinking work. It’s information moving through systems.

• Research lives in databases or note systems.

• Drafts exist in specific tools with version histories.

• Reviews happen through particular platforms.

• Final outputs have required formats and distribution channels.

We already think in workflows because your work requires it.

When I research and collaborate with organizations like Motorola or Hitachi Energy on their content operations, the people who excel in operational roles typically come from content backgrounds. Not because they’re more creative (though they might be), but because they can see how information moves through a system and articulate what happens at each stage … in operational terms, not just intellectual ones.

That’s not a skill you automatically gain from learning to code or from studying literature. It’s a skill you develop from working in environments where content moves through systems and someone has to understand and improve those systems.

Why Workflow Mapping Matters Now

The ability to articulate intellectual process has always been valuable for writers. But AI integration requires workflow thinking … understanding the operational system, not just the thinking moves.

During the exercise, I asked students to identify friction points. Where do things slow down, where do you get stuck?

The English majors identified intellectual friction.

• “I hate writing conclusions.”

• “The transition from outline to draft feels like starting over.”

• “Synthesizing research into coherent arguments is hard.”

The technical students identified task friction.

• “Finding sources takes too long.”

• “Bibliography formatting is tedious.”

• “Blank page anxiety.”

But when I pushed them to think operationally, different patterns emerged.

One English major got specific about how she can write body paragraphs fine once she has her evidence organized. But she struggles with synthesizing everything into a conclusion that doesn’t just repeat what I already said.

This is process articulation. She knows the intellectual move that’s difficult (synthesis and elevation rather than summary). That’s valuable.

But then I asked: “Where are your body paragraphs when you’re trying to write the conclusion? What tool? What format? Can you see all your arguments at once, or are you scrolling?”

Long pause.

She then talked about how she writes each section separately, and by the time she gets to the conclusion, she’s forgotten what’s in the earlier sections … so she’s scrolling back and forth a lot.

Now we’re talking about workflow. The friction isn’t just intellectual. It’s operational. Information is organized in a way that makes synthesis difficult. The tool setup creates the problem as much as the intellectual challenge.

Compare that to “finding sources takes too long.” Also legitimate, but what’s the actual workflow friction? Is research scattered across multiple databases with different interfaces? Are you re-searching for things you’ve already found because notes aren’t organized? Is the problem identifying keywords, or is it that each source requires switching contexts and losing your train of thought?

Without mapping the actual workflow, AI integration becomes throwing technology at a vague sense of difficulty.

What AI Actually Needs From Writers

AI does one thing extraordinarily well: pattern-matching. It recognizes structures, suggests approaches, provides frameworks based on millions of examples.

But it can’t tell you which approach fits your specific situation unless you can map your actual workflow … not just the ideal process, but the operational reality.

The content professional who understands both the intellectual challenge (synthesis in conclusions) AND the workflow friction (information scattered across tools, requiring constant context-switching) can integrate AI strategically.

Maybe AI helps by consolidating key points from different sections. Maybe it suggests synthesis patterns. Maybe it’s not an AI solution at all. Maybe the workflow needs redesigning so information is visible when you need it.

The one who just says “help me write my conclusion” gets generic output because they haven’t mapped where the real friction lives.

This is why workflow mapping makes writers MORE valuable in AI environments, not less. Because effective AI integration requires:

• Understanding how work actually flows through tools and systems

• Identifying where in operational workflows AI can help (and where it can’t)

• Distinguishing intellectual challenges from operational friction

• Redesigning workflows when needed, not just automating bad processes

• Making both intellectual work and operational systems visible for improvement

These are content operations skills. The kind of thinking you develop from working in environments where content moves through systems and someone has to understand, document, and improve those systems.

The engineers in my class will build better chatbots than the English majors in a functional sense. No question. But content professionals already understand how to map the operational reality of knowledge work, not just describe the idealized process. Both are needed.

The Mixed Group Revelation

By the end of class, the groups that produced the richest insights probably won’t be the homogeneous ones. They will be the mixed groups where an English major, an engineer, and an education major talked through their processes.

The English major can learn that version control isn’t just for code. It’s a strategy for managing drafts. The engineer can discover that concept mapping could organize technical documentation. The education major can realize her strategies for making content accessible to children were universal design principles.

They can teach each other to see their own processes differently.

This mirrors what I see in successful content operations teams. The organizations that integrate AI effectively have people who can bridge technical capability with operational thinking. They need people who can build the systems AND people who can articulate the workflows those systems support.

Content professionals bring that operational thinking. Writers understand modularity, that writing happens in chunks (research phase, outline phase, drafting phase) and that AI works best with chunked information and targeted requests. You know the difference between extending capability at a specific friction point versus replacing judgment in complex synthesis.

You already think in workflows. You just might not have recognized it as a marketable skill.

What This Means for Your Work

The panic narrative says AI replaces writers because it can generate text. But text generation was never the whole job.

The actual job includes:

• Diagnosing what needs to be communicated and why

• Mapping how information flows through operational systems

• Identifying where automation helps versus where human judgment is essential

• Designing workflows that support quality at scale

• Maintaining meaning and accuracy across complex operations

These are content operations skills. Operational thinking applied to knowledge work. Workflow mapping, not just process description.

When organizations implement AI successfully, it’s because someone can map the operational workflows clearly enough to know where AI fits. When implementations fail, it’s usually because they’re throwing AI at undefined processes or, worse, automating workflows that were already broken.

The computer engineers in my class will learn to map their writing workflows this semester. But they’re starting from further back than content professionals and writers who already understand that content moves through systems.

We need to recognize that workflow mapping as the strategic skill that makes you valuable in AI environments.

What’s Next

I’m teaching this class all semester as a laboratory for systematic AI integration. The students are building an AI tool for disaster communication, which is high-stakes work where accuracy matters and hallucinations aren’t acceptable.

I’ll be sharing my thoughts as we move through the semester.

Next week I want to talk about the difference between AI pattern-matching and human reasoning. This is key to understanding AI workflows.

But you can’t integrate AI effectively into workflows you haven’t mapped.

If you want to move beyond casual AI use to systematic integration, I’m opening beta access to my Writing with Machines course for paid subscribers. It’s built around this exact challenge: mapping your actual workflows and integrating AI strategically at specific friction points.

The beta runs through this semester, giving you the complete framework while I refine it based on real-world feedback.

Learn more about the beta course → (paid subscribers only)

Subscribe now

P.S. Want to test this for yourself? Try mapping your own workflow for a typical content project. If you can articulate not just what you do but why you do it at each stage, you’re already ahead of most people trying to integrate AI. If you find it harder than expected—you’re in good company, and it’s worth figuring out.

Paid subscribers now get first access to something I’ve been developing for the past year: a complete course on systematic AI integration for content professionals and technical writers.

This isn’t just about casual ChatGPT tips or prompt hacks. It’s about building the operational framework you need to integrate AI strategically into your actual work …the kind that makes you more effective without replacing your judgment or expertise.

I’m making the beta version available to you paid subscribers, before the polished video course launches publicly. Here’s what that means and why it might interest you.

Subscribe now

What This Course Actually Is

Writing with Machines teaches content professionals how to move from casual AI use to systematic integration. It’s built around a core insight: AI works best when you can articulate your process clearly enough to identify exactly where it helps.

The course covers ten chapters:

1. Understanding Your Workflow - Making your invisible intellectual process visible and identifying friction points

2. What AI Actually Does - Pattern-matching vs. reasoning, and why the distinction matters

3. The Five Information Types - Structuring content so AI can process it effectively

4. Grounding AI in Knowledge - Building reliable knowledge bases instead of hoping for accurate responses

5. Prompt Design - Moving beyond trial-and-error to systematic prompt design

6. Pairing Prompts with Process - Matching specific prompts to specific workflow stages

7. Style and Temperature - Controlling AI output characteristics deliberately

8. The Structured Principles - How content operations thinking improves AI integration

9. Building Your Prompt Taxonomy - Organizing prompts as operational assets

10. Workflow Redesign - The capstone where you transform an actual process using everything you’ve learned

This isn’t just theory. Each chapter includes practical exercises, templates you can adapt, and frameworks you’ll use immediately. Chapter 10 has you redesign a real workflow from your work. That’s your deliverable.

What Beta Access Means

The beta course is complete. All ten chapters are written and functional. But it’s developmental. It’s text-based with exercises and templates rather than polished video lessons. You’re getting the systematic framework and practical tools, not production value.

Here’s what you get as a beta participant:

The exchange is simple: You get early access to systematic AI integration frameworks. I get insights that make the final course better. Both of us benefit.

Why Beta Access Now

I could wait until everything is polished and perfect before releasing anything. But that’s not how I work, and frankly, it’s not how good courses get built.

The best educational materials come from real interaction with real learners.

You’re professionals doing actual content work. You have real workflows, real constraints, real stakeholders. Your friction points are different. Your applications will be different. Your feedback will be invaluable.

Also, I’m learning things from my research and teaching this semester that are making the course better week by week. The disciplinary differences I wrote about in this week’s newsletter? That came from watching students map their workflows. Those insights are already improving how I teach the concepts.

If you wait for the polished version, you pay more and you miss the opportunity to shape it. If you join the beta, you get the systematic framework now (which you can use immediately) and you influence what the premium version becomes.

Who This Course Is For

What About Cost?

If you’re already paying for Cyborgs Writing. This beta access is included in your subscription. No additional payment required.

The premium video course, when it launches, will cost significantly more. Early beta participants will get preferred pricing on that, but right now, your subscription covers this beta access completely.

If cost is a barrier: I’m willing to provide access to people who can’t afford or prefer not to pay for the subscription but are genuinely committed to the process and feedback. If that’s you, email me at Lance.cummings@hey.com and we’ll work something out. The goal is to build an excellent course, not to restrict access unnecessarily.

Access beta form below.

Read more

I spent the last week of December doing something I hadn’t planned: modeling my own Substack archive as a knowledge graph.

I’ve been writing about AI-ready content, structured knowledge, and retrieval systems for two years now, and I’ve been wanting to see if I could structure my posts in a way that would make them more useful … to me, to readers, and eventually to AI systems that might help people navigate the archive.

What I didn’t expect was how much the modeling itself became a form of reflection.

So as my first post of 2026, I thought I’d give you some of my reflections on where I’m going with this Substack and how this knowledge graph exercise is helping me in the process.

The Year of Narrowing

First, some context. 2025 was a year of cutting away.

When I started writing about AI in education and technical communication, fewer people were doing it. That’s no longer true. Everyone writes about AI and plagiarism now. Everyone has opinions about classroom policy. The general conversation doesn’t need another voice.

And, well, if I’m honest, my ADHD brain kind of finds those discussions a bit boring these days.

So I began focusing on other things in information design by narrowing in on questions where my background in rhetoric and professional writing gives me something distinctive to say:

• How does retrieval-augmented generation work from a compositionist’s perspective?

• What can classical rhetorical frameworks tell us about prompt design?

• How do we test content systems, not just prompts?

Posts like “Is Structured Prompting Dead?” and “Testing as Rhetorical Proof” came from this narrowing.

So did the Deep Reading podcast episodes on topoi and AI hallucination.

I wrote less often but went deeper when I did.

This focus also shaped my teaching and research. I’ve got a couple academic articles in the works, and I’m starting this semester with an interdisciplinary grant project, where students and faculty from English, Sociology, and Computer Engineering are building a knowledge graph for a crisis food communication tool.

The structured content work I’ve been exploring publicly is now something I’m building with students, in real time, with actual users (and trying to bring into scholarly conversations).

My goal as an online writer has always been to bridge the space between the workplace and academia. In the world of AI, this is more important than ever.

What the Model Revealed

Back to the knowledge graph experiment.

I used Claude along with Neo4j’s data modeling MCP server, which is essentially a tool that helps you design graph structures.

For those unfamiliar: a knowledge graph represents information as nodes (things) connected by relationships (how those things relate). Instead of storing content as documents, you store it as a web of connected entities.

I started by asking: What are the meaningful units in my archive? Posts, obviously. But what else?

The first interesting decision was distinguishing Concepts from Rhetorical Frameworks. Concepts are the ideas I write about, for example structured prompting, RAG, AI literacy, vibe coding. Rhetorical Frameworks are the lenses I use to interpret those ideas—kairos, topoi, stasis theory, rhetorical proof.

I could have lumped these together. They’re all “topics” in some sense. But separating them forced me to articulate how the classical rhetoric is the interpretive layer through which I read everything else.

• Kairos informs how I think about vibe coding.

• Topoi shapes my understanding of RAG.

• Memory connects to knowledge graphs.

Mapping these relationships made explicit what had been implicit across dozens of posts.

The second decision was adding Claims. Most writers think of posts as being “about” topics. But when I added a node type for Claims—with properties like statement, claim type, and confidence level, each post became a collection of assertions at various stages of development. Some claims I state definitively. Others I mark as provisional. A few are speculative, ideas I’m testing rather than defending.

Here’s the knowledge layer of the model:

This helps me understand that my archive isn’t just a collection of articles about topics. It’s a slowly developing argument, with claims that build on earlier claims, supported (or not yet supported) by evidence, attached to concepts that relate to each other in particular ways.

The EXTENDS relationship between claims might be the most useful part. My December post on rhetorical proof extends the prompt testing work from November.

Seeing that connection mapped changes how I understand what I’ve been doing. Not as isolated posts, but intellectual development over time.

The Ontology Encodes the Epistemology

The usefulness of this activity goes beyond my own navel-gazing.

Knowledge graphs will become a key component to understand what we know (and what machines know) as AI becomes more ubiquitous.

Every knowledge graph encodes assumptions.

• What becomes a node?

• What becomes a relationship?

• What properties matter?

These aren’t neutral technical decisions. They’re interpretive choices about what counts, what connects, what gets left out.

The model I built privileges argumentation—claims require evidence, ideas have lineages. It privileges rhetorical tradition as a distinct layer of interpretation.

Someone else modeling the same archive might structure it entirely differently. A computer scientist might emphasize technical concepts and tool relationships. A historian might organize by period and influence.

The structure reflects a worldview.

This has implications for the AI systems we’re all building. When we create knowledge graphs for retrieval-augmented generation (and when we decide how to chunk content, what entities to extract, how to represent relationships) we’re doing philosophy whether we recognize it or not.

The ontology (or how the graph is built) shapes what the system can know and how it can know it.

But as an academic, I also understand the impulse to structure knowledge exists alongside the recognition that any structure is partial. A knowledge graph doesn’t capture knowledge. It creates a frame for retrieval. What lies outside the frame matters too.

There’s something almost mystical about this. The more carefully you map what you know, the more visible the boundaries of your knowing become. Structure reveals mystery rather than eliminating it.

The graph doesn’t contain the territory—it just makes certain paths through the territory easier to find.

This is where our own cultural and religious backgrounds, many of which are invisible, become key to understanding how we set up AI systems.

I don’t have this fully worked out. It’s one of the threads I want to pull on this year.

• How do philosophical and contemplative traditions inform how we design these systems?

• What would it mean to build a knowledge graph that acknowledges its own limits?

I’ve been wanting to explore this for a while now. Focusing my work even tighter around these topics and arguments give me the opportunity to go deeper in 2026.

What’s Ahead

For 2025, a few things:

I’ll continue the work on structured content, knowledge graphs, and testing systems—but now with the grant project providing a concrete laboratory. Students will be building something real. I’ll be writing about what we learn.

The Deep Reading podcast will keep going. Research that connects AI systems to rhetoric, history, and context. Short episodes, but substantive.

And I’ll be developing my course, Writing with Machines, which takes the prompt operations material I’ve been sharing and structures it into a learning path that helps writers and teams integrate AI in ways that enhance expertise without taking away agency.

Paid subscribers get access to the beta version of this course. I’ll be testing new material with them before it becomes a more polished offering through Firehead Digital Communications. If you want to dig into the structured prompting work and help me think through what’s useful, that’s the way in. More to come on this soon.

The university doesn’t provide resources for this kind of public scholarship. No course releases, no dedicated funding. Paid subscriptions help me continue the work—and I’m genuinely grateful for every one of them.

Subscribe now

One last thought.

Building this knowledge graph was supposed to be an organizational project. It turned into something more like an examination of what I actually believe about my own work. The modeling surfaced assumptions I hadn’t articulated, connections I hadn’t named, and limits I hadn’t acknowledged.

Maybe that’s what structured content does at its best. Not capturing knowledge, but creating conditions for reflection—for ourselves, and eventually for the systems we build alongside us.

Here’s to a year of structuring, and of honoring what escapes the structure.

Metadata as Practice

One thing I’m committing to this year: tagging my own posts with the structure I’ve been writing about. Not just talking about AI-ready content—making it.

Here’s the metadata for this post:

• Concepts: Knowledge Graphs, AI-Ready Content

• Framework: Memory, ontology

• Tools: Claude, Neo4j MCP

• Builds on: “Is Structured Prompting Dead?”, “Testing as Rhetorical Proof”

Claims I’m making:

1. The ontology encodes the epistemology (definitive)

2. Building a knowledge graph is a form of reflection (provisional)

3. Structure reveals mystery rather than eliminating it (speculative)

The confidence levels matter.

• I’m certain about #1.

• I believe #2 but want more evidence.

• #3 is something I’m testing—it might not survive contact with further thinking.

Over time, these tags will let me (and eventually you) trace how ideas develop across posts. That’s the hope, anyway.

What do you think?

Leave a comment

Last semester, my students and I built a writing feedback chatbot for our technical communication course. In testing, it worked beautifully. Clear, specific feedback that maintained professional warmth. We deployed it.

Within two weeks, students started reporting inconsistent experiences. The same submission structure that earned detailed feedback on Monday produced superficial responses on Wednesday.

One student showed me screenshots. For example, the chatbot had praised her conclusion as “effectively synthesized” in the morning, then flagged the identical paragraph as “needing stronger connections” that afternoon. Same prompt. Same model version. Same student text.

This isn’t a bug. It’s just the way it is when working with language models. And it’s why prompt testing requires something more rigorous than “try it and see if it looks good.”

Evaluation as Craft

The ancient Greeks had a term for what we need: kritikē technē … Or the art of judgment. The word kritikē comes from krinō (to separate, to decide), and technē means a teachable craft. Together, they describe a disciplined practice for evaluating the worth, correctness, and fitness of language.

The grammarians who were a part of the library of Alexandria developed kritikē into a systematic discipline that created repeatable procedures for evaluating texts. Working with multiple manuscript copies of Homer, they faced a problem familiar to anyone testing AI outputs: variant versions of the same content, with no obvious way to determine which was best.

Their solution was to operationalize judgment. They established criteria, collected evidence across variants, applied standards consistently, and recorded their decisions so others could follow the reasoning. Judgment became a craft that could be taught, reproduced, and improved.

This is exactly what prompt testing requires. When we evaluate AI outputs, we’re not asking “does this sound good?” We’re asking whether the outputs meet specific criteria reliably enough for a particular purpose.

That question demands a method—articulated standards, systematic procedures, transparent documentation.

Why Structured Prompting Demands Systematic Testing

When people claim structured prompting is dead, they’re usually working with single interactions or more dialogic collaborations. Ask a question, get an answer, move on (or continue working on that one instance). In that context, casual prompting often works fine.

But the moment you’re building something that needs to perform reliably across users, sessions, and contexts, you’re no longer in single-interaction territory.

This could be:

• a classroom assistant

• a documentation helper, or

• a content generation workflow.

You’re building a system. And systems require consistency that casual prompting can’t guarantee.

My research on prompt format bears this out. When I tested the same complex task across four different structures, the outputs varied dramatically. Not just in efficiency (processing time ranged from 64 to 120 seconds) but in character. The unstructured prompt produced exploratory, wandering responses. JSON triggered mechanical, compliance-document prose. Natural structure with clear sections generated focused, efficient communication.

Each format created different statistical conditions for the model’s token prediction. JSON tokens co-occur with technical documentation patterns in training data, so generating JSON-formatted input increases the probability of formal, exhaustive output patterns. Unstructured conversational input co-occurs with exploratory discussion, so the model follows those statistical tendencies.

Structure gives you leverage over consistency, but only if you verify that your structure actually produces the consistency you need. And structure isn’t the only variable. Temperature settings, model selection, and context length all affect output character. (I explore temperature’s effects in a separate lesson.) Testing helps you understand how these variables interact for your specific use case.

Testing Prompts vs. Testing Code

When we test software, we’re verifying logical operations. Given input X, does the function return output Y? The relationship is deterministic. Run the test a thousand times, get the same result a thousand times.

Prompt testing operates on different principles. We’re examining rhetorical reliability, not verifying logic. Does the relationship we’ve established between human intention and machine interpretation remain stable across time, context, and varied inputs?

This distinction matters because it changes what we’re looking for. Code either passes or fails. Prompts exist on a spectrum of reliability, and our job is to understand where on that spectrum a given prompt sits for a given purpose.

The writing feedback chatbot didn’t “fail” in any binary sense. It produced plausible feedback every time. The question was whether that feedback remained consistent enough to be pedagogically useful … And whether students could trust that the evaluation criteria were being applied reliably rather than arbitrarily.

That’s a question of judgment, not logic. And answering it requires a method for judgment.

What We’re Judging

When you test a prompt systematically, you’re evaluating three aspects of the human-AI collaboration you’ve created.

Stance stability. Every prompt establishes a rhetorical stance, or a position from which the AI speaks. “You are a writing tutor who provides constructive feedback focused on argument structure and evidence use” isn’t just an instruction. It’s establishing a consistent voice and perspective.

But does that stance actually persist?

With our classroom chatbot, testing revealed that the constructive-tutor stance held firm for the first few exchanges in a conversation, then gradually drifted toward generic encouragement.

This could happen because the model’s training data contains patterns where tutoring interactions soften over time, or because earlier instructions lose influence as the context window fills with conversation.

Whatever the mechanism, the effect was measurable: stance drift under extended use. Testing helped us identify where drift occurred so we could add stabilizing elements—periodic reinforcement of the evaluative criteria, structural markers that maintained the rigorous-feedback pattern.

Interpretive framework reliability. Your prompt doesn’t just tell the AI what to do. It shapes how inputs get processed. When our chatbot prompt said “evaluate based on the technical communication rubric criteria,” we were creating conditions where rubric-related language would influence the output. But those conditions had gaps we didn’t anticipate.

The rubric worked well for standard assignments because the model had seen similar patterns. But when students submitted creative approaches, like an infographic, the statistical patterns broke down. The model couldn’t match rubric language to unfamiliar input formats, so it defaulted to surface-level observations about grammar and formatting. Testing with diverse input types revealed these blind spots. The fix wasn’t clarifying instructions—it was providing examples of the rubric applied to non-standard formats, giving the model patterns to match against.

Collaborative boundaries. Every prompt creates what I think of as a collaborative space, or the zone where human intention and machine capability overlap productively. Testing maps the edges of this space.

For the classroom chatbot, we needed to know: What types of student writing produce useful feedback? Where does the feedback quality drop off? What submission characteristics cause confusion or generic responses? Which edge cases does the prompt handle gracefully, and which break it entirely?

These boundaries aren’t obvious from the prompt text. They emerge only through running varied inputs through the system and observing where reliability holds and where it fractures.

Knowing what to judge is only half the challenge. The Alexandrian grammarians understood this. They didn’t just identify what made a text authentic or well-formed. They also developed procedures for making those judgments systematically: comparing variants, marking uncertainties, documenting reasoning.

Prompt testing requires both dimensions. We need criteria for evaluation—what counts as stable stance, reliable interpretation, appropriate boundaries. And we need procedures for applying those criteria.

The Rhetorical Appeals as Evaluation Criteria

The Alexandrian grammarians faced a problem we might recognize: they had no original to compare against. When scholars assessed a line of Homer, they weren’t checking it against some authoritative master copy—none existed. Homer was oral tradition committed to writing centuries after composition, and every manuscript was a copy of copies, each with its own variants and corruptions.

So how did these scholars develop criteria for judgment? By immersion in the corpus itself. They studied patterns across many manuscripts, inferring what Homeric diction typically looked like, identifying metrical conventions from the poems themselves, developing a sense of stylistic consistency through deep familiarity with the work. They’re standards emerged from the body of texts, then got applied back to evaluate individual passages.

We’re doing something similar with AI outputs. There’s no “ideal response” to compare against—just multiple outputs from which we infer what “good” looks like for a particular purpose. Our criteria emerge from examining what works, identifying patterns that characterize successful responses, and then applying those standards to evaluate new outputs.

But rhetoric offers a framework that accelerates this process: the three appeals. Aristotle identified ethos (credibility), pathos (emotional engagement), and logos (reasoning) as the fundamental dimensions of persuasive communication. These aren’t just persuasion techniques—they’re categories for evaluating whether communication works.

We’ve applied them to speeches, text, digital media … And now AI outputs.

When we adapt them for prompt testing, they become three lenses for examining output quality.

Ethos Testing: Can the Output Be Trusted?

Ethos in classical rhetoric establishes the speaker’s credibility and character. For AI outputs, we’re not assessing whether the model has credibility (it doesn’t, inherently), but whether the outputs are trustworthy enough for the intended purpose.

Trustworthiness breaks down into two components: consistency and accuracy.

Consistency asks whether the same prompt produces comparable outputs across multiple runs. This matters because inconsistent outputs can’t be trusted for systematic use. If a documentation prompt generates comprehensive coverage on one run and superficial summaries on the next, you can’t build a workflow around it.

Testing for consistency is straightforward: run the same prompt with the same input multiple times and compare the outputs. But “same” doesn’t mean identical. The question is whether variation falls within acceptable bounds for your purpose.

Consider a blog title generator. Testing the same article summary five times might produce five different titles—but if all five maintain brand voice, include relevant keywords, and target the right audience, that variation is a feature for brainstorming purposes. The prompt has sufficient ethos for generating options.

Contrast that with a product description prompt. If testing reveals 30% variation in which technical specifications get mentioned, the prompt lacks the consistency required for that task. Product descriptions need completeness, not creativity. The prompt would need explicit checklists and verification steps until testing shows reliable coverage of required elements.

Accuracy asks whether the outputs are factually correct and appropriately grounded. This is particularly critical for prompts that draw on domain knowledge or make claims that could be verified.

Testing for accuracy requires reference points—either human expert review or comparison against known-correct information. For our classroom chatbot, we tested accuracy by having instructors evaluate whether the AI’s feedback aligned with how they would assess the same submissions. Where the AI and instructors diverged significantly, we examined whether the prompt’s criteria were unclear or whether the model was introducing its own evaluation standards.

The ethos question for any prompt is: Can I trust this output enough to use it for its intended purpose? Testing answers that question with evidence rather than hope.

Pathos Testing: Is the Emotional Register Appropriate?

Pathos in classical rhetoric involves emotional appeal—engaging the audience’s feelings appropriately for the context. For AI outputs, we’re testing whether the tone and emotional register remain appropriate across different inputs and contexts.

This matters more than many practitioners realize. Tone inconsistency can undermine otherwise solid content. A customer service prompt that sounds helpful for simple questions but becomes condescending for complex ones will damage relationships regardless of how accurate the information is.

Imagine an automated feedback system for student writing. The prompt might maintain an encouraging tone when reviewing strong work but shift to patronizing reassurance for weaker submissions. Phrases like “You tried your best” and “Don’t worry, writing is hard” appearing only in responses to struggling students would unintentionally signal that the system had already judged them as less capable.

In this scenario, the prompt’s ethos could be fine—consistent, accurate feedback. But its pathos would be off, treating different students with different levels of respect based on submission quality.

Testing pathos requires diverse inputs that trigger different emotional contexts. For a feedback system, this means testing with:

• Strong submissions (does it avoid excessive praise that might seem hollow?)

• Weak submissions (does it maintain respect while identifying problems?)

• Frustrated student language (does it respond with patience rather than matching the frustration?)

• Confused questions (does it clarify without condescension?)

For a customer service prompt, you’d test across complaint types, customer tones, and issue severity. For documentation, you might test whether the prompt maintains appropriate professional distance when explaining both mundane features and exciting new capabilities.

The pathos question is: Does the emotional register remain appropriate across the full range of likely inputs? Testing reveals where tone calibration breaks down.

Logos Testing: Is the Reasoning Sound?

Logos in classical rhetoric involves logical argument, or the structure and validity of reasoning. For AI outputs, we’re testing whether the logical framework established in the prompt actually governs how outputs get generated.

This goes beyond checking factual accuracy (that’s ethos). Logos testing examines whether the prompt’s stated priorities, decision rules, and evaluation criteria actually shape the output—or whether they get overridden by other patterns in the model’s training.

Consider a documentation prompt that claims to prioritize accuracy but consistently chooses simpler explanations over precise ones. The prompt might include both “maintain technical accuracy” and “explain in accessible language.” In practice, accessibility could win out every tim—the AI sacrificing precision for readability without letting the user know.

This wouldn’t be a failure of the model. It would be a logical contradiction in the prompt that testing reveals. “Accurate and accessible” sounds reasonable until you encounter cases where accuracy requires technical precision that isn’t accessible. Without guidance for resolving that tension, the model could default to patterns from its training data, where accessible explanations are more common than technically precise ones.

Testing for logos means deliberately creating inputs that force your prompt’s priorities into conflict:

• If your prompt says “be concise but thorough,” test with topics that can’t be covered both concisely and thoroughly. Which wins?

• If your prompt prioritizes “user benefit” and “technical accuracy,” test with features where the accurate description doesn’t sound beneficial. What happens?

• If your prompt establishes an evaluation hierarchy (“first check X, then Y, then Z”), test with inputs where X and Y suggest different conclusions. Does the hierarchy hold?

The logos question is: When the prompt’s instructions compete, does the output resolve conflicts the way I intend? Testing surfaces hidden contradictions and reveals which instructions actually govern behavior.

Combining the Three Lenses

Most prompt testing requires all three lenses, but their relative weight depends on purpose.

For a research summarization prompt, logos dominates. You need the reasoning structure to govern output reliably. Ethos matters for accuracy, but pathos is less critical since emotional register in research summaries is relatively narrow.

For a customer-facing chatbot, pathos may matter most. Users will forgive minor inconsistencies or occasional reasoning gaps if the tone feels right. They won’t forgive condescension or inappropriate cheerfulness when they’re frustrated.

For a compliance documentation prompt, ethos is paramount. Consistency and accuracy are non-negotiable. Pathos and logos matter, but trustworthiness is the threshold requirement.

When designing your testing approach, identify which appeals are critical for your use case and weight your testing accordingly. A prompt can have strong ethos but weak pathos (consistent and accurate but tonally inappropriate), or strong logos but weak ethos (sound reasoning but inconsistent execution). Testing across all three reveals the full picture.

From Criteria to Procedures

Knowing what to evaluate doesn’t tell you how to evaluate it. The Alexandrian grammarians understood this. They developed not just standards for judgment but systematic procedures for applying those standards: methods for comparing variants, marking uncertainties, and documenting reasoning so others could follow or challenge their conclusions.

These procedures translate surprisingly well to prompt testing. The Alexandrians were solving a version of our problem: multiple variant texts, no definitive original, and the need for judgments that could be taught, reproduced, and defended.

Recension: Multi-Run Comparison

The Alexandrians framed this work as diorthōsis and ekdosis. They collated multiple manuscript “witnesses” to identify variants, marked doubtful lines, and recorded their comparative reasoning in commentaries. Rather than trusting any single copy, they corrected the text and then issued a stabilized edition, choosing readings based on consistent patterns across the evidence.

For prompt testing, this becomes multi-run comparison. Never evaluate a prompt based on a single output. Run the same prompt with the same input multiple times and compare results.

This sounds obvious, but it’s surprisingly rare in practice. Most prompt development follows a pattern: write prompt, test once, adjust if the output looks wrong, test once more, deploy. That’s like Alexandrians looking at one manuscript and declaring it authoritative.

Multi-run comparison reveals what single tests hide. When I tested my four prompt formats, I didn’t just run each once. Each variation ran under controlled conditions, with metrics tracked across runs. The patterns emerged from comparison, not from any single output.

For practical testing, I recommend a minimum of three runs for informal evaluation and five or more for anything you’ll deploy in production. Compare outputs looking for:

Structural consistency. Do the outputs follow the same organization? If your prompt specifies a format, does that format hold across runs, or does it drift?

Coverage variation. Do all runs address the same key points, or do some outputs omit information that others include? For the blog title generator, variation in titles is fine. For product descriptions, variation in which features get mentioned is a problem.

Tonal range. Do all outputs stay within the same emotional register, or do some runs produce noticeably different tones? This is your pathos check.

Priority adherence. When the prompt contains competing instructions, do all runs resolve the conflict the same way? This is your logos check.

The goal isn’t identical outputs. That’s neither possible nor desirable. The goal is understanding the range of variation your prompt produces and determining whether that range falls within acceptable bounds for your purpose.

Athetesis: Marking Uncertainty

When Alexandrian grammarians encountered lines they suspected were spurious or corrupted, they didn’t simply delete them. They marked them with an obelos—a horizontal line indicating doubt. This kept them visible in the text. Future scholars could see the judgment, assess the reasoning, and reach their own conclusions.

This practice, called athetesis, prioritized transparency over tidiness. A marked line told readers: “This is questionable, but I’m preserving it so you can evaluate my judgment.”

For prompt testing, this becomes uncertainty flagging. When you identify problems in AI outputs, mark them explicitly rather than silently fixing them or discarding the output entirely.

This matters for two reasons. First, patterns of uncertainty reveal prompt weaknesses. If you’re consistently flagging the same type of problem, you’ve identified where your prompt needs revision. Silent fixes hide these patterns.

Second, flagged outputs become training data for your own judgment. Over time, a collection of marked outputs teaches you (and your team) what to watch for. The Alexandrians built scholia (or commentary traditions) around their marked texts. You can build similar institutional knowledge around flagged AI outputs.

A simple flagging system might include markers like:

• H for hallucination (unsupported claims or fabricated details)

• T for tone problems (inappropriate emotional register)

• I for incompleteness (missing required elements)

• C for contradiction (conflicts with prompt instructions or internal inconsistency)

• D for drift (departure from established stance or format)

The specific markers matter less than consistent use. Pick a system and apply it across all your testing so patterns become visible.

Scholia: Documenting Your Reasoning

The Alexandrians didn’t just mark problems—they explained their judgments. Marginal notes called scholia documented why a line was suspect, what alternatives existed, and how the editor had reasoned through the decision. These annotations accumulated over generations, creating a scholarly conversation around the text.

For prompt testing, this becomes documented evaluation. Don’t just record that an output passed or failed—record why.

This is where most testing falls apart. Teams run prompts, glance at outputs, make a gut judgment, and move on. Nothing gets written down. A month later, no one remembers why certain prompt versions were rejected or what problems the current version was designed to solve.

Documented evaluation doesn’t require elaborate systems. A simple log capturing the following for each test proves valuable:

The input used. What specific content did you feed the prompt? Save it so tests can be reproduced.

The output received. Keep the full output, not just a summary or judgment.

Your assessment. Did it pass or fail on ethos, pathos, logos? What specific problems did you identify? Use your flagging system.

Your reasoning. Why did you judge it this way? What would have made it better? This is the scholia—the part that teaches future evaluators (including future you) how to think about the prompt’s performance.

When you revise a prompt based on testing, document what you changed and why. Link the revision to the specific test failures that motivated it. This creates a trail that makes prompt development cumulative rather than circular.

Putting Procedures into Practice

These Alexandrian procedures provide the methodological foundation. But you still need practical workflows for implementing them. The approach you choose depends on your technical resources and scale.

Read more

What happens when AI gets “confused”?

I recently discovered a metric called semantic entropy.

Before your eyes glaze over at the word “entropy,” let me explain why its important.

Semantic entropy measures how much an AI’s responses vary in meaning when you ask it the same question multiple times.

High entropy means the model generates different meanings each attempt—it doesn’t have stable knowledge, so it improvises. Low entropy means consistent responses.

This is one reason why AI hallucinates.

For this podcast, I’m going to try to bring this concept down to earth and make it actionable through the eyes of ancient rhetoric.

From an ancient rhetoric perspective, high semantic entropy is when your AI model is walking through a house with no rooms.

Let me explain.

For those reading, this is a transcript of the podcast, which can be listened to above or in your favorite podcast player.

Recent research into semantic entropy

A few months ago, a paper came out in Nature called “Detecting hallucinations in large language models using semantic entropy.”

They had developed and refined a way to measure when AI seems confused, even when it sounds confident.

Here’s how it works.

You ask an AI the same question multiple times. For example, “What are the installation steps?”

And you get back five different answers. Now, those answers might use different words, but do they mean the same thing?

If answer one says “First, power down the system” and answer two says “Begin by turning off power”—that’s low semantic entropy. Different words, same meaning. The AI got the response right.

But if answer one says “power down first” and answer three says “leave power on during installation,” then you’ve got high semantic entropy. The meanings contradict, and the AI is improvising. It probably isn’t building its answer on solid information.

This happens even when the AI seems confident. It’s not hedging with “maybe” or “possibly.” It’s just... making stuff up to fill the gap.

The researchers showed that semantic entropy can predict hallucinations with pretty good accuracy. When entropy is high, you’re about to get unreliable information.

Why this matters

Now, you might be thinking, “Okay, that’s interesting from a computer science perspective. But I’m a writer, a professor, a content developer. What does this have to do with me?”

Everything.

Because while this semantic entropy research is newer, a broader principle has been established across other studies: how you structure source content directly affects AI performance.

Research on RAG systems, or the technology most organizations use for AI-powered search and question-answering, shows that chunking strategy can impact performance as much as or more than the choice of AI model itself.

Think about what causes high entropy. The AI generates variable meanings because it doesn’t have stable grounding in what the source material actually says. In a way, it’s uncertain or guessing.

And what causes that “uncertainty”? The research suggests it’s often the source material. When documents are poorly organized, the AI does what a confused human reader would do. It fills gaps. Makes assumptions. Creates different interpretations.

The semantic entropy metric gives us a way to measure this instability. But the underlying principle isn’t new: structure matters for machine comprehension just like it matters for human comprehension.

 I should add a note here. The AI model isn’t actually getting uncertain and that’s really part of the problem. The knowledge it’s working from is uncertain, but the AI is trained to be confident, and in the end, that’s what causes semantic entropy.

So you need confident information or knowledge behind your model to match the actual training to be confident.

Why Ancient rhetoric is still important

This problem isn’t new.

Ancient rhetoricians figured this out thousands of years ago.

They had to create speeches on the fly. In the Assembly, in the courts, at public ceremonies. No time to prepare.

Just, “here’s your topic, now speak.”

How did they do it? They used something called topoi.

The word literally means “places” or “rooms.” They organized their knowledge like a house with clearly labeled rooms.

• Need to define something? Go to the definition room.

• Need to compare two things? The comparison room.

• Need to trace cause and effect? That room.

Having these stable mental spaces, or patterns, meant they could reliably find what they needed and construct coherent arguments quickly.

In 1984, Carolyn Miller wrote what became one of the most cited paper in rhetorical studies, called “Genre as Social Action.” And she argued that this is how all communication works. We recognize recurrent situations, and we reach for typified patterns of response.

When the situation recurs in recognizable form, we know what to do. We have stable knowledge structures to draw from.

When it doesn’t, we improvise. We hedge. We contradict ourselves across attempts.

Topoi in machine rhetorics

High semantic entropy is the computational version of lacking stable topoi.

When you ask an AI the same question multiple times and get semantically different answers, the model is doing exactly what a rhetor without proper topoi would do. It’s improvising under uncertainty. It lacks the organizational patterns, or the “rooms,” where specific types of knowledge reliably live.

But … you can create those rooms through content structure.

When you write a procedure with clear steps, properly labeled, you’re creating the “procedure room.”

When you write a concept explanation with a definition, characteristics, and examples in consistent order, you’re creating the “concept room.”

When you use consistent terminology throughout, you’re making sure the rooms have clear labels.

This is what structured content does. It is creating stable topoi for machines.

Low semantic entropy means the AI knows which room it’s in and what that room contains. It’s not guessing. It has reliable patterns to draw from.

What does this mean for you?

So what do you do with this?

First, understand that structure isn’t just about making content look organized. Structure is a signal. It’s how you communicate to both humans and machines.

“This is what kind of information this is, and here’s how to use it.”

Second, recognize that the same principles that help human readers help AI systems. Clear headings. Focused chunks. Consistent terminology. Explicit organization.

Third, start thinking of yourself not just as a writer but as an information designer. Your job isn’t just to explain things clearly. It’s to create reliable knowledge structures that work across contexts, including computational ones.

The content professional who understands this is going to be incredibly valuable as AI becomes more central to how information gets used.

Challenge

So here’s my challenge to you: Next time you’re creating content, ask yourself: Am I building a house with clearly labeled rooms? Or am I creating an unmarked space where readers, human or machine, have to guess what goes where?

Because high semantic entropy isn’t just an AI problem. It’s a content problem.

And content problems? Those are solvable.

Are you wondering how we might test for semantic entropy. Well, stay tuned. More on that soon!

Until then, I’m Lance Cummings. Keep reading deeply.

And if you’re testing this stuff in your own work, I want to hear about it. Find me on LinkedIn or drop a comment on the newsletter.

Talk to you next time.

Leave a comment

“Structured prompting is dead.”

The proclamation came through my feeds like so many others.

Yet another casualty of AI’s rapid evolution. As models get smarter, the argument goes, we no longer need to carefully design our instructions.

“Just talk naturally. The AI will figure it out.”

Like so many claims like this, its not been tested publicly much (partly because its pretty difficult when you get into the nitty gritty). While some have researched AI outputs (like this Microsoft study), we still lack a clear understanding of why different structures create such different outcomes.

I’ve been using structured prompts exclusively in my work—for teaching, for content generation, and for the AI writing tools I build with students.

Yes, its because they’ve always delivered consistent results, but also its much easier for humans to work with well-designed instructions, so why not machines.

It works, so why bother testing. Well, this discussion about the death of prompting got me wondering.

What am I missing by not comparing approaches? Am I clinging to unnecessary complexity while everyone else has moved on to conversational simplicity?

So I ran an experiment.

Not to defend structured prompting, but to understand what structure actually does—to processing time, to cost, to output behavior.

The results tell a story about information design that goes beyond prompt engineering.

How we shape information for AI isn’t just about getting better outputs or faster processing. It’s about choosing your rhetorical stance in a human-machine collaboration.

When I say “rhetorical stance,” I mean the relationship and role that gets established between speaker and audience through how something is communicated.

Think of it like this: the same information delivered in a legal contract, a friendly email, and a technical manual creates fundamentally different relationships between writer and reader.

• The legal contract positions the writer as an authority establishing binding terms.

• The friendly email creates a peer-to-peer collaboration.

• The technical manual sets up an instructor-student dynamic.

With AI, prompt format works the same way by signaling to the model what kind of interaction this is.

JSON says “we’re doing technical documentation now.” Natural language says “we’re having a focused discussion.” Unstructured rambling says “let’s explore ideas together.”

The format becomes a cue that shapes not just what the AI produces, but how it understands its role in the conversation.

How I Performed the Test

I took a prompt I’d developed for the WAC Clearinghouse prompt library (revised version coming soon)—a complex instruction set for helping student writers analyze their microessays through rhetorical appeals.

This wasn’t a simple “write me a paragraph” task. It required the AI to perform multi-layered analysis: identify patterns across texts, develop ethos strategies, create emotional connections, and suggest logical organization. The kind of sophisticated content work that professionals do every day.

Using PromptLayer’s testing tools, I ran four variations in a single session to ensure clean comparisons:

1️⃣ The structured prompt with semantic tags. My original version with clear sections like [ROLE], [CONTEXT], [TASK], and specific subsections for different types of analysis.

2️⃣ The structured prompt with tags removed. All the organization remained—the logical flow, the clear sections—but without the explicit XML-style markers.

3️⃣ An unstructured version. I asked Claude to rewrite the same prompt as if a student were explaining the task conversationally. Same requirements, same goals, but delivered as natural language.

4️⃣ A JSON version. After a LinkedIn commenter asked if I meant JSON when I said “structured,” I converted the entire prompt to JSON format—pure data syntax with nested objects and arrays.

Each test tracked three metrics: processing time, cost, and output token count. I also compared outputs to see if there were any rhetorical comparisons to be made.

While I think the results were enlightening, I should be clear that this is an imperfect test. Other factors like how busy a server or API is can influence these statistics. But I think it is good enough to make some observations.

What the Numbers Revealed

The patterns were striking, though not for the reasons I initially thought.

The structured prompt without semantic tags completed in 64 seconds producing 598 output tokens. This provided a baseline that was pretty clean and efficient.

Adding semantic tags increased time to 83 seconds with 711 output tokens. My theory is that the tags acted as “expansion cues,” signaling that each section deserved elaboration.

The JSON format took 97 seconds and generated 812 tokens. But further research shows that it wasn’t struggling to process JSON.

The format triggered what researchers call “technical documentation mode” that pattern matches more technical genres, producing exhaustive and mechanical prose.

The unstructured version was slowest at 120 seconds with over 1,000 output tokens—nearly double the baseline. My theory is that the conversational prompt triggered exploratory generation, treating the task as an invitation to discuss rather than execute.

Initially, I assumed the timing differences reflected processing difficulty—surely JSON must be harder for the model to parse. But the math tells a different story. In language models, input processing happens in parallel (~0.24ms per token), while output generation is sequential and 20-400x slower per token.

Those 214 extra tokens JSON produced account for most of the 33-second timing difference. The model wasn’t struggling to read JSON—it was taking longer because JSON triggered a more verbose response mode.

This actually makes the finding more interesting. It’s not about computational efficiency but about rhetorical stance. Different formats don’t just organize information differently. They cue fundamentally different generation patterns:

• JSON: Triggers technical documentation mode (exhaustive, formal, mechanical)

• Natural structure: Activates focused communication patterns

• Semantic tags: Signals “elaborate on each section” behavior

• Unstructured: Invites exploratory discussion

The character of the output revealed even more than the metrics. The JSON response read like a compliance document—numbered sections, technical language, etc.

After completing my tests, I discovered that Microsoft researchers had recently published similar findings, documenting performance variations up to 40% based solely on prompt format across multiple GPT models and benchmarks.

Their comprehensive study validates how prompt format significantly affects model performance, with no universal optimal format even within the same model family.

That’s to say … Prompt format is rhetorical!

Their research confirms several key points:

• GPT-3.5 models showed dramatic performance swings—in some cases over 200% improvement when switching formats

• Different models prefer different formats (GPT-3.5 favoring JSON, GPT-4 favoring Markdown)

• Larger models like GPT-4 demonstrate greater resilience to format changes, though notably not immunity

So my findings aren’t necessarily quirks of my specific test prompt or isolated anomalies. The pattern is systemic across tasks and models.

However, what the Microsoft research doesn’t address is why these patterns exist. They document the what but not the why.

The progression from natural structure to mechanical syntax reflects a fundamental principle about human-AI communication.

Why Structure Behaves Like Rhetoric

Structure doesn’t just organize information; it establishes the collaborative stance between human and AI.

Optimal performance comes from matching structure to the rhetorical nature of the task. Natural structured writing processes fastest because it aligns with how the model was trained to understand human communication.

When we provide semantic tags, we’re creating what I call “expansion cues.” The model sees [ETHOS STRATEGY] and understands this as a space requiring elaboration. It sees [TASK] and knows to provide comprehensive detail. Tags act like rhetorical zones that encourage certain types of discourse.

JSON creates an entirely different rhetorical space that focues on compliance and formality. This activates what researchers identify as “technical mode,” producing exhaustive, formal outputs. The 2.3x tokenization penalty of JSON (all those brackets, quotes, and repeated keys) compounds the effect, consuming attention budget without adding semantic value.

Each format implies a different relationship between human and machine.

• Natural language says: “Let’s communicate clearly”

• JSON says: “Complete this technical specification”

• Semantic tags say: “Document this thoroughly”

• Unstructured says: “Let’s explore this together”

The irony? The “most machine-readable” format (JSON) produces the least useful output for human consumption, while natural human organization produces the most efficient machine behavior.

This connects directly to my research on chunk size in RAG systems. Just as different chunk sizes serve different types of questions (64 tokens for facts, 1024 for complex reasoning), different prompt structures serve different collaborative needs.

The design isn’t just technical—it’s rhetorical. We’re not optimizing for machines. We’re designing collaborative spaces.

The Scale Mathematics

For a single prompt, these differences might seem academic. An extra 56 seconds here, three-tenths of a cent there. But content operations don’t run on single prompts. They run on thousands, tens of thousands, millions of interactions.

So let’s make this concrete. Say you’re running a content operation that processes 1,000 prompts daily—not unusual for automated content generation, customer service, or educational applications.

Switching from unstructured to naturally structured prompts saves 56 seconds per query. That’s 15.5 hours of processing time per day—nearly two full workdays. Over a month, you’re looking at 465 hours of saved processing time.

The cost compounds too. While individual prompt costs seem negligible, at scale the differences matter. Those additional 400+ tokens per response in unstructured prompts mean 40% more data volume. Your storage costs increase. Your analysis tools work harder. Your editors spend more time trimming verbosity.

But the real cost isn’t measurable in dollars or seconds. It’s in variance and character. Unstructured prompts produce unpredictable outputs—sometimes brilliant, sometimes meandering, always different. JSON produces consistent but mechanical prose—reliable but soulless.

When you’re building content operations, you need to choose not just your efficiency point but your rhetorical stance. Do you want exploratory collaboration, professional documentation, efficient communication, or bureaucratic compliance?

A Framework for Prompt Design Decisions

After running these tests and analyzing the patterns, I’ve developed a decision framework that treats prompt structure as a rhetorical choice rather than a technical optimization.

Start with output shape, not input format. Before writing any prompt, define exactly what you need: A three-paragraph summary? A five-point action plan? A detailed analysis with specific sections? Your output requirements should drive your structural choices, not the other way around.

Next, identify your rhetorical needs. What kind of collaboration do you want with the AI?

• Natural structure (no tags): For efficient, focused communication. When you need speed and clarity without elaborate formatting.

• Semantic tags: For professional documentation requiring consistent sections. When repeatability and scannable output matter more than speed.

• JSON: For data transformation tasks or when you need absolute consistency, accepting the trade-off of mechanical prose and slower processing.

• Unstructured: For exploration, ideation, or when you want the AI to think expansively about possibilities.

Then assess your scale needs. Running ten prompts a day? The performance differences might not matter. Running ten thousand? Every second and cent compounds. An 88% performance difference between best and worst approaches becomes operationally critical at scale.

Consider your variance tolerance. If you need consistent, repeatable outputs—think templates, reports, standardized responses—structure is non-negotiable. The tighter your structure, the lower your variance. But remember: JSON’s consistency comes with a rhetorical cost. You get reliability but lose voice.

For semantic tags specifically, I’ve found they work best when you need scannable, sectioned outputs that others will consume—documentation, lesson plans, reports. They’re rhetorical markers that say “this content has formal zones.” But they also encourage elaboration, adding processing time and output length.

Skip tags when you need conversational brevity—emails, status updates, quick summaries. The model treats unmarked structure as a cue for flowing prose rather than sectioned content. You get the benefits of logical organization without the expansion behavior that tags trigger.

Avoid JSON unless you explicitly need data structure output or are willing to accept bureaucratic prose for absolute consistency. The processing overhead and rhetorical stance rarely justify its use for content generation tasks.

Where Prompt Engineering Is Actually Heading

The death of structured prompting has been greatly exaggerated. What’s actually dying is the simplistic binary of “structured versus unstructured.” The reality, as these tests reveal, is a spectrum of rhetorical choices, each creating different collaborative dynamics with AI.

What we’re witnessing isn’t the evolution of prompt engineering as a technical skill. It’s the emergence of a new literacy—one where writers understand not just how to communicate with humans, but how to design information for collaborative intelligence.

Where content professionals design not just for reading, but for processing.

Where rhetoric extends beyond human audiences to include artificial ones.

The future is less about structured vs. unstructured and more about developing rhetorical awareness, or understanding how different information architectures create different collaborative possibilities with AI systems.

Sometimes you need the precision of semantic tags for repeatable documentation.

Sometimes you need the efficiency of natural structure for rapid processing.

Sometimes you need the exploration of conversational prompting for ideation.

And yes, sometimes you might even need JSON’s mechanical consistency for pure data transformation.

Understanding structure as rhetoric—not just formatting—becomes essential as AI systems become more central to content work.

This isn’t the end of structured prompting. It’s the beginning of rhetorical information design.

What’s your experience with prompt structure? Have you noticed different AI behaviors with different formatting approaches? I’d particularly love to see tests comparing other structural formats—Markdown, YAML, or even programming language syntax. Drop your observations in the comments, or better yet, run your own tests and share the data. Together, we can map the full spectrum of rhetorical possibilities in human-AI collaboration.

Leave a comment

This semester I’ve been frustrated by the inadequacy of institutional tools for doing the kind of writing I want to do in the classroom.

The biggest problem? Tracking student work without grading.

I don’t give grades in my writing courses because they only get in the way of learning to write. When students face a grade, they stop experimenting.

They play it safe and avoid the kind of risk-taking that actually develops writing ability. Experimentation is key to becoming a good writer, and students are less likely to experiment with a grade on the line.

This year I’m taking a “writing gym” approach with my essay course. The concept is simple: treat writing like strength training. You build writing ability through consistent daily practice, not occasional high-stakes performances.

Students:

• track daily word counts

• work in three-person accountability pods, and

• focus on showing up rather than producing perfect drafts.

Complete or not complete? Did you do your practice or not?

But learning management systems are built for grades. They’re designed around point systems, percentage calculations, and assignment submissions. They make tracking daily practice without numerical judgment nearly impossible.

So at the start of the semester, we used a tool called 750 Words, which was perfect for this approach.

This tool encourages writers to write 750 words every day with a simple interface that tracks and rewards regular writing. It’s a great tool … You should check it out.

Then halfway through the term, I discovered they no longer offer a free version. I can’t make students pay for a required tool after the semester has started, and I certainly can’t afford to pay for 25 students myself.

So I thought: Why not build my own tool?

As most of you know, I’ve been experimenting with using instructional chatbots in the classroom. More recently, I’ve been using Claude to develop apps on top of my structured knowledge that can helps students in more focused ways than a chatbot can.

This seemed like the perfect opportunity to go deeper.

I’m currently revising my PromptOps course into something more foundational called Writing with Machines, exploring how structured content and knowledge architecture make AI collaboration actually work.

Building this tracker would let me test those principles in practice.

Last year I played around with vibe coding tools like Replit and Lovable, but found them fairly limited. They cap the number of interactions you can have daily, and deployment remained complicated for someone without deep technical background.

After spending three months developing the writing gym pedagogy through conversations with Claude, I realized that the accumulated knowledge turned out to be more valuable than any deployment feature a specialized platform could offer.

The Strategic Choice

Andrej Karpathy, OpenAI co-founder, coined the term “vibe coding” to describe developers who “fully give in to the vibes” and “forget that the code even exists”—focusing on iterative testing and prompt refinement rather than examining actual implementation.

But what are “vibes” actually? From a rhetorical perspective, vibes represent an intuitive sense of what works—a feel for audience, context, and appropriate response that skilled practitioners develop through experience.

The ancient Greeks called this metis … or cunning intelligence, the ability to navigate complex situations through accumulated practical wisdom rather than formal rules.

I’ve written before about metis in the context of accessibility and neurodivergent use of AI. Many people have developed sophisticated intuitive strategies for working with AI systems that appear casual but reflect deep understanding of communication patterns.

The “vibes” in vibe coding work the same way. They’re not mysterious or magical. They’re accumulated rhetorical knowledge about what prompts work, what contexts matter, how to frame problems effectively.

But it’s important to realize that any kind of vibes or cunning intelligence rely on practical wisdom, which emerges from expert knowledge. So you can support your vibes (or your metis) by grounding those vibes in structured thinking about your domain.

I didn’t stumble into Claude for this project. I chose it deliberately because that’s where my knowledge lives. My best vibes work on top of knowledge I’ve thoughtfully curated.

For three months before attempting to build anything, I’d been developing my “writing gym” pedagogy through conversations with Claude:

• Working out how students track progress,

• Testing why pod-based collaboration creates accountability,

• Refining what “complete/incomplete” assessment means for student motivation, and

• Exploring how fitness metaphors reframe writing as daily practice rather than occasional performance.

Each conversation added to a growing knowledge base that Claude could access and build upon.

When I finally said “help me design an app for this,” Claude was able to generate interfaces that reflected my pedagogical philosophy without much work.

In the main submission interface, the tagline alone—”Build your writing muscles, one rep at a time!”—captures the entire pedagogy without me ever explicitly saying “use fitness metaphors throughout.”

Also notice the reminder at the bottom: “Remember: Authentic practice builds real strength.” I never wrote that copy. Claude generated it based on our conversations about how consistency matters more than perfection and how authentic practice builds capability.

The interface doesn’t save the actual writing, only the word count. This wasn’t a technical limitation, but a pedagogical choice that emerged from our discussions about removing evaluation anxiety. Students can practice freely knowing the text disappears, but their commitment to showing up gets tracked.

The leaderboard structure reflects our conversations about different types of motivation:

• individual tracking

• pod-based collaboration, and

• achievement systems.

Notice the red flag indicator: “3+ days without writing.” Not punitive, not grade-based, just a visible accountability signal that emerged from discussions about how to maintain consistency without shame.

The empty state copy continues the fitness metaphor throughout the entire experience: “No writers in the gym yet! Be the first to start building your writing muscles."

This wasn’t me writing every piece of microcopy. This was Claude understanding the tonal consistency required by the pedagogical framework.

The badge system demonstrates how Claude structured the gamification badges around the core principles we developed.

Strength Badges (💪) focus on cumulative output: Word Builder, Word Warrior, Word Astronaut, Heavy Lifter.

But notice they’re not competitive in a harmful way—they recognize volume while the “Heavy Lifter” badge celebrates daily record holders.

Consistency Badges (🔥) prioritize what matters most in the writing gym: showing up. Week Warrior, Diamond Hands, Iron Writer, Perfect Week.

Now, honestly, I do think these metaphors could use some work. For example, I’m not so sure about warrior metaphors or diamond hands … But that’s easy to change.

These names didn’t come from me listing badge options, but rather from Claude understanding that consistency matters more than perfection in building writing capability.

The categorization itself (Strength, Consistency, Growth) maps directly to our pedagogical conversations about what writers actually need to develop. I never handed Claude a badge taxonomy. The structure emerged from accumulated knowledge about how writing skill develops.

This is what working with pre-structured knowledge looks like. Not prompting from scratch, but building from a foundation of organized thinking.

The Bridge to Implementation

After developing the app concept in Claude, I faced reality: Claude can’t deploy backends. Any app that needs to work for multiple users on multiple devices requires a database, authentication, and other security measures.

I needed to move to a platform with deployment capabilities. But I didn’t have to start from scratch.

I had Claude generate a comprehensive prompt for Lovable that captured our entire design process, using my structured prompt principles.

Below are a few excerpts. Paid subscribers can find the full prompt here.

[CONTEXT] I’m building an educational web application to help students 
develop consistent writing habits in a classroom setting. The concept 
treats writing practice like fitness training—students track their 
“gains” and compete on a leaderboard to stay motivated. Teachers need 
a simple way to monitor participation and total output without reading 
every submission.

...

[GUIDELINES]
Design and Visual Theme: Create a modern fitness-inspired interface 
that feels motivating rather than academic. Use a purple gradient 
background flowing from #667eea to #764ba2. 

Messaging: Use fitness-themed encouragement throughout, such as “Build 
your writing muscles, one rep at a time!” Provide positive feedback 
after submissions. Keep the interface clean and distraction-free so 
students focus on writing rather than navigating complex menus.

...

[CONSTRAINTS]
* Store student information including names, pods, word counts, and 
  submission timestamps in the database
* The leaderboard must update in real-time when any student submits 
  new writing
* When they submit, save only the word count to the database (not the 
  actual text content)

The decision to save word counts but not actual text isn’t because we can’t build that kind of app. It was a pedagogical choice that emerged from our conversations about removing evaluation anxiety.

The fitness-themed encouragement is a result of the metaphorical framework for reframing writing practice embedded in the prompt.

The real-time leaderboard updates reflect discussions about visibility and motivation in collaborative learning environments.

This prompt became the bridge between platforms. It packaged several months of structured thinking into a transferable format. Lovable could deploy what Claude had conceptualized because the knowledge was already organized, structured, and ready to transfer.

The prompt was a pedagogical philosophy translated into implementation requirements through expert knowledge and practical wisdom. Every design decision, every constraint, every interface element traced back to structured knowledge about how students learn to write.

Here’s the economic reality that makes this approach practical: developing that comprehensive prompt in Claude (where I pay for a subscription) allowed me to create a working app within Lovable’s free interaction limits.

Instead of burning through dozens of back-and-forth iterations trying to explain my vision from scratch, I had a single, well-structured prompt that captured everything.

The investment in building structured knowledge in Claude paid for itself immediately. Without that foundation, I would have exceeded Lovable’s free tier trying to communicate context, clarify requirements, and refine the design through iterative prompting.

I learned something new in this process. Paying for the platform where you build and structure knowledge can save significant money when you deploy to constrained platforms.

The upfront investment in organization returns dividends when you transfer that knowledge efficiently.

The Implementation Reality

Let me be honest about what happened next. The backend challenges were real and humbling.

Lovable gave me deployment capabilities, but I still needed to understand database structure, authentication logic, and security policies.

My app has real limitations:

• security gaps I’m still closing,

• password reset mechanisms, and

• database authentication.

For my use case, these limitations are acceptable. We are just sharing word counts and friendly competition … Nothing high-stakes. This probably wouldn’t work for a lot of educational use cases.

But it go me thinking about things “vibe-coders” don’t tell you and why this asymmetry between frontend and backend exists?

Frontend development offers visual, immediate feedback. You see results instantly in browsers, making debugging intuitive. Visual errors are obvious. The declarative nature of HTML/CSS and component-based architectures map well to AI training patterns.

This is great for prototyping ideas fast. That shouldn’t be discounted, because it is an important part of software development. But if you want to deploy that prototype, it takes a backend.

Backend errors can corrupt databases, expose sensitive data, or cause catastrophic production failures. Database management requires optimization techniques that AI struggles to implement contextually. Deployment, infrastructure, real-time synchronization, security, and compliance all require architectural thinking that pattern-matching approaches fundamentally struggle with.

So it takes considerably longer to turn that prototype into a functional app.

That said, my app serves its purpose within real constraints. It’s functional enough for students to track daily writing, participate in pods, and build consistency habits while I continue learning what “good enough” actually requires in the world of vibe coding.

Start with Knowledge

So here is my key takeaway for anyone looking to give vibe-coding a try.

Start where your knowledge already lives.

The months I spent structuring my pedagogical thinking in Claude created a foundation that made everything else possible. When I finally needed to build, the knowledge was ready to go.

This validates something I’ve been exploring in my research on AI content systems: pre-structuring knowledge beats post-processing it.

Whether you’re preparing content for RAG retrieval systems (which I’ve been writing about lately), designing for accessibility (which I explored at the CAKE Conference), or building applications through AI collaboration, the pattern holds.

The strategic organization of information upfront determines how well AI systems can work with it later.

You can’t fix bad structure with better algorithms.

You can’t compensate for unorganized knowledge with clever prompts.

The fitness metaphors, pod structures, assessment philosophy, and motivation principles in my app weren’t generated through prompt engineering—they emerged from months of structured conversation that Claude could access, understand, and transfer.

The “vibe” in vibe coding isn’t casual or magical. It’s accumulated expertise that’s been deliberately structured through systematic thinking and conversation.

It’s the knowledge architecture you build before you ever start prompting for code.

This is metis in action—practical wisdom that appears intuitive but is actually the result of careful organization and accumulated understanding.

My app worked not because I had perfect prompts, but because I had structured knowledge that let me evaluate what Claude generated, understand why certain design decisions mattered, and transfer that understanding across platforms.

I’ve been telling students for years that good writing requires genuine thinking, not just assembling words.

Turns out AI development works exactly the same way. The quality of what you build depends entirely on the knowledge you bring to the conversation … and more importantly, on how you’ve structured that knowledge for retrieval and recombination.

Where Are You Building Your Knowledge?

The future of AI collaboration isn’t in finding the perfect platform or crafting the perfect prompt. It’s in doing your thinking in places where that knowledge can accumulate, be structured, and transfer when needed.

Where are you doing your knowledge work?

Is it accumulating somewhere that AI can access and understand?

Are you structuring it through conversation, documentation, or systematic organization?

That choice might matter more than any tool decision you make.

Because when you finally need to build something, you’ll discover that the knowledge architecture you’ve been creating all along determines what’s possible.

The vibe in vibe coding? It’s not casual intuition or lucky guessing.

It’s metis—the cunning intelligence that comes from curating your domain expertise so thoroughly that working with AI feels intuitive.

It’s rhetorical knowledge that got there first.

What’s your experience been with building knowledge bases for AI collaboration? Where does your expertise live, and can AI systems access it effectively?

Reply and let me know what you’re discovering.

Leave a comment

A note for paid subscribers: You’re getting early access to this prompt before the main post goes live. This is part of what makes your support so valuable—you see behind the scenes of how I’m developing content and experimenting with AI systems.

On that note, I’ve been revising my PromptOps course into something more foundational called Writing with Mac…

Read more

In these presentations, I move beyond the basics of prompting to discuss what comes next in our evolving relationship with AI systems.

This presentation captures where my work has been heading over the past year—from the structured prompting techniques we’ve explored in PromptOps to something much more fundamental: how we architect knowledge itself for human-machine collaboration.

While casual users may not need sophisticated prompt techniques, the principles that made structured prompts effective haven’t disappeared. They’ve evolved.

As AI systems gain the ability to work with uploaded content and knowledge bases, we’re discovering that these same structured principles apply at a larger scale.

The prompt hasn’t disappeared—it’s expanded to encompass entire knowledge systems.

In the presentation, I walk through this progression: from breaking prompts into reusable blocks, to organizing those blocks into libraries, to ultimately structuring the content that AI systems work with.

This mirrors the journey many of us are taking—moving from prompt engineering to what I’m calling context engineering.

I also demonstrate three different approaches to structuring the same information and the results of some simple testing I did to compare outputs.

When we organize content using microcontent, AI systems retrieve and apply that information far more effectively than when we simply dump unstructured notes into a knowledge base.

This research into AI-ready content is shaping the next iteration of my work. I’m revising my Writing with Machines course around this exact progression, and paid subscribers will have free access to audit the updated material as it develops.

Subscribe now

I’m also developing methods for testing whether content is truly AI-ready—practical approaches you can use to evaluate your own knowledge systems. More on that coming soon.

For now, I hope this presentation gives you a broader view of where machine rhetorics is heading.

The future of rhetoric isn’t about choosing between humans and machines. It’s about designing knowledge systems that shape our ways of knowing the world.

Share

I’ve been doing a lot of research on RAG lately, or retrieval-augmented generation … I mean really digging into the research papers. If you’re creating content for AI systems, you need to understand how these systems actually work.

Not just the marketing version. The real mechanics.

And here’s what I’ve discovered: most explanations of RAG are technically correct but can be misleading.

They’ll tell you “RAG retrieves relevant documents from a database and uses them to generate answers.”

True. But that definition hides crucial details that completely change how you should think about creating AI-ready content.

Today, I want to walk you through what RAG really does, step by step, drawing on the foundational research and recent surveys, because understanding these details will shift how you approach content structure.

By the way, I’m revising and re-platforming my course on Writing with Machines. As I work through this redesign, the course will be available for paid subscribers to preview and provide feedback.

If you’re interested in being part of that process, consider subscribing. A separate message will go out to paid subscribers soon.

Subscribe now

The Chunking Revelation

Let me start with something that surprised me. Your RAG system never actually sees your documents.

Here’s what happens.

When you upload a user manual or a knowledge base article to a RAG system, before anyone even asks a question, the system immediately breaks that document into chunks.

For more on chunking, check out my last deep reading.

The 2024 survey by Gao, et al. on RAG for large language models notes that these chunks typically contain 100 to 500 tokens—roughly 75 to 375 words each. Each chunk gets converted into a numerical representation called an embedding and stored in a database as a completely separate, independent item.

From that moment forward, the RAG system only retrieves and works with those individual chunks.

It has zero awareness that chunk 23 and chunk 24 came from the same manual. It doesn’t know they were originally adjacent. It doesn’t know they both came from the installation section.

Think about what this means. Your carefully structured 50-page troubleshooting guide becomes 100 disconnected fragments. The system treats each one like a separate document.

This is why I say the basic definition is misleading. When people say “RAG retrieves relevant documents,” what they really mean is “RAG retrieves relevant chunks.” And chunks are not documents. They’re fragments that the system created, often by just counting words and cutting wherever it hits the limit.

If you write a procedure and it gets split across three chunks, the system might retrieve chunk one without chunk two, giving an incomplete answer.

The beautiful document structure you created? Gone the moment it gets chunked.

The Multi-Stage Pipeline

Now let’s look at how RAG actually retrieves those chunks, because this is where it gets really interesting.

The original RAG paper by Lewis, et al. back in 2020 introduced the foundational architecture with the goal of combining what they call parametric memory, which is the model’s internal knowledge, with non-parametric memory, which is your external database.

But what’s evolved since then is how sophisticated the retrieval process has become.

Most people think of RAG as: search, find, answer. But Gao’s 2024 survey identifies what they call “Advanced RAG” and “Modular RAG” approaches that use multi-stage pipelines.

Stage one is initial retrieval. When you ask a question, the system searches through all those chunks and pulls back a broad set of candidates—typically 20 to 100 chunks.

This stage is fast but imprecise. It’s like casting a wide net. The system uses hybrid search, combining both keyword matching with algorithms like “BM25 and semantic similarity” using “dense vector embeddings”.

In other words, it’s looking for your exact words AND for content that means the same thing using different words.

Stage two is re-ranking. Now the system gets more careful. Research by Nogueira, et al. on multi-stage document ranking showed that cross-encoders, that jointly encode the query and chunks together, can significantly outperform the initial retrieval.

This stage takes those 20 to 100 candidates and examines each one more closely. It looks at your question and each chunk together, as a pair, and scores how well they actually match.

This is computationally expensive but much more accurate. This stage narrows the results down to maybe 2 to 6 chunks—just the best ones.

Stage three is generation. Only those final ranked chunks get passed to the language model to actually generate your answer.

Why this Matters

Now why does this three-stage process matter for content creators?

Because different content optimizations help at different stages.

1. Clear, descriptive titles help stage one’s keyword matching. If your title is vague or generic, the initial retrieval might miss relevant content entirely.

2. Consistent terminology helps stage two’s semantic scoring—if you use three different terms for the same concept, you’re making it harder for the re-ranker to recognize relevance.

3. And focused, single-topic chunks help stage three generation, because the language model can work with content that’s directly relevant rather than having to extract the useful parts from a mixed-topic chunk.

This is why understanding the pipeline matters. You might not be optimizing for one search—you could be optimizing for a three-stage process, and each stage has different needs.

Practical Implications

So what does this mean for how you create content?

First: Think in chunks from the beginning. Don’t create long documents and hope the chunking works out.

Design each section to be a complete, standalone unit—one topic, clearly titled, with all necessary context. Because that’s what the RAG system will work with. If your content naturally maps to good chunks, you’re starting with an advantage.

Second: Optimize for each stage of the pipeline. That means descriptive titles for retrieval, consistent terminology throughout for re-ranking, and focused topics for generation.

A chunk titled “Overview” tells the retrieval system nothing. A chunk titled “Installing the Database on Linux Servers” tells it exactly what’s inside.

Third: Test how your content actually chunks. If you’re serious about AI-ready content, you need to see what happens when it gets chunked.

Does a procedure get split? Does the context get separated from the steps? Understanding this lets you adjust your structure before problems appear.

RAG isn’t magic.

It’s a mechanical process with specific steps, and understanding those steps completely changes how you think about content structure.

You’re not writing for human readers who can flip back and forth through a document. You’re writing for a system that will fragment your content, search through those fragments, and reassemble pieces into answers.

That requires a different approach to organization, and that’s what I’m exploring with structured content and microcontent principles.

What’s Next

So the next time someone tells you they’re optimizing content for RAG, ask them how and for what kind of RAG? Because that detail matters.

I’m currently developing testing protocols to verify some of these AI-ready content methods—actually putting structured content approaches through systematic evaluation with RAG systems.

If you’re working on similar research or want to compare notes, reach out. I’m exploring this stuff out loud because I think we’re all trying to figure out what good content looks like in this new context.

Until next time—keep reading deeply.

Sources

Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., Wang, M., & Wang, H. (2024). Retrieval-augmented generation for large language models: A survey. Transactions of the Association for Computational Linguistics, 12, 1-25. https://arxiv.org/abs/2312.10997

Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W., Rocktäschel, T., Riedel, S., & Kiela, D. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. Advances in Neural Information Processing Systems, 33, 9459-9474. https://arxiv.org/abs/2005.11401

Nogueira, R., Yang, W., Cho, K., & Lin, J. (2019). Multi-stage document ranking with BERT. arXiv preprint. https://arxiv.org/abs/1910.14424

Leave a comment

Last week at the CAKE Conference in Kraków, I found myself surrounded by brilliant minds tackling the intersection of content and AI.

For those of you who don’t know, CAKE is a new conference in Kraków, Poland organized by a vibrant content and writing community that I’ve been working with for years (and a cornerstone to my design thinking study abroad).

It is truly one of the most exciting and welcoming writing communities out there! Come see us next year.

(Kraków is also one of the best cities in Europe!)

While many sessions in this conference reinforced skills I’d seen at technical communication conferences, one theme kept surfacing that I wasn’t exactly expecting: accessibility.

What struck me wasn’t just that accessibility appeared in so many sessions, but how it revealed something fundamental about our relationship with AI that often gets missed.

We’re approaching AI from the wrong angle entirely if our only focus is on efficiency, productivity, or even ethics broadly speaking.

The Accessibility Revelation

Rather than treating accessibility as a specialized concern or compliance requirement, these presentations helped me think of it as a fundamental lens for understanding how content works—for everyone.

Three speakers approached accessibility from different angles: Anna Dulny-Leszczyńska focused on structural principles for assistive technologies, Sara Grądziel explored automation and AI integration, and Wojtek Kutyła challenged us to see accessibility as inherently political and human.

For me, they demonstrated why accessibility thinking should be central to any content strategy, especially one involving AI.

Structure and Semantics

Anna, a product designer at Synergy Codes, opened her presentation with a powerful demonstration using the viral dress photo from a few years where different people saw different colors. She asked the audience whether they saw it as white and gold or blue and black. This illustrates how we all process visual information differently. But this kind of diversity extends far beyond perception to language, ways of thinking, and physical abilities.

She outlined three core principles for accessible content design.

Structure: Make content logical for assistive technologies. She noted that 71% of screen reader users navigate by headings, jumping from heading to heading to decide what content to consume. This requires descriptive, unique headings, proper use of lists, and avoiding text embedded in images.

Understandability: Ensure everyone shares the same comprehension by using terms familiar to your specific audience and following plain language guidelines.

Operability: Make actions predictable and clear. She showed how Figma’s “lock/unlock” button creates confusion because users can’t predict what will happen when they click it.

What’s striking is how these same principles also help machines work with content at a scale … Or at least help us think about ways our content can work with diverse machines, not just humans.

Automation Layers

Sara Grądziel’s session posed an equally compelling question: Can accessibility become as simple as spell check? She framed automation as a paradigm shift for accessibility, where we now need to understand how machines play a role in developing accessible content.

Sara outlined three layers of accessibility work.

Automation layer: This involves code checks like alt text presence, color contrast ratios, form label verification, HTML validity, and heading structure. Prime targets for AI agents.

AI-Assisted layer: This requires human collaboration to evaluate alt text accuracy and relevance, discover patterns of repeated issues, assess link text clarity, check heading semantics, and evaluate form error handling. This is where you need human-machine collaboration.

Human-only layer: This layer is about checking tone of auto-generated content, testing keyboard and screen reader navigation, assessing content clarity and cognitive load, and experiencing real assistive technology use. Best leave AI out of this, at least with current technologies.

She emphasized that automation isn’t just about checking stuff, but needs to include practical implementation guidance, contextual review of issues, AI-generated content suggestions, and continuous monitoring rather than one-time reports.

Also known as “human-in-the-loop.” So, yeah, accessibily-first thinking will be key to keeping your job in an AI economy.

The Human Dimension

Wojtek Kutyła’s presentation on accessible web content took a strikingly different approach—personal, political, and unapologetically human.

“Language is ours—and texts are political.”

In other words, content choices have real consequences for people’s lives, and writers have the power to harm, exclude, or help through their language choices. I would argue the same when developing AI content systems.

Wojtek emphasized that content must be easy to understand for actual users, not directors or executives. He compared overly formal insurance language (”Acts of God! Insurrection!”) with clearer alternatives (”natural disasters, war, or civil unrest”). He asked content professionals to make content engaging and human, use headings and sections to break up text, test with real people, and remember that just because you think its good, doesn’t mean it is.

He advocated for an intersectional perspective, acknowledging that we all exist in a matrix of privilege and vulnerability, and our writing choices reflect that. For me, this is also true of AI.

Why This Matters for AI

What struck me most about these three presentations was a realization they didn’t all explicitly state but collectively demonstrated: the same structural principles that help screen reader users also create exactly the kind of systematic, well-organized information that AI systems process most effectively.

When you design content with proper heading structure, logical hierarchy, and descriptive labels for human assistive technologies, you’re simultaneously optimizing for AI retrieval systems.

The semantic richness that makes content accessible to humans makes it processable by machines. And Wojtek’s emphasis on human, political, context-aware writing reminds us that accessibility thinking goes beyond technical compliance to fundamental questions about how we communicate with and respect people.

Technical writers aren’t just writers, we are custodians of information experiences. When you approach content design with accessibility as a guiding principle, you create better information experiences across the board.

This insight connects to something I’ve been exploring in my work on rhetorical theory—the ancient Greek concept of metis, or cunning intelligence. Metis represents the ability to adapt, improvise, and find clever solutions to complex problems. It’s not just theoretical knowledge or technical skill, but practical wisdom applied creatively to overcome constraints.

Accessibility-first thinking is metis in action.

Cyborgs Writing

How to Channel the Goddess of Wisdom in Your AI Collaborations

Well, I’m back from a brief pause … after getting attacked by colds from two directions. My twin 5 year olds, and the college students in my classroom…

Read more

2 years ago · 7 likes · 6 comments · Lance Cummings

Accessibility as Strategic Intelligence

When content creators approach accessibility strategically rather than as mere compliance, they develop a form of cunning intelligence that naturally improves all aspects of content design.

Consider how accessibility principles mirror the foundational elements of effective AI collaboration:

Clear structure helps everyone. When you organize content with proper headings, consistent navigation, and logical information hierarchy for screen readers, you’re also creating the kind of well-structured information that AI systems can process most effectively.

Plain language serves multiple audiences. Writing that works for users with cognitive differences also works better for non-native speakers, tired readers, and AI systems trying to extract meaning from your content.

Alternative formats expand reach. Creating content in multiple modalities—text, audio, visual—doesn’t just serve users with different abilities; it provides AI systems with richer, more contextual information to work with.

What the CAKE speakers demonstrated repeatedly was that accessibility-first thinking creates content that performs better across every metric that matters: user engagement, search visibility, translation quality, and yes, AI effectiveness.

Now, some might point out a technical limitation here: LLMs process text by breaking it down into tokens and calculating positional relationships between them. In this process, semantic structures like heading tags or ARIA labels get tokenized just like regular text. The model doesn’t inherently “understand” that an <h1> tag signals something different from a <p> tag the way a screen reader does.

This is technically accurate—but it misses the bigger picture about how accessibility thinking shapes AI content systems.

Accessibility-first design influences AI effectiveness at multiple levels beyond just text generation. When you structure content with clear information hierarchy, you improve how retrieval systems find and rank information before any LLM even starts generating text.

When you write in plain language and chunk information logically, you provide better context windows for models to work with.

When you create multiple modalities and build in redundancy, you enable more robust content systems that can adapt to different uses and users.

The real value is in how accessibility principles shape content architecture, information workflows, and system design, all of which influence how well AI tools retrieve, process, and work with information at every stage, not just during generation.

Many AI-assisted content systems rely on structured data, metadata, and semantic layers for retrieval, routing, and orchestration. These elements remain crucial even if they get tokenized away during the generation step itself.

The goal isn’t to make LLMs magically “understand” accessibility features. It’s to recognize that the strategic thinking that produces accessible content also produces the kind of systematic, well-organized information architecture that makes AI systems more reliable and effective.

The Metis Advantage in Content Strategy

This approach to accessibility represents a sophisticated form of metis because it requires seeing beyond immediate constraints to identify opportunities for systemic improvement.

Rather than viewing accessibility requirements as limitations or constraints, this way of thinking recognizes them as design principles that enhance the entire content ecosystem.

A technical writer applying this approach might create documentation that works seamlessly for users with visual impairments, non-native speakers, and AI summarization tools simultaneously, not through separate efforts, but through integrated design decisions that serve all these needs at once.

A content strategist with this mindset designs information architectures that support assistive technologies while also providing the semantic richness that makes AI tools more effective and reliable.

Most discussions about AI in content creation focus on efficiency, automation, and keeping up with technological change. But the accessibility perspective suggests we’re asking the wrong questions entirely.

Instead of “How can we use AI to create more content faster?” perhaps we should ask “How can we create content systems that serve the widest possible range of human needs while also enabling AI to enhance rather than replace human insight?”

This reframe reveals accessibility not as a specialized concern but as a fundamental lens for understanding content quality in an AI-enabled world. When you optimize for different ways of thinking, different abilities, and different technologies, you naturally create the conditions where human-AI collaboration flourishes most effectively.

Subscribe now

The Path Forward

The CAKE Conference reinforced my conviction that accessibility thinking should be central to any serious content strategy, especially one that involves AI tools. This isn’t just about moral imperative (though that matters too)—it’s about strategic advantage.

Organizations that embed this kind of strategic thinking into their content operations will build more resilient, more effective, and more genuinely innovative content systems. They’ll create content that serves diverse human needs while also providing AI systems with the structured, contextual information needed for truly helpful automation.

The ancient Greeks understood that metis, or cunning intelligence, was essential for navigating complex, changing conditions. As we navigate the rapidly changing world of AI-enabled content creation, accessibility thinking offers us exactly this kind of practical wisdom: a way to find clever, adaptive solutions that serve human needs while harnessing technological capabilities.

The most sophisticated AI strategy isn’t about chasing the latest tools or maximizing automation. It’s about developing the cunning intelligence to create content systems that expand access, enhance understanding, and amplify human creativity rather than replacing it.

That’s a form of metis worth cultivating.

What connections do you see between accessibility thinking and your own AI content strategy? I’d love to hear about your experiments with this approach—reply to this email and share your insights.

Leave a comment