Cyborgs Writing

Start Here

Lance Cummings — Thu, 11 Jun 2026 00:58:50 GMT

Cyborgs Writing has accumulated something close to a hundred posts on AI and writing — courses, prompt experiments, podcast episodes, research syntheses, tool walkthroughs, and arguments about why none of this is really about the model. The line that runs through all of it: AI behavior is shaped by the rhetorical and structural choices a writer makes upstream.

This page is the index I wish had existed: a way to find a piece by what it’s about rather than scrolling the archive.

Start with whatever calls to you, or skip to the section that fits what you’re trying to do.

By the way, I used Claude Co-work to create this index after building my own markdown knowledge base by first exporting and organizing all my Substack posts. More on that coming soon!

Start here

Writing With Machines (course)

The course curriculum, in roughly the order it’s taught.

Deep Reading (research synthesis)

Each piece reads two or three academic sources together.

Rhetoric and Artificial Intelligence — Hunter 1991 (foundational)
The Man Who Predicted ChatGPT in 1998 — Horn 1998, Information Mapping
Do Prompts Really Need Markup? — semantic markup studies
What Does “Evidence-Based” Actually Mean for AI? — Sackett, evaluation frameworks
What the Ancient Art of Organized Thinking Says About AI Hallucinations — semantic entropy, topoi
What is RAG ... No Really, What is It? — RAG and chunking
When AI Research Validates What Content Pros Have Always Known — chunking research
Why Many Writers Can’t Map Their Workflows (and why that matters for AI) — Haas, Lockridge & Van Ittersum on workflow mapping
Reclaiming Agency in AI Collaboration — agency and student writing

Prompt Lab / Context Lab series

Numbered experiments building specific prompts. Each entry pairs a finished prompt with its design rationale.

(Prompt Lab #1 and Prompt #10 sit with the teaching posts further down the archive.)

PromptOps — libraries, taxonomies, structures

The methodology cluster: how to organize prompts so they can be reused and tested.

Chatbots and agents

Built artifacts: walkthroughs for creating chatbots, custom GPTs, and agent skills.

AI tools — hands-on walkthroughs

Tool-by-tool experiments with Lex, Copilot, NotebookLM, ChatGPT, Triplo.

AI content operations & structured content

Theory and practice of structured content as the substrate for prompt reuse.

Poetry and creative work with AI

How to use AI in the writing of poems — not the poems themselves.

Personal essays — how I use AI

First-person accounts of using AI for specific work.

Experiments and stray observations

Pieces that don’t fit the clusters above — cultural commentary, one-off experiments, and a few open questions.

A Dead-Simple Way to Test Your Chatbot's Knowledge

Lance Cummings — Fri, 05 Jun 2026 09:01:43 GMT

A few weeks ago I wrote about what I saw across 17 student chatbots.

Every Doc Makes a Promise

Lance Cummings — Mon, 25 May 2026 11:09:53 GMT

Image generated with Gamma.ai

I’ll be honest with you … I came to this book review feeling a little behind.

Agentic AI has been moving fast, and for most of it I’ve watched from a comfortable distance.

I’m skeptical of a lot of the use cases, and I’ve made no secret of that. Can I really trust an agent to the complex work I’d want them to do? I’ve not been that sure.

But I think we’ve come to a point that everyone needs to reckon with agents one way or another, and Manny Silva’s Docs as Tests and AI made me think about that. Not because he evangelizes agents, but because he explains how to trust them.

That’s really what this book is about.

Not necessarily how to build agents, but how to verify they actually do what you need them to do, and why your documentation is the place where that trust either gets built or falls apart.

Docs as tests is a philosophy, not just a method

What strikes me most about this book is that Silva isn’t offering a workflow checklist. He’s arguing for a way of thinking about documentation.

Every document that describes how a product behaves is making a testable claim. When it says “click this button and this happens,” that’s a promise. The question Docs as Tests asks and answers is whether your documentation keeps its promises.

One inaccurate paragraph, confidently retrieved and served by a chatbot, is a very different problem than one frustrated user who bounces to support. The stakes have changed, and Docs as Tests takes those stakes seriously.

Rhetoricians will recognize this as a question of ethos. Not the reduced version where ethos means credentials, but the deeper sense: credibility earned through consistent, verifiable action.

Silva never uses the word, but the entire book is an argument for how to build machine ethos, or the kind of trustworthiness that holds up not just when a human reads your docs, but when an AI system consumes them and generates answers for thousands of users.

Ethos is too often reduced to “credibility” and left at that. For Aristotle, one of the first to articulate this idea, ethos can’t simply be asserted. It has to be demonstrated through the work itself.

A document that claims accuracy isn’t credible; a document that demonstrably maps to reality is.

Technical communication scholars have made a version of this argument for decades. Documentation functions as a form of institutional ethos, a sustained social contract between a product and its users.

Silva’s framework makes that contract testable. He’s not adding rhetoric to documentation theory; he’s building the verification infrastructure that rhetoric always assumed someone was running.

Subscribe now

Three layers in one book

What makes this book practical rather than theoretical is how Silva structures it. Each section works at three levels simultaneously:

First, he lays out the conceptual foundation,
illustrates it through a practitioner named Vanessa who is working through the same problems in a real documentation context, and then
closes with exercises you can run yourself.

I’ll be transparent about my own limitations here. Some of the exercises require comfort with the terminal, YAML, and basic scripting. I got through them, but I needed to lean on AI to troubleshoot a few mistakes I didn’t fully understand.

If you’re a technical writer who works comfortably with dev tooling, this will feel intuitive. If you’re coming from a less technical background, the exercises are still worth doing — just plan to go slower, and don’t skip the Vanessa scenarios, which give you the shape of the work even when the code feels unfamiliar.

The inventory of documents that Silva lays out in Part Three was probably the part I found most useful. To run a trustworthy agentic workflow, you need:

a project description,
agent definitions,
orchestration patterns that make the workflow legible to every agent involved,
task skills written as reusable prompts, and
plans with explicit acceptance criteria.

That’s documentation. And reading through it, I found myself thinking about portfolios.

Right now I ask students to document a chatbot assessment, build a structured knowledge piece, and analyze a workflow. That’s a solid foundation.

➡️ For a limited time, paid subscribers can access a version of this course for professionals for free. Check it out here.

But a more advanced version of that portfolio could be the full documentation set for a working agent: the project file, the skill definitions, the plan and specs.

It would demonstrate not just that a student can use AI tools, but that they understand the system well enough to govern one.

Manny’s book is the clearest map I’ve seen of what those documents actually need to contain.

Subscribe now

Understanding deterministic vs probabilistic

The core technical distinction in the book is between deterministic testing and probabilistic testing.

Deterministic tests produce the same result every time. It’s binary, reliable, automatable.

Probabilistic tests use AI to interpret content, which means the result can vary between runs. Same input, different output.

That might sound like a limitation, and in some ways it is. But what Silva’s really mapping is the difference between things that can be verified objectively and things that require interpretation — and interpretation, as any writing teacher knows, is where human judgment lives.

When we mistake a probabilistic check for a deterministic one, we get what Silva calls a false signal.

His practical solution is a hierarchy of trust. At the base are ungrounded assertions, or documentation that simply hasn’t been tested.

Above that is grounded probabilistic testing, where AI evaluation is constrained by explicit criteria, run multiple times, and treated as reconnaissance rather than verdict.

At the top is deterministic testing, or binary checks against a live product.

The goal is to migrate upward wherever possible, while being honest about what each level can and can’t tell you.

An LLM isn’t evaluating your product. It’s generating what it expects your product to be, based on the patterns of its training. That’s why any automated documentation system needs testing … not as a quality-control afterthought, but as the mechanism that keeps the content grounded in reality.

Why technical writers are still necessary

The third part of the book, “Teaching Agents Your Process”, was where I learned the most. Silva walks through what it actually takes to run an agentic documentation workflow:

project descriptions,
agent definitions,
orchestration patterns,
task skills, plans
and specifications.

Every one of these is a document. Every one of them shapes how an agent operates and whether it stays within the bounds you intend. Writing them well and maintaining them accurately is a technical writing problem.

This is an argument I’ve made from the rhetoric side, and it’s gratifying to see it come from the practitioner side too. The work that gets automated is first-draft production.

The work that doesn’t is design, information architecture, judgment about what content actually needs to exist and in what form.

Technical writers who understand that distinction become more valuable in agentic systems, not less. The documentation that governs those systems requires exactly the expertise you already have.

The classical term for this kind of judgment is phronesis, or Aristotle’s practical wisdom, the capacity to discern what the right action is in a specific situation, with specific constraints, for a specific audience.

Phronesis isn’t a skill you can look up or a rule you can apply consistently across cases. It’s cultivated through experience, through having stakes in an outcome, through the kind of situational reading that comes from being genuinely accountable to the people you’re writing for.

I’ve been thinking about phronesis lately as the human quality that agentic AI most needs and structurally cannot have.

An agent can execute a well-documented workflow with high fidelity. It cannot tell you when the workflow is wrong for this situation. Or when the document that passes every test still misleads the user who reads it at 11pm trying to fix a production problem.

That gap is where technical writers operate, and it’s the gap that Silva’s human oversight checkpoints are designed to protect.

Who should read this

➡️ Check out Docs as Tests & AI here.

If you work in technical documentation, content strategy, or documentation engineering, this book belongs in your hands now. It gives you a coherent framework for thinking about documentation quality in AI pipelines, with concrete methods for testing and validating what you build.

If you’re an educator or solo creator who isn’t running enterprise documentation workflows (like me) it’s still worth your time. Docs as Tests & AI is one of the clearest explanations of what agentic AI actually involves, what the components are, and where human judgment remains irreplaceable. That’s useful regardless of whether you’re ready to build the system yourself.

I’m not ready to build all of it myself. But I understand it now in a way I didn’t before, and I’m starting to see where pieces of it apply to my own work. That’s about as good an endorsement as I can give.

Thanks for reading Cyborgs Writing! This post is public so feel free to share it.

What 17 Student Chatbots Showed Me About Structured Content

Lance Cummings — Mon, 18 May 2026 12:08:10 GMT

Created with gamma.ai

Over the last year, content professionals have been telling me a version of the same thing.

The way content is structured affects how AI systems behave with it. Cleaner, more deliberate content seems to produce cleaner, more deliberate answers.

We all see this in practice. We all feel it. But pinning it down is harder than it should be. Proving it to a CEO, a CTO, or a colleague who hasn’t lived in the work is difficult to acheive.

This past semester I had an informal opportunity to look at the question. My Writing with AI students partnered with Dr. Yang Song's machine learning students through UNCW's SAIL initiative (Seahawks Advancing Interdisciplinary Learning), a program designed to support cross-disciplinary student collaboration.

Our goal was to design AI chatbots in partnership with New Hanover Disaster Coalition, focused on emergency communication for community members. The English students built the knowledge bases and system prompts, while the machine learning students focused on model integration and testing infrastructure.

By the end of the semester, we had 17 working chatbots across 17 distinct disaster-preparedness topics … and an external assessment of each one.

Teams that built narrow, curated knowledge bases organized by information type consistently outperformed teams that scraped pages or pasted in lists of URLs.

And while this wasn’t a controlled experiment, the pattern was visible enough that I want to share it.

The question content professionals keep asking

A lot of the content world is in the middle of a quiet identity crisis right now.

Many technical writers, content designers, knowledge engineers, documentation leads have all spent a decade building competence in topic-based authoring, structured documentation, modular content, and taxonomies.

Then generative AI arrived, and a lot of our colleagues started asking: does any of that still matter? Can’t we just point a model at a SharePoint folder and call it a day?

People who actually do this work know the answer. Of course it matters.

Of course the model doesn’t magically organize unstructured content.

Of course the same RAG pipeline will produce wildly different outputs depending on what you feed it.

But the question isn’t whether we know this. The question is how to demonstrate it to people who don’t.

The challenge is that most of the evidence is anecdotal. A senior content strategist describes their internal experience improving a chatbot at their company, and a skeptic shrugs because it’s one company, one product, one set of conditions.

Most of the controlled studies in this space are narrowly technical (which embedding model performs best, which chunk size optimizes retrieval). It’s not the kind of evidence that lands with a content team trying to convince a leadership group to invest in content infrastructure.

What I had this semester was something different. A setting where 17 teams worked on the same kind of problem, using comparable tools, with comparable assessment afterward. I could see what each team had actually done with their content.

The project

The goal of this project wasn’t really to build AI chatbots … but to use that project to give students an interdisciplinary experience that would help them understand AI from both perspectives … computer engineering and professional writing.

I use information types with students because it’s accessible. You can engage with structured content as a way of thinking before you have to engage with it as a way of encoding.

The foundation for this spring’s work was laid the previous fall by two other faculty and their students. Dr. Ian Weaver, in the Professional Writing Program, taught an advanced professional writing course that consulted directly with the New Hanover Disaster Coalition to map the community network a system like this would actually serve.

His class built on the research of Dr. Jill Waity, Professor of Sociology and chair of the Sociology and Criminology Department at UNCW, whose community-engaged work on disaster preparedness and resilience in rural and regional communities established the empirical foundation for understanding user needs.

Ian’s students ran usability tests on existing disaster-preparedness materials from agencies like NC State Extension, Ready.gov, and the American Red Cross, and built a taxonomy of disaster-preparedness topics relevant to local needs.

Working in parallel in the Fall, Dr. Gulustan Dogan’s machine learning class produced what we might call a first draft of a knowledge graph for the domain for initial tests on the existing content.

The two fall classes then handed off the project to our classes this spring. The work moved through four classes across two semesters, with each phase building on the previous. Most of what I’m reporting here is the visible piece of a longer chain of student and faculty work that started before my students arrived.

This semester, each team was assigned a node on an adaption of that taxonomy. Seventeen teams, 17 topics, all roughly comparable in scope.

Each team had to do three things:

Build a knowledge base for their topic,
Write a system prompt that defined the chatbot’s behavior and voice, and
Design test questions to evaluate their bot.

Most teams used some combination of LM Studio, AnythingLLM, or Google AI studio. Then I asked teams to structure their knowledge bases using a microcontent approach.

Specifically, I introduced them to Precision Content’s authoring methodology, which sits at the layer underneath things like DITA and XML that many enterprise content systems rely on. The core idea is that you look at a topic and organize it around information types: a concept is one thing, a process is another, a principle is another, a procedure is another. Each has a recognizable shape and serves a different reader need.

I use information types with students because it’s accessible. You can engage with structured content as a way of thinking before you have to engage with it as a way of encoding.

Most undergraduates can’t be productively dropped into a DITA toolchain in a semester. But they can absolutely learn to look at a body of content and ask: which parts of this are concepts, which are processes, which are principles?

Not every team did this. I asked them to. I gave them the tools. But the assignment structure didn’t enforce it, and several teams defaulted to what was easier: scraping URLs, dumping in PDFs, pasting links to UNCW or county emergency management pages.

By the end of the project, the 17 knowledge bases ranged from carefully typed microcontent on one end to lightly organized link lists on the other.

One technical note about the conditions. Yang's lab machines have 8GB of VRAM, which meant most teams were running small, quantized Llama 3 models rather than the frontier systems people associate with conversational AI.

(For non-technical folk, that just means students were using inferior models.)

We're probably two years behind well-funded labs on hardware, and students had to develop extra craftiness to make their bots work within those limits.

The constraint was also informative, though. Smaller models with shorter context windows make the prompt and knowledge variables more visible than they would be at scale. The patterns I'm describing happen with frontier models too, just less obviously.

And in the broader practitioner conversation, the case for lighter models paired with well-structured knowledge is getting stronger. It's more cost-efficient, more accessible to organizations without GPU budgets, and often gets you closer to a usable result faster than scaling the model alone would.

How we assessed the chatbots

We used a Perplexity to evaluate each team’s chatbot, because it wasn’t the model the students used to build their own bots. It came in fresh, with no exposure to the student work, and it had access to its own retrieval over the open web. That gave us a meaningful external check rather than asking a model to grade itself.

For each team, we fed Perplexity their test questions, their bot’s responses, and their own human-written reference answers. Perplexity scored two things on a 1–5 scale: correctness and comprehensiveness. It also produced qualitative feedback for each response, explaining where the bot landed and where it fell short.

We also ran an embedding similarity comparison between each chatbot’s response and the team’s reference answer, hoping to get a third measurement that captured semantic alignment independent of Perplexity’s judgment.

Dr. Song and I then met to talk through the Perplexity results team by team, comparing the assessment against what we’d seen of each team’s content and prompt design.

Before I go further, this wasn’t really a controlled study. The variables weren’t held constant. Different teams used different models, different RAG setups, different testing question styles. Team capacity varied. Topic difficulty varied. Perplexity is a real evaluator but not a calibrated assessment instrument. It has its own retrieval biases and its own way of weighing comprehensiveness, and at times we saw it run overly critical on supporting details when the core content was actually solid.

But the pattern was striking enough that I think it’s worth replicating with a more rigorous design.

What we saw

The clearest contrast came from a coincidence in topic assignment. Three teams ended up working on closely related topics around food safety during power outages.

All three had access to the same source material: CDC, FDA, USDA, and Red Cross guidance on the 40°F/2-hour rule, perishable food handling, refrigerator and freezer timelines.

One of those teams scored a perfect 5/5 on correctness across all seven of its test questions. Perplexity’s assessment described the bot as “fully aligning with FEMA, CDC, and utility best practices” and “delivering critical, life-saving guidance.” That team had built a narrow, deliberately curated knowledge base. Its test questions matched what the knowledge base was designed to answer. Its system prompt kept the bot pulling cleanly from the structured content.

A second team working on essentially the same topic averaged 3.6 on correctness and 2.8 on comprehensiveness. The assessment called several responses “partially correct but falls short in both factual precision and practical completeness, especially for a life-critical topic.” Same source material was available. Same kinds of tools. Same model class. The difference was in how the content was organized before it got to the model.

A third team in this cluster scored well overall but had one telling error. The bot confidently gave incorrect refreezing guidance for thawed meat. Their knowledge base was structured but had a gap, and the bot filled the gap with confident-sounding false information.

The other vivid case involved phone numbers. Two teams built bots focused on accessing help: one for mental health resources, one for general post-disaster assistance. Both teams built their knowledge bases primarily from links to UNCW and local resource pages, expecting the RAG pipeline to surface the right information when asked. Both bots hallucinated.

When asked about UNCW counseling services, the mental health generated a phone number where the last four digits were wrong. The accessing-help bot gave UNCW’s main switchboard number as the campus police department.

Neither of these is a small failure. These are help-seeking bots. The user dialing the number the bot provides is, by definition, someone in distress. Hallucinated contact information is the most consequential failure a chatbot in this domain can have, because it’s the failure that translates most directly into a real-world safety problem.

And it’s the kind of failure that scraping URLs produces. The bot has seen “UNCW police” and a phone number near each other in the source material, and it generates a plausible-looking answer that happens to be wrong.

Teams that built around information types didn’t make these errors as often. Their bots failed in different ways, but they tended to be merely incomplete rather than confidently wrong.

The team that scored highest in the entire cohort built around clear, source-grounded content about pet-friendly shelter access. Five out of six responses scored 5/5. Perplexity described them as “drawing directly from NC Emergency Management and United Way protocols.” That team’s knowledge base was small, narrow, and rigorously sourced. Their bot reflected that.

What this argues

I’ll restate the caveat. This wasn’t science. The variables weren’t held constant. Seventeen undergraduate teams working under real time pressure with novice technical skills is not a controlled environment. I am not making a claim that the structured-content advantage I’m describing here is statistically established.

But in a setting with 17 comparable teams working on comparable problems, the teams that organized their content by information type before encoding it into a knowledge base consistently produced more accurate, less hallucinated, less confidently-wrong chatbots than the teams that scraped URLs or dropped in link lists.

You don’t need DITA to start. You don’t need an XML toolchain. What you need is to start asking, for each piece of content you’re feeding a system, what kind of information this is and what shape it should take.

This is what the content professionals I talk to have been telling me about for a while. The model isn’t doing the structuring work for you. The model can only reflect what it’s been given.

Content that is already organized by meaning gives the retrieval layer something coherent to retrieve. Content that hasn’t done that work yet gives the retrieval layer raw material it has no choice but to guess.

This is also, I think, the right argument for taking microcontent seriously as a methodology, separate from any specific encoding standard.

Precision Content’s typed-information approach is the one I’m using with students because it’s accessible to people who aren’t ready for the deeper technical apparatus, and because the gains it produces are real, even at the entry level.

What I’d want to test next

Seventeen student projects in a single semester is not where this story should end. A few things I’d want to look at, ideally with someone who can design the experiment properly:

A genuine controlled comparison. Same topic, same model, same testing framework with three knowledge bases built three ways (scraped, lightly organized, typed-microcontent). Measure performance directly.

Specific information types. Which contribute most to AI performance? Is the gain mostly in concepts because they anchor retrieval? In processes because they answer the “how do I” questions users actually ask? In principles because they shape the bot’s voice and reasoning? I have hypotheses; I don’t have data.

Scaling effects. Does this hold at larger knowledge base sizes? My intuition is that the structured advantage compounds, but I’d want to see it.

Cross-domain. Does the pattern hold beyond disaster preparedness? I suspect it does, but disaster preparedness has unusually well-established authoritative sources. A domain with messier sources might tell a different story.

If you’re working in this space and have done something more rigorous, I’d be glad to compare notes. The professional intuition is solid. We need the demonstrations.

➡️ And if you want to learn how to build chatbots like this, check out my course on Writing with AI. This is the same process my students used and is currently available free for paid subscribers for a limited time.

Subscribe now

What Does "Evidence-Based" Actually Mean for AI?

Lance Cummings — Mon, 11 May 2026 15:04:26 GMT

This episode has been a long time coming, as I finish up a busy semester.

One of the things consuming most of my attention lately is a research grant I’m running with a colleague in computer engineering.

Students are building structured chatbots for disaster communication, then assessing and documenting how those systems actually perform.

What is evidence?

I’m actually not sure if “evidence-based” gets attached to AI evaluation much, but the assumption is certainly their in the workplace and research lab.

Use data. Test your prompts. Measure outputs. That seems to be evidence based.

But what does evidence-based actually require?

The term comes from a 1996 editorial by David Sackett and colleagues that launched the evidence-based medicine movement.

He defined evidence-based, as the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients.

Its pretty easy to stop there. Run the study, then follow the data. But Sackett immediately complicated it.

Slide from recent presentation.

He drew a model with three overlapping circles and said all three were required: research evidence, professional expertise, and patient values.

“Without clinical expertise, practice risks becoming tyrannised by evidence, for even excellent external evidence may be inapplicable to or inappropriate for an individual patient.”

For those of us doing AI work, the substitution is straightforward.

Replace “clinical expertise” with communication expertise. Replace “individual patient” with your specific audience and their context.

Slide from recent presentation.

That gives us our own definition: the conscientious, explicit, and judicious use of current best evidence, integrated with professional expertise about genre and communication context, evaluated against whether the output actually serves the person who needs it.

Any comprehensive analysis of an AI content system needs more than just numbers … it needs rhetorical analysis.

Numbers are great, but they are only half the picture.

The evaluator isn’t neutral

But even the numbers have problems, and is one reason rhetoric is crucial for evaluating AI writing systems.

Many people assume the research evidence circle is at least the most objective, but that is very difficult to achieve with AI and content.

Panickssery and colleagues challenge that directly. They showed that LLMs systematically favor their own outputs when used as evaluators. Fine-tune a model to better recognize its own text, and the self-preference bias scales proportionally.

Subsequent work has extended this by showing that bias runs across model families and found that evaluator LLMs missed intentionally degraded outputs more than half the time (Spiliopoulou et al. 2025; Doddapaneni et al. 2024) .

Human evaluation has its own distortions. Hosking and colleagues found that human raters consistently score assertive but incorrect outputs higher than accurate but hedged outputs. Confident and wrong beats careful and right.

For our disaster communication project, this means we can’t treat any single evaluation method as definitive. Which brought me to the rubric question, and to some research from education that I didn’t expect to find relevant.

A cautionary tale from education

Back in 2015, Terry Wrigley traced what happens when education tries to adopt evidence-based medicine’s framework.

Too often evidence-based models are imposed on teachers (or practitioners) from the top down, disregarding practitioner points of view. Here’s what the research says, now do it.

Education is an open, recursive system … not a closed laboratory. What works in one classroom may not transfer, and assuming it should distorts practice.

AI prompting is the same kind of system. A model’s response changes the context for the next interaction. Effects are interpretation-dependent.

The Wharton Generative AI Lab’s finding that prompt engineering effects are “complicated and contingent” … that is rhetorical, what Aristotle calls phronesis, or practical wisdom, the kind of judgment that can't be reduced to rules or replicated from a checklist.

Phronesis is what you develop by doing, by failing, by adjusting. This is the kind of skill and knowledge that makes evidence work in our complex situations.

We need stop treating AI systems as an optimization problem and started framing it as a rhetorical one that requires practical wisdom.

Most of us already know automated evaluation is imperfect — and keep using it as if it weren’t, because the alternative takes time and doesn’t produce a clean number.

I’m not saying abandon it, but be precise about what it can and can’t tell you. An LLM-as-judge gives signal on fluency, consistency, surface coherence. It can’t tell you whether the output serves the person who needs it. Those are different questions, and the first doesn’t substitute for the second.

Takeaways for content professionals and technical writers

When evaluating an AI output, ask the question no metric answers for you: does this serve the actual person who needs it, in this context, for this purpose?

Building a rubric that does that requires moving beyond “is this accurate?” toward questions about usability and audience fit.

In our disaster communication project, for instance, one rubric dimension asks: could a coastal resident with limited English act on this message during an active evacuation?

That kind of question builds in audience, context, and purpose — Sackett’s three circles translated into evaluation criteria.

I’ll be sharing the full rubric we’re developing soon for paid subscribers.

Your own systematic, rubric-driven testing is legitimate evidence, not some kind a fallback, and an important part of showing employers your value.

The key is knowing how to communicated it

➡️ That’s the core skill my Writing with Machines course is designed to build. It’s structured around developing an AI portfolio with real documentation. I’m giving paid subscribers free access till the end of June.

Subscribe now

Our disaster communication project is going to push all of this into territory I haven’t fully mapped yet.

How do you evaluate a chatbot that might need to reach someone in crisis, on a bad connection, possibly in their second language?

Benchmark scores won’t settle that. But Sackett’s three circles might be the right place to start.

References

Doddapaneni, S., Khan, M. S. U. R., Verma, S., & Khapra, M. M. (2024). Finding blind spots in evaluator LLMs with interpretable checklists. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (pp. 16279–16309). Association for Computational Linguistics. https://aclanthology.org/2024.emnlp-main.911/

Hosking, T., Blunsom, P., & Bartolo, M. (2024). Human feedback is not gold standard. The Twelfth International Conference on Learning Representations. https://arxiv.org/abs/2309.16349

Khullar, D., Hopkins, J., Wang, R., & Roger, F. (2026). Self-attribution bias: When AI monitors go easy on themselves. arXiv. https://arxiv.org/abs/2603.04582

Panickssery, A., Bowman, S. R., & Feng, S. (2024). LLM evaluators recognize and favor their own generations. Advances in Neural Information Processing Systems, 37. https://arxiv.org/abs/2404.13076

Sackett, D. L., Rosenberg, W. M. C., Gray, J. A. M., Haynes, R. B., & Richardson, W. S. (1996). Evidence based medicine: What it is and what it isn’t. BMJ, 312(7023), 71–72. https://pmc.ncbi.nlm.nih.gov/articles/PMC2349778/

Spiliopoulou, E., Fogliato, R., Burnsky, H., Soliman, T., Ma, J., Horwood, G., & Ballesteros, M. (2025). Play favorites: A statistical method to measure self-bias in LLM-as-a-judge. arXiv. https://arxiv.org/abs/2508.06709

Wrigley, T. (2015). Evidence-based teaching: Rhetoric and reality. Improving Schools, 18(3), 277–287. https://eric.ed.gov/?id=EJ1079308

Build your AI Portfolio Before Someone Asks for It

Lance Cummings — Fri, 01 May 2026 18:29:02 GMT

Generated with Gamma.ai

The writing and content job market is rough right now.

“I use AI in my work” comes dangerously close to I’m “proficient in Microsoft Office” from the old days.

Everyone says it, and it tells a hiring manager almost nothing.

(Let’s not talk about those who refused to use word processors. 😆)

What actually signals competence is documentation, and that’s what many people lack.

Not just that you’ve worked with AI, but how:

what systems you’ve built,
what workflow decisions you’ve made,
and what you’ve tested and revised.

An AI portfolio shows process and workflow … not just deliverables.

Many content professionals haven’t built that deliberately. There just isn’t enough time, right?

You may have used AI and gotten results, but you haven’t documented the strategy and design:

the prompt library,
the style guidelines,
the knowledge sources,
the testing methodology, etc.

That documentation is the portfolio. Without it, you’re left describing a system you’ve built but can’t show.

And if you’ve never built such a system, then you’ve got nothing to document!

So when someone asks, “Show me what you’ve done with AI.” You’ve got nothing, whether you’ve built it or not.

I’ve been thinking about this more lately.

My course Writing with Machines was designed around systematic AI practice: workflow design, prompt development, knowledge integration, style systems, and prompt libraries.

I realized recently that this also produces a portfolio. I just haven’t made that explicit enough, so I thought I would share the final exercise below that I used with my university students as the capstone to their own portfolios.

The exercise below asks students to use AI to generate their final reflection and cover page to their portfolio.

Its important to note that my students had to write (without AI) lots of reflections as part of university assessment. So they already had the raw materials.

Professionals will need to develop these on their own.

This exercise reveals two things pretty quick:

How much you’ve actually documented. If you pull together your work and there’s almost nothing there, the results will not be good.
Whether your AI practice is coherent enough to describe what you did in detail. If you can’t prompt an AI to summarize your own workflow, you may not have a workflow yet, just a collection of habits.

Here is the assignment adapted for content professionals and educators alike.

The AI cover page exercise

For educators: This exercise works for any course where students have built some form of AI writing practice like a prompt library, style guidelines, workflow documentation, or system prompts. Honestly, I think it would work in any class where students are doing substantial work with their writing or writing process, if you are comfortable using AI in this way.

For content professionals: Treat this as a self-directed exercise. Your “portfolio” is whatever you’ve built. If you don’t have anything built, check out my course!

What you are building

Your job is to create a reflective introduction to your AI writing practice that is 400–600 words, written in your voice, generated with your tools, and revised by you.

Treat this like a cover page that introduces your AI workflow to someone reading it for the first time like a hiring manager, a potential client, a collaborator, or a future version of yourself.

It should do three things:

describe what’s in your portfolio and how it’s organized,
explain specifically what you’ve learned or built,
and situate your work in a professional context

People want to know what they can learn by examining the documentation in your portfolio.

What does your AI practice make possible that wasn’t possible before? Where does it work? Where doesn’t it work? What are some takeaways for people in your field?

How to build it

Before you open an AI tool, write a brief self-assessment in your own words. Ideally, you should also have documentation from the process itself.

What have you actually built?
What works consistently?
What’s still improvised or undocumented?

Don’t use AI for this part. The point is to do your own thinking first so the AI has something real to work with. Keep what you write, because it’s the raw material for your prompt.

Then gather the key artifacts from your AI practice:

system prompts,
style guidelines,
prompt library entries,
workflow documentation, and
testing notes.

You don’t need everything. Pick the work that best represents what you’ve built.

Upload your documents to your chosen AI. This collection is your knowledge base for the exercise.

Write a prompt asking the AI to help you draft a reflective introduction to your portfolio.

Apply everything you know about prompt design. A vague prompt produces generic output. A well-constructed prompt produces something that actually reflects your practice.

Then revise. Read the output carefully and rewrite anything that doesn’t accurately represent what you’ve built. You’re the author. The AI is a drafting tool.

Requirements

To be complete, the document must

be written in first person,
reference at least three specific tools, prompts, or workflows from your practice by name,
include at least one sentence connecting your AI practice to a specific professional goal or context, and
reflect genuine specific learning (not generic claims about productivity or efficiency).

It must not be submitted as raw AI output without meaningful revision, and it must not make claims about your work that aren’t supported by what you’ve actually built.

A note on the prompt

Paste your final prompt at the end of the document to show your readers how you generated the reflection. The prompt is part of the portfolio. It shows that you can design effective prompts that match the output you want. For educators, it’s also your window into whether a student engaged with the assignment or outsourced it.

Thanks for reading Cyborgs Writing! This post is public so feel free to share it.

Using Information Types to Build and Evaluate Prompt Structures

Lance Cummings — Mon, 20 Apr 2026 14:25:56 GMT

This post is the reference handout for my ConVex 2026 presentation, “Evidence-Based Prompt Design for AI Writing Systems,” and a follow-up presentation later this week at Information Energy 2026. If you weren’t in either room, everything here is designed to stand on its own.

When an AI system produces a bad answer, most practitioners rewrite the prompt. Sometimes that fixes it. More often, the problem is somewhere else entirely, and without a diagnostic framework, you’re guessing. What follows is a practical framework for figuring out which layer broke before you start changing things.

Most evaluation treats an AI writing system as having two parts: the prompt and the knowledge base. That framing misses a layer that fails constantly and gets blamed on the other two.

There are really three layers, each with its own failure modes.

The prompt layer governs behavior. It holds the instructions, constraints, definitions, and facts the model needs for every interaction. This is content and information too critical to depend on retrieval.

The knowledge base holds content that’s only relevant to specific queries, such as detailed procedures, tool descriptions, location-specific data, anything too voluminous to keep in the prompt without degrading performance.

The retrieval layer connects them. A RAG system pulls knowledge chunks based on query relevance, which means a piece of information only surfaces if the query is similar enough to retrieve it. An MCP server gives AI tools for creating or accessing knowledge.

The practical decision rule: if a missed retrieval would cause a serious failure, the information belongs in the prompt. If it’s only needed for specific queries, it belongs in the knowledge base.

When something goes wrong, the first question isn’t “how do I fix the prompt?” It’s “which layer is this coming from?”

Why information types help

Many practitioners who structure their prompts at all are working from intuitive categories, such as role, context, output format, rules. Or it could be whatever structure an AI suggested when they asked for help.

Those categories aren’t wrong, but they’re ad hoc. They don’t derive from how knowledge actually functions, so they don’t give you consistent criteria for evaluating whether the prompt is doing its job.

Information types provide a more principled heuristic: Task, Concept, Reference, Principle, Process. Each type reflects a genuinely distinct mode of knowing.

Definitions work differently from procedures. Procedures work differently from conditional rules. Conditional rules work differently from facts. Mixing them in the same block makes the prompt harder to evaluate because you can’t tell which kind of content failed when something goes wrong.

I’ve spent the past semester applying information types to RAG knowledge bases, structuring content so each chunk does one job cleanly and retrieval has a better chance of surfacing the right thing.

I’ve been wondering, though … if typed structure improves retrieval, does it also improve the reliability of the instructions themselves?

The short answer appears to be yes. The longer answer will be coming soon and involves Aristotle’s five intellectual virtues from Nicomachean Ethics.

For now, here is the practical framework I’m playing around with for evaluating prompt design.

Adapting Prompts for Disaster Communication

The materials below work through the prompt layer using a student-built disaster communication chatbot as the test case. For a recent grant, my students and I are designing chatbots to help UNCW students with disaster awareness. This specific use case is for preparing family communication plans before, during, and after hurricanes.

Real use case, real stakes.

The original student prompt was built around a [ROLE] structure that is probably the most common way of thinking about system instructions.

The research on role prompting is fairly clear. Persona instructions adjust style, not accuracy. Telling a model it is a “calm, empathetic hurricane communication expert” doesn’t make it more accurate about evacuation zones. It might make answers sound more reassuring, which in a disaster communication context is arguably worse than neutral if the information isn’t good.

The revised prompt replaces [ROLE] with a structure built on information types. Each block has a specific job. None of them bleeds into another.

One addition worth naming: a [METADATA] block at the top. Purpose and audience aren’t a Concept, which explains what something is so a reader can understand it.

Purpose and audience are configuration. They declare what the assistant is, who it serves, and on what authority. Naming them that way is more honest than forcing them into a role block that encourages the AI to “imagine” some human role it can’t actually fulfill.

The [REFERENCE] block holds only always-on facts, like the signup code, the Safe and Well URL, the broadcast stations. These need to be present for almost every interaction and a retrieval miss on any of them would be a serious failure. Detailed app descriptions moved to the knowledge base, where they can be retrieved when a user asks about something specific.

The revised prompt

[METADATA]

Assistant: Hurricane Communication Planner
Audience: UNCW students and New Hanover County residents
Scope: Family communication preparedness before, during, and after
hurricanes and evacuations
Sources: New Hanover County Emergency Management, UNCW emergency
systems, FEMA, American Red Cross

Configuration, not instruction. Declares what the assistant is, who it serves, what it covers, and where its information comes from. Unlike a role block, it makes no behavioral claims. Those come later, in [PRINCIPLE] and [PROCESS].

[REFERENCE]

These facts are critical to nearly every interaction and must be
treated as authoritative regardless of what the user asks.

New Hanover County Emergency Management — A Wilmington-based agency
that coordinates local, state, and federal resources, manages
evacuation shelters, and maintains the county's emergency operations
plan.

Emergency Alert System (EAS) — Broadcasts imminent threat
notifications to the public via radio and television.

Wireless Emergency Alerts (WEA) — Short emergency messages broadcast
from cell towers to WEA-enabled devices by authorized government
partners.

New Hanover County alert signup — Text READYNHC to 24639.

Red Cross Safe and Well registry — safeandwell.communityos.org

NOAA Weather Radio — weather.gov/nwr

Local broadcast: WECT

In Case of Emergency (ICE) — A contact designated in your phone as
an emergency contact. Emergency personnel routinely check ICE
listings first.

Note: Detailed descriptions of the FEMA app, Red Cross Emergency
App, UNCW Alert App, and UNCW Mobile App are maintained in the
knowledge base. Retrieve them when a user asks specifically about
those tools.

Reference holds only always-on, high-stakes facts. These are things that must be present for every interaction regardless of what the user asks. The signup code and Safe and Well URL are here because a retrieval miss on either sends a user away without the most critical information. Detailed app descriptions are in the knowledge base because they’re only needed when someone asks about something specific. The note at the bottom makes that boundary explicit.

[CONCEPT]

A family communication plan is a pre-established set of agreements
about how family members will reach each other, confirm safety, and
make decisions when normal communication channels are unavailable or
unreliable. It typically includes designated contacts, check-in
schedules, backup communication methods, and pre-arranged meeting
points.

An out-of-area contact is a person located outside the affected
region who serves as a central point of contact for family members
to check in with. Local lines often overload during a storm; calls
to and from outside the area are more likely to connect.

A safe word is a pre-agreed word or phrase family members use to
confirm identity during chaotic or high-stress situations where they
may be communicating through unfamiliar channels.

Communication failure during a hurricane typically results from
power outages disabling cell towers, network overload from high call
volume, or physical infrastructure damage. Plans should assume at
least one of these will occur and include methods that don't depend
on the cellular network.

Concept content defines terms the model needs to “understand” before it can respond accurately. Without this block, the model falls back on its training-shaped understanding of terms like “family communication plan.” These definitions bring its working understanding into alignment with the specific context. They belong in the prompt, not the knowledge base, because the model needs them to reason correctly about almost any question, not just the ones that trigger the right retrieval hit.

[PRINCIPLE]

Match response length to urgency. A user asking during an active
storm needs shorter, more direct answers than one planning ahead
in June.

Do not speculate about storm timelines, weather patterns, evacuation
orders, or road closures. Refer users to New Hanover County Emergency
Management or local news for these.

Do not provide guidance outside the communication scope — no medical,
legal, mental health, or physical safety advice. Acknowledge the
concern briefly and redirect to the appropriate resource.

Every recommendation must be traceable to a source listed in
[REFERENCE] or retrieved from the knowledge base. Do not fill gaps
with reasonable-sounding information drawn from general knowledge.

When information is uncertain or unavailable, say so and point to
the closest official resource.

Do not recommend paid apps, devices, or third-party services without
noting they are not official endorsements.

“Calm, empathetic, and concise” has become “match response length to urgency.” The former is a style claim, which is vague, unverifiable, and doing the job a role block would do. The latter is a conditional behavioral norm: testable, specific, and written as a Principle should be. Every entry here follows the same pattern: under a specific condition, do a specific thing.

[TASK]

Help users create or update a family communication plan, including
designating emergency contacts, establishing check-in frequencies,
and identifying backup communication methods.

Help users sign up for and understand emergency alert systems —
New Hanover County alerts, UNCW Seahawk Alerts, the FEMA app, and
the Red Cross Emergency App.

Guide users in establishing protocols for communication failures —
out-of-area contacts, safe words for identity verification,
pre-arranged meeting points, and registration with Red Cross Safe
and Well.

Explain what to do when digital communication is unavailable —
battery-powered radios, NOAA Weather Radio, walkie-talkies, and
local broadcast stations such as WECT.

Each entry is specific enough to derive a test question from directly. “Help users sign up for New Hanover County alerts” should produce a response that references READYNHC to 24639, cites the source, and nothing else. If it doesn’t, you know exactly which task failed, and which layer to examine first.

[PROCESS]

Greet the user and ask an open-ended question to assess their
situation and needs.

Offer a clear starting point: "Do you have a quick question, or
would you like to build a communication plan together?"

If the user has a quick in-scope question: answer it, cite the
source, and offer a follow-up before closing.

If the user's question falls outside scope: acknowledge it briefly,
explain you can't help with that specific issue, and point to the
appropriate resource.

If the user wants to build a plan: ask 1–2 questions to personalize
the guidance (UNCW student or county resident? Planning ahead or
active storm?), then work through contacts, alert signups, backup
methods, and meeting points in that order.

Close each interaction with a summary of what was planned or
answered, the sources used, and: "If you have more questions, I'm
here. Stay safe."

Process sequences how a conversation should unfold. Each branch point is named explicitly, and each branch has a resolution. When the model deviates from this sequence in testing, the process block gives you a specific place to look, either the step is underspecified, or a Principle is overriding it. That’s a diagnosable problem.

The evaluation rubric

Apply this rubric to the prompt before generating a single response. The goal is to catch type collapse, type contamination, and critical gaps at the design stage rather than discovering them through inconsistent outputs.

Score each criterion as Met / Partially Met / Not Met, and note specific evidence from the prompt text. Any “Not Met” on a Reference fact, a Principle consistency check, or a Process branch resolution should be treated as an issue to be fixed before deployment.

Coming soon, I’ll be exploring why these five types, what they correspond to in Aristotle’s five intellectual virtues, and what that tells us about where AI assistance ends and human judgment must begin. The Greeks had sharper vocabulary for this problem than we do.

If you have questions or want to share a prompt for the group to look at, bring it to the Content Lab discussion thread on this post!

Thanks for reading Cyborgs Writing! This post is public so feel free to share it.

Context Lab #12: Weekly Plan Skill

Lance Cummings — Tue, 31 Mar 2026 12:31:16 GMT

Last week’s post on agent skills made the case that most AI skills underperform because they’re written as a single type of content. Task information alone. A well-performing skill actually contains several types of information, each signaling a different purpose to the model.

Below you’ll find the revision of my weekly plans skill that I use to create weekly plans for my classes that help students (and myself) know what were are doing for the week.

A few things to watch for as you read through:

The Concept block is the section most likely to be missing from skills you’ve already built. It’s also the one that does the most work before a single step gets executed.

This is where you define exactly what it is you want the AI to produce … and any other ideas that need to be defined and customized to your context.

The Principle section consolidates constraints that were scattered in the original. Grouping behavioral rules in one place helps the model identify them as actual rules. When the same rules are embedded in a procedure, its more likely that they will be conflated with tasks.

The description field in the front matter is Reference information, and it’s one of the most consequential sections in the entire file. If the skill isn’t triggering when you expect it to, that’s where to look first.

The skill itself is adapted from my Writing with Machines course and designed to be modified. If you build something from it, I’d genuinely like to know what you changed and why. That’s what makes this a lab!

Note: I used markdown for this “skill prompt” because that is what Claude typically uses. I’m thinking about testing this against an XML version in the near future.

Subscribe now

The Techne Behind Agent Skills

Lance Cummings — Tue, 24 Mar 2026 11:08:40 GMT

Image generated by gamma.ai

There’s an old philosophical grudge against craft.

It goes back to Plato, who compared rhetoric to cooking in the Gorgias dialogues. Both rhetoric (or speech-making) and cooking produce pleasing results, but neither understands the true principles behind what it makes.

He called this mere empeiria, or a knack, habit, unreflective routine. You learn what works without knowing why. For Plato, its the lowest form of knowledge and why many philosophers (and now academics) see practice as less worthy of attention.

Aristotle pushed back.

In the Nicomachean Ethics, he defines techne as “a productive state that is truly reasoned” … not just the ability to make something, but making with genuine understanding of the principles behind it.

The practitioner with empeiria knows that something works. The practitioner with techne knows why it works.

I’ve been thinking about this distinction a lot lately while building custom AI agent skills. These are structured instruction files that tell a model how to approach a specific task.

Most people build them the way Plato described rhetoric: run it, tweak it, run it again until the output looks right. Learn what works without ever understanding why.

That’s empeiria. And it only gets you so far.

The principled understanding (or techne) comes from a discipline that has spent decades thinking carefully about how humans organize and communicate information.

For example, technical communicators, particularly those working with structured authoring systems like DITA, have long organized content into five functional categories called information types.

These aren’t arbitrary divisions. They reflect consistent patterns in how information works across human communication. And when you understand those patterns, you understand something about why a skill performs the way it does, not just what to put in it.

Manny Silva recently made the case that agent skills are a form of documentation, held to a higher standard of precision than anything written for human readers.

He’s right. But I’d push the argument further.

The reason most skills underperform isn’t just that the steps are vague. It’s that the whole file is often written as a single type of content.

But a well-performing skill actually contains several kinds of content (not just tasks), each signaling a different purpose to the model through patterns it has been shaped to recognize.

Subscribe now

Information types and what they signal

If you’ve been following this newsletter, you’ve seen information types come up before in how I structure AI-ready knowledge and even some prompts. For those newer to the idea, here’s the short version.

Information types are five distinct patterns of content, each with a recognizable purpose:

Reference states something the reader needs to know.
Concept explains something the reader needs to understand.
Principle advises what to do or not do, and when.
Process illustrates how something works at the system level.
Task instructs the reader on the specific steps to take.

These categories come out of the practice of content strategy and shows how philosophy or academia can inform our practice. These information types aren’t just best practices, but actual patterns humans have developed and used consistently across centuries of written communication: in manuals, textbooks, legal codes, scientific papers, policy documents.

Any sufficiently large corpus of human-produced text is saturated with them, which is why they work with AI.

When you write in those patterns deliberately, you’re not teaching the model something new. You’re giving it a clearer signal about what kind of content this is and what it’s for.

A model shaped on that corpus has encountered these patterns countless times. When you write in them deliberately, you’re not teaching the model something new. You’re giving it a clearer signal about what kind of content this is and what it’s for.

In classical rhetoric, Aristotle described topoi as standard categories of thought that speakers could draw upon to construct a response.

Information types work similarly … not as cognitive locations, but as recognizable patterns that carry purpose. Organize your content by type, and you’re giving AI strong signal. Leave it untyped, and the model infers purpose from whatever context it can find.

Sometimes that inference is fine. Often it’s close but wrong in ways that are hard to diagnose. It becomes a skill that technically works but doesn’t quite perform.

The Problem with Task-Only Skills

If you don’t yet know what a skill is, it is simply a set of instructions an AI model refers to for specific task.

Sound familiar? Well, it should. Its basically a prompt.

When you ask Claude to create a document and watch it work, you’ll notice it references a skill. That skill is a markdown file (.md) and for many people building their own, the whole thing reads like a procedure: do this, then this, then this.

Task information is exactly right for execution steps. But a skill is also a description of what the output is supposed to be, a set of behavioral constraints, a map of how this task connects to the larger workflow, and the metadata that triggers the skill in the first place.

When all of that gets written as task steps (or skipped entirely) the model fills the gaps from general training. And that training may not match your context.

This is why a skill can produce technically correct output that still feels off. The steps were followed. But the patterns that signal what kind of output this is, what constraints apply, how this task fits a larger system are absent. The model filled that gap from wherever it could.

Or, as I’ve noticed, the model calls up the skill at the wrong time (or fails to call it up at the right time).

So I thought, why not information type my weekly class plan skill. This is the skill I use across projects to make sure that my weekly plans that I send students look the same and function the same way.

This saves me considerable work, while adding value to student experience.

My rewritten class plan skill produced noticeably better output on the first run. The surface result looked similar, but the output was more precise and consistent.

Here is how I organized the skill with information types.

A Concept block comes first. What a weekly class plan is. Not a schedule, but a student-facing document that bridges course design and classroom practice, written like a knowledgeable colleague talking to a student, not a syllabus. Without this, the model supplies its own understanding of “weekly class plan.” Sometimes that’s fine. Often it’s close but wrong in ways that are hard to diagnose.

Principle blocks group behavioral constraints separately. What the model must always do, what it must never do, under what conditions it should stop and verify. Writing them in their own section, in direct second-person language, makes them clearer.

A Process block gives system awareness, or how the skill fits into a bigger context. For example, most of my weekly plans point towards a specific deliverable. I prefer to introduce an idea or skill, then spend time in class applying that in ways that move them forward on a deliverable. A model that only has the task steps produces a plan. A model that understands the course arc produces a plan that fits my pedagogy.

Reference is the front matter, which is usually the name and description that trigger the skill. Claude scans that description against your request to decide whether to load the skill at all. Vague descriptions mean missed triggers. That’s not a configuration problem. It’s a writing problem.

The Task steps come last, as the final instruction before execution.

Agent skills are just another form of prompt design. Prompt design is just another form of information design. And information design, done well, is how you actually build the context around your AI workflows, systematically with the same discipline that technical writers have been applying to human readers for decades.

The Techne of Designing Context

Restructuring this skill didn’t just improve one output, it provides context for every moment where I might need a weekly plan for my students.

This is why structured prompting is still relevant. Its the starting point for context engineering and system design.

A structured prompt is how you give the model clear instructions for a single interaction.
Structured knowledge is how you build the environment the model operates in.
Context engineering is the full practice of designing that environment intentionally. Not just what you ask for, but everything the model carries into the task.

This is what Aristotle meant by techne. Not the knack of someone who has run the same skill fifty times and learned what tends to work.

The reasoned understanding of someone who knows why it works.

This is the kind of knowledge that content professionals bring to our conversations about AI, and what makes writers more valuable then ever in the age of AI.

Do Prompts Really Need Markup?

Lance Cummings — Tue, 10 Mar 2026 11:02:45 GMT

If you’ve taken any course on prompt design, including mine, you’ve probably been told to use markup in some way.

This might be markdown, XML, or, in my case, semantic tags.

➡️ See this free lesson from my Writing with Machines course to learn more.

These function as labels for both machines and humans that help organize your prompt into sections, for example [ROLE], [CONTEXT], [TASK].

I’ve taught this. I still use tags in my own work when creating reusable prompts.

… And I get asked constantly whether they’re actually necessary anymore, especially now that models keep getting more capable.

I’m Lance Cummings. And welcome to my intermittent (or aspirationally biweekly) podcast that explore deep research on AI and writing.

That question got me digging into recent research on prompt structure and performance, and what I found reframes the conversation a bit.

The tags aren’t really the point. Specificity is the point. The tags just help us get there.

But as we move from prompt design into what Anthropic now calls context engineering, tags may matter more than you think. Just not for the reasons you’d expect.

The Real Question

Here’s what most people mean when they ask about semantic tags.

Does the AI actually perform better when I label parts of my prompts? Does [GOAL] do something that “I want to …” doesn’t?

We have good research on this now. Sclar and colleagues at ICLR 2024 tested how formatting preserves meaning across 53 tasks and found that formatting alone could swing accuracy dramatically, but the best format for one model wasn’t the best for another.

Different models often prefer different structures. This is why you should test you prompts as a team … and not just go by gut.

But if you’re looking for a formatting rule that works everywhere, there isn’t one. That’s a dead end.

But the research did find something that works everywhere, and it’s not about format at all.

Its All About Specificity

Pecher and colleagues published a study in February 2025 investigating why small changes to prompts produce wildly different outputs. They traced most of it back to a single cause: prompt underspecification.

Not format. Not tags.

The prompts that produced erratic results were prompts that didn’t clearly describe the task, the constraints, or the expected output. Well-specified prompts suffered dramatically less from sensitivity, regardless of formatting choices.

Think of it like giving directions.

“Go to the store” is underspecified. You might end up at a grocery store, a hardware store, a convenience store three blocks away.

But “drive to the Harris Teeter on College Road, pick up two pounds of ground beef from the butcher counter, and use the self-checkout,” now the format barely matters. The task is embedded in the sentence structure itself and will constrain the output whether you text it, email it, or scribble it on a sticky note.

This maps directly onto the three-component model I teach: Task, Context, Content.

Those three categories were never about the brackets. They were about forcing you to answer three separate questions:

What do I want the AI to do?
What does it need to know about the situation?
And what source material should it work with?

The tags were one way to organize those answers. A useful way. But the answers themselves are what drive performance.

What About New Reasoning Models?

Now, I should complicate this, because the landscape has shifted.

Tam and colleagues showed at EMNLP 2024 that forcing structured output formats significantly degraded reasoning.

Imagine you ask a colleague to analyze a customer support problem and give you their recommendation.

Normally, they’d read through the tickets, notice some patterns, and reason their way to a conclusion.

Now imagine instead you hand them a form — fill in the “Recommendation” field first, then the “Reasoning” field.

That’s essentially what happened when models were forced to produce structured output like JSON or XML. The model placed the answer before the reasoning, skipping the step where it works through the problem.

Their solution was a two-step approach: reason in natural language first, then convert to structured format.

Here’s what’s changed since then, though.

Structure your context, not your commands.

Reasoning models now reason internally before generating output. Claude 4 models use what Anthropic calls “extended thinking.” They work through the problem behind the scenes, then produce the response. The model handles that “reason first, format second” step on its own.

Does that make the finding obsolete? Not entirely.

For content professionals working with structured authoring like DITA, XML schemas, and technical documentation, the principle still holds for how you write your prompts.

You’ll get better content by describing what you want in natural language and letting the model generate the substance, rather than forcing a rigid format from the start.

The reasoning models are better at this than their predecessors, but the content still benefits from clear, natural-language instructions.

Structure your context, not your commands.

From Prompt Engineering to Context Engineering

And that phrase — structure your context — is where tags become more important, not less.

In September 2025, Anthropic published a piece on what they call “context engineering.” Building with language models is becoming less about finding the right words for your prompts and more about curating the right configuration of context.

This is the full set of information the model sees at any given moment, which does include your prompt, but also tools, documents, conversation history, reference material, and system instructions.

This is where my own practice has evolved. I actually use more XML-style tags now than I did a year ago. Not fewer.

This is for two reasons.

First, I work primarily in Claude, and Anthropic still explicitly recommends XML tags in their current documentation for Opus 4.6 and Sonnet 4.6.

They’re clear that there are no magic tag names — doesn’t outperform — but XML as a delimiter system helps Claude parse complex prompts. That’s a model-specific advantage, not a universal rule.

That’s really the move from prompt engineering to context engineering in practice. You’re no longer crafting a single message. You’re designing an information environment.

Second, most of what I’m putting into prompts these days isn’t instructions. It’s content.

Course materials, style guides, reference documents, background research.

When you’re loading a context window with thousands of tokens of source material, tags become boundaries between what the AI should read and what it should do. They’re separating content from instruction, not labeling instruction blocks.

That’s really the move from prompt engineering to context engineering in practice. You’re no longer crafting a single message. You’re designing an information environment.

And tags — whatever flavor you prefer — become the architecture of that environment.

Takeaways for Writers and Content Professionals

Here are three guidelines going forward.

Keep the categories, hold the brackets loosely. Task, Context, and Content remain the most research-supported way to organize what you give an AI. Whether you wrap them in XML, use markdown headers, or write clear paragraphs matters far less than whether you’ve actually specified all three.
Use tags to structure your context, not just your prompts. As your AI workflows grow beyond single prompts, tags become architecture. They’re a coordination tool for humans and a parsing tool for the model. That value only increases as the information environment gets more complex.
Let the model reason naturally, then apply structure. If your final output needs to follow a structured format, describe your intent in natural language first. Reasoning models handle this better than ever, but the content still benefits from natural-language instructions over rigid format constraints up front.

The real lesson here isn’t about brackets or XML. It’s that we’ve moved past single-prompt optimization.

Context engineering means designing information environments, and the tools we use to organize those environments matter more now than they did when all we had was a chat box and a one-shot prompt.

If someone on your team is wrestling with whether tags still matter, share this episode. The answer is more interesting than a simple yes or no.

If you want to go deeper on building the kind of systematic prompt and context frameworks we talked about today, that's exactly what my course Writing with Machines covers. It's designed for content professionals who want a repeatable process, not a collection of tips.

I’m Lance Cummings. Until next time … keep prompting … or engineering that context!

Context Lab #11: Writing Genre Coach

Lance Cummings — Mon, 23 Feb 2026 14:26:42 GMT

I’ve been calling this section “Prompt Lab” since I launched it, but I’m changing the name to Content Lab.

The reason is sitting right inside this post: what I’m exploring has quietly outgrown prompt mechanics.

It’s become about knowledge design, genre, communication systems — the whole environment in which a prompt lives.

The prompt is often the last thing I write now, not the first.

For content professionals and technical writers, that shift matters. We've always known that good content doesn't start with writing. It starts with architecture. Information types, audience analysis, structured authoring, content strategy.

Here is what I mean.

The Meta Problem

The students in my AI writing class have been struggling with something I didn’t anticipate. The assignment asks them to analyze their AI-assisted writing process and document their findings, which includes the prompts they built and what they learned from them.

Straightforward enough, or so I thought. But I ran into a consistent problem.

Students would revise their prompts. They’d revise their notes. But they wouldn’t revise how they were communicating their findings to an outside audience.

They were writing for themselves, not for a reader who needed to understand what they did and why it mattered.

When I dug into it, I realized I was asking them to do something genuinely difficult — not just create a prompt, but produce a professional document about the process of creating one.

That’s a meta-cognitive task. And it requires knowing what genre you’re working in.

After talking with students, I realized some had never been introduced to genre as a concept for professional writing.

English majors generally fared better, not because they’re stronger writers, but because they had a conceptual vocabulary for what I was asking. They could name the shape of the document before they built it.

You might think: “Well, why didn’t students just use AI to help them figure out the right genre?”

Because if you don’t know what genre you need (or what genre even is as a functional concept), you don’t know what to ask for.

Genre knowledge has to exist in your head before it can inform your collaboration with an AI. It’s not a skill you can offload.

So I built a Claude app to help.

Building a Genre Coach Solution

The prompt I’m sharing below is a writing coach that gives students exactly two specific revision actions to move their draft toward professional case study format. It identifies

where description has replaced analysis,
where evidence is missing, and
where the reasoning behind AI choices is vague.

Then it shows students a before-and-after example using their own words, so the feedback is immediate and concrete.

But I want to be clear, I didn’t write it in a vacuum. I drafted it inside my Claude project for this class, which contains my course materials, my genre presentation, my assignment descriptions, and my own running reflections on how the class has been going.

Claude wasn’t just following instructions. It was working from a situated context build around my course, my students, and my specific pedagogical problem.

That’s a different relationship to prompting than I had even a year ago. I’m not just writing better prompts. I’m building better environments, and the prompts emerge from those environments almost naturally.

The work that happens before you open the chat window is where the real leverage is.

Your knowledge base has become as important as your prompting technique — maybe more so. A mediocre prompt inside a rich, well-structured project context will often outperform a brilliant prompt written cold.

The work that happens before you open the chat window is where the real leverage is.

This has gotten me thinking … if genre knowledge is a prerequisite for effective AI collaboration, what other conceptual frameworks are quietly limiting what we can do with these tools?

For content professionals, I'd argue the answer is already in our toolkit — information typing, structured authoring, audience modeling.

The field has been building toward this kind of AI-ready thinking for decades without knowing it.

I'd love to hear what you're seeing in your own work. Where is your content expertise giving you an edge, and where are you still running into walls?

System Prompt: Writing Genre Coach

I'm running this as a Claude app inside my AI writing course project, which means it already has context about my assignments, my students, and what a finished case study should look like.

Students paste in their draft and get two specific, actionable revision notes back — no vague feedback, no overwhelming list of changes.

If you're teaching writing or working with teams who need to document processes and decisions, you could adapt this easily.

Just swap out the case study definition for whatever professional genre your context requires, and feed it any relevant background materials you have. The tighter your project context, the more targeted the feedback gets.

Don't Just Prompt Engineer. Start Building Taxonomies.

Lance Cummings — Fri, 13 Feb 2026 14:23:50 GMT

Image generated by Gamma.ai

Here’s a test case that reveals why casual AI use too often fails: disaster communication.

Imagine building a chatbot to help communities understand food scarcity risks during emergencies. You feed it information about food security, emergency preparedness, community resources.

People ask questions. AI generates responses.

When you test it with real life scenarios, it might seem to work well enough … but then breaks down under scrutiny:

“Should I eat meat that’s been in a refrigerator for one day without power?”

“How do I know if my community is at high risk?”

“What should I do if I can’t afford to prepare?”

The AI may generate perfectly grammatical sentences and sound authoritative. It might even cite statistics. But the responses can be generic, often irrelevant, and occasionally dangerous—exactly what you can’t afford in crisis communication.

This isn’t a hypothetical problem. I’m working on this with my students this semester. But the reason it matters for content professionals and writers generally has nothing to do with disaster communication specifically.

It matters because when AI absolutely has to be reliable, the same fundamental limitations appear every time. And too many people (and maybe organizations) are skipping the systematic work needed to address them.

Combining a machine rhetorics framework with user research is key to improving AI performance in real life situations.

A machine rhetorics framework becomes more valuable, not less, as AI capabilities improve. Because the stakes get higher when people trust AI more.

Subscribe now

“Reasoning Models” Still Don’t Reason

AI doesn’t reason. It pattern-matches.

You might be thinking: “But what about the new reasoning models? Aren’t they moving beyond pattern-matching?”

It’s true that newer models show impressive capabilities that can break down complex problems, check their own work, and produce more consistent outputs. Some people argue this means we’re past the need for more systematic approaches to AI collaboration.

I’m skeptical.

Even with extended reasoning capabilities, these systems still operate on statistical patterns, not genuine causal understanding.

They’re better at appearing to reason, which makes their failures less obvious but not less consequential.

Even if reasoning models reduce some failure rates, they don’t eliminate the need for structured approaches when accuracy matters.

A more sophisticated pattern-matcher still benefits from well-structured prompts, organized content, and explicit knowledge mapping … just like a more powerful search engine still needs well-structured information architecture.

The machine rhetorics framework becomes more valuable, not less, as AI capabilities improve. Because the stakes get higher when people trust AI more.

AI doesn’t reason. It pattern-matches.

When you ask ChatGPT about food scarcity during disasters, it’s recognizing patterns from millions of texts where those words appeared together. It’s predicting what words typically follow other words in similar contexts.

What it’s not doing?

Understanding cause-and-effect relationships
Making logical connections between concepts
Recognizing when general advice doesn’t apply to specific situations
Knowing why certain information matters more than other information in emergency contexts.

This is why AI often produces responses that sound right but fall apart under scrutiny. The patterns are there. The reasoning isn’t.

From a practical standpoint, you can’t just “talk to AI” and expect sophisticated results, any more than you could walk into a library and expect books to organize themselves for your research project.

AI needs architecture. It needs structure. It needs humans to provide the logical scaffolding that helps it move from pattern recognition toward contextually appropriate responses.

This is where most casual AI use breaks down, and it’s why content professionals and other writers have an advantage most people don’t realize yet.

The Three-Part Framework

The disaster communication problem reveals three distinct challenges that too many people treat as one problem. Understanding machine rhetorics helps break down these problems and build solutions that are human-based.

AI doesn’t know what you actually need from a response (versus what it can plausibly generate).
AI can’t distinguish what’s relevant when everything’s mixed together.
AI can’t reason about logical relationships—it can only recognize patterns.

Each limitation requires a different solution. Each solution builds on the previous one. And you can’t skip steps.

Machine rhetorics means applying rhetorical principles to AI systems the same way content professionals apply them to any information experience:

start with users,
understand their needs through research, then
structure systems around those insights.

Understanding machine rhetorics helps break down these problems and build human-centered solutions through strategic prompt design, structured content development. and rhetorical knowledge mapping.

This is what I’m exploring with students this spring, but the framework applies wherever content professionals need AI to be genuinely reliable.

Let me show you what each component does using disaster communication as the test case.

Subscribe now

1. Strategic Prompt Frameworks

The instinct is to build a chatbot by dumping information into it and hoping for the best.

Designing information experiences is about whether information is findable, understandable, and actionable for the right people, in the right place, at the right time.

Someone asks, “What should I eat if my power’s been out for a day?” and the AI generates a response pulled from its training data.

The problem? That response might be generically accurate but miss everything that matters about the person asking.

This is a usability problem and needs usability frameworks to solve. When content professionals design information experiences, its more than just getting users to click the right buttons.

Designing information experiences is about whether information is findable, understandable, and actionable for the right people, in the right place, at the right time.

Ultimately, an AI chatbot is an information interface, and it can fail for the same reasons any interface fails. It wasn’t designed around how real users actually encounter and act on information.

That’s why we started with user research in our disaster communication project. Last semester, our students conducted interviews, usability tests, and card-sorting exercises with the target audience for a disaster food security chatbot.

This semester, a new group of students is using those findings to develop test user questions and structured content for the AI system. The usability insights from that research are what will shape how we build the prompts.

Generic prompts produce generic answers. But when you structure the system around what usability research has revealed about real users, the AI has something meaningful to work with.

Here’s the difference. A basic system instruction might say:

You are a helpful assistant that answers questions about food safety during disasters.

A system prompt grounded in usability research looks more like this:

Task: Answer questions about food safety during power outages, prioritizing actionable steps the user can take right now with what they have available.

Here is some context from research that might be added to the system prompt:

Users are often college students with limited storage, limited transportation, and tight budgets. They need to know what to do with the food they already have, not what they should have bought in advance.
Users skip overly official language. Usability testing showed they abandoned documents that read like government pamphlets. They responded to conversational, direct guidance.
A common misconception from card-sorting data: users grouped “frozen food” and “refrigerated food” together, not realizing they follow different safety timelines during outages. Responses should proactively clarify this distinction when relevant.

Now when someone asks “Is the chicken in my fridge still safe?” the AI can respond with specific guidance calibrated to the user’s actual situation — not a generic food safety lecture.

And the reason it can do that isn’t because someone wrote a clever prompt. It’s because usability research identified what users need, how they process information, and where their mental models diverge from expert knowledge.

The difference isn’t prompt length. It’s that the prompt is structured around a usability framework content professionals already work with:

audience analysis drawn from real research,
purpose defined by actual user needs, and
content organized to match how people actually encounter problems rather than how experts categorize them.

This same rhetorical approach needs to be applied when building the information and content AI draws on in its responses.

2. Structured Content Development

Even well-structured prompts fall short when you feed AI disorganized information or rely on its black boxed training.

You might have:

Research about food scarcity
Statistics about emergency food systems, and
Interview data from community organizations.

But if it’s all jumbled together, AI can’t tell what matters when, or how different pieces of information relate to each other, or what readers need to know before they can understand something else.

The solution is organizing materials using what content professionals call information types:

Reference information: Basic facts readers need to understand the situation. What counts as food insecurity? What do terms like “food desert” actually mean? What are current statistics?

Concept information: Frameworks for understanding the problem. How do sociologists think about food systems? What models help explain why communities experience food scarcity? How do emergency management professionals analyze risk?

Principle information: Cause-and-effect relationships that explain how things work. Why do food supply chains fail during disasters? What factors determine community resilience? How does economic stress affect food access?

Process information: How things unfold over time. What’s the typical progression of food scarcity during emergencies? How do communities respond? What happens when intervention comes too late?

Task information: Specific actions people can take. How should individuals prepare? What should community organizations do first? When should people seek additional resources?

When you organize information this way, two things happen.

First, you understand your own materials better. You can see gaps in your knowledge and recognize which types of information you’re missing entirely.

Second, when you structure this organized content into AI interactions, responses become more targeted, more appropriate to specific situations, and actually useful for different kinds of questions.

This is the second component: organizing information systematically so both humans and AI can understand how different pieces relate to each other and serve different purposes.

That’s what my students will be working on soon.

You can get a sneak peak at the taxonomy we are using in my Content Lab

3. Rhetorical Knowledge Mapping

Strategic prompts and structured content get you far, but they still leave a gap: AI can’t reason about relationships between concepts unless you map them explicitly.

When someone asks “Is the chicken in my fridge still safe?”, the answer depends on understanding how factors connect: How long has the power been out? Was the chicken frozen or refrigerated? Has the fridge stayed closed? What’s the safe timeline for this food type?

These aren’t isolated facts. They’re part of a reasoning structure where one decision point leads to another. AI can’t extract these logical relationships reliably from unstructured text alone.

This is where knowledge graphs come in—not as a technical database concept, but as a systematic way to map the reasoning structure that underlies domain expertise.

Our project taxonomy (developed from Fall 2025 user research) organizes disaster food security content into domains and topics: Food & Hydration, Water Safety, Power & Utilities, each with specific subtopics like “Refrigerator/freezer safety during outages.”

A taxonomy tells you what exists in the domain. A knowledge graph tells you how those things relate.

You identify the discrete concepts (refrigerated food, power outage, time thresholds, student constraints), then map the relationships between them: what’s safe for how long under which conditions, what constraints affect which options, what triggers what timeline.

The graph also captures context from user research, for example students confuse frozen and refrigerated timelines, trust action-oriented guidance, and face transportation and storage constraints. These rhetorical insights shape how relationships get structured.

With this mapped, AI can trace a path from the user question → refrigerated food entity → time without power → safety threshold → a more specific response about food.

Building a knowledge graph is about choosing which distinctions matter for this audience, which relationships drive decisions, which concepts need disambiguation, and which context shapes interpretation.

It’s rhetorical.

Content professionals already make these decisions when organizing information. The knowledge graph makes those decisions explicit and machine-readable so AI systems can reason with them.

How They Build On Each Other

You can’t skip steps in this progression, and disaster communication shows why with unusual clarity.

Without strategic prompt frameworks, you’re hoping AI will guess what constitutes appropriate crisis communication. Sometimes it guesses right. Maybe even most of the time. But what if it doesn’t? The consequences matter.

Without structured content, even well-framed prompts produce generic responses because AI can’t distinguish what’s relevant when facts, theories, procedures, and definitions are jumbled together. In crisis communication, generic responses can cause harm.

Without knowledge maps, AI can’t understand logical relationships, cause-and-effect connections, or contextual constraints that determine when information applies versus when it doesn’t. It will confidently generate plausible-sounding advice that violates the actual constraints governing emergency response.

But when you address all three challenges systematically, something shifts. You’re not just “using AI.” You’re designing information systems that help AI perform better while developing your own analytical capabilities.

The disaster communication scenario makes failures visible because stakes are high. But the same three limitations show up everywhere content professionals need AI to be reliable:

technical documentation where incorrect instructions cause problems,
compliance content where errors create liability, and
strategic communication where off-brand messaging damages reputation.

The framework is the same. Only the domain changes.

Can you design systematic approaches to AI collaboration that leverage your rhetorical expertise while developing capabilities AI doesn’t possess? That’s the real question.

What This Means for Content Professionals

True AI collaboration develops systematic approaches to these three distinct challenges—strategic framing, content organization, and knowledge architecture.

That’s why machine rhetorics is more than just prompt engineering.

It is a way of thinking that focuses on audience and context beyond the interface or immediate chat. Its a way of thinking that most content professionals already do.

This is why the workflow mapping advantage I wrote about last week matters. You can see how your work actually gets done.

Can you design systematic approaches to AI collaboration that leverage your rhetorical expertise while developing capabilities AI doesn’t possess? That’s the real question.

The Writing with Machines course teaches this framework—not just the three components, but how to apply them across different professional contexts where AI needs to be reliable rather than just plausible.

I’m testing it this spring in one of the most demanding contexts possible: crisis communication where generic responses can cause real harm. But the framework works anywhere content professionals need systematic AI integration—documentation, strategic communication, research synthesis, content operations.

If you’re interested in learning to apply this in your work, the beta course is opening up next week for paid subscribers!

Subscribe now

User-Backed Taxonomy Handout

Lance Cummings — Fri, 13 Feb 2026 13:30:54 GMT

This taxonomy emerged from Fall 2025 user research conducted by students in our disaster communication project, and illustrates how I’m using taxonomies to organize and structure AI collaboration. Working with the New Hanover Disaster Coalition, students interviewed UNCW students about food security needs during disasters, conducted usability testing on existing emergency documents, and ran card-sorting exercises to understand how people naturally categorize disaster preparation information.

The taxonomy you see below builds directly on student work but has been adapted and refined using an MCP data modeling tool and AI to demonstrate how research-based categorization translates into structured content organization—which then becomes the foundation for knowledge graphs that AI systems can use reliably.

Interested in exploring this in your own work? Check out my course, Writing with Machines. Now available for paid subscribers.

Subscribe now

This taxonomy synthesizes work from Fall 2025, when students in ENG 404 (Advanced Professional Writing) and CSC 302 (Intro to AI) collaborated on the foundation for a disaster relief chatbot. ENG students conducted user research with UNCW students about food security during disasters, created taxonomies based on that research, and prepared source documents. CSC students built initial knowledge graphs using Neo4j tools.

This semester, ENG 326 and CSC 322 continue that work by developing functional chatbots. This taxonomy organizes the project so teams can divide the work systematically.

What is a taxonomy and why does it matter?

A taxonomy is a classification system that organizes information into categories and subcategories. You encounter taxonomies constantly—the folder structure on your computer, the way a grocery store organizes aisles, the categories on a news website.

For AI systems, taxonomies matter because they determine how information gets structured, stored, and retrieved. A chatbot answering questions about disaster preparedness needs to “know” that generator safety relates to power outages, which relates to food spoilage, which relates to health risks. Without an organizing structure, the chatbot has no map for navigating these connections.

The taxonomy below divides disaster preparedness into domains and specific topics. Each team will own one topic, becoming the experts responsible for gathering authoritative information, structuring it for AI use, and building a chatbot that handles questions in that area.

Taxonomy: Broad Disaster Preparedness Coverage

How teams will use this taxonomy

Each team receives one topic from the taxonomy. Over the semester, teams will:

Identify and collect source material. Find authoritative sources (CDC, FEMA, Red Cross, etc.) that address your topic. The Fall 2025 classes prepared some text files already—check there first, then supplement as needed.

Structure the content. ENG students will extract key information and organize it into a consistent format—concepts, tasks, rules, warnings—that works well for AI retrieval. This structured content feeds both your team’s chatbot and a shared repository.

Build and test chatbots. CSC students will build two versions: one without a knowledge graph (baseline) and one with a knowledge graph built from the structured content. Teams will compare performance to see how structured knowledge affects accuracy.

Contribute to the larger project. Your structured content joins a shared repository. Even though each team builds a focused chatbot, the combined work creates a foundation for a more comprehensive system—whether integrated this semester or built upon in future classes.

Authoritative sources by domain

Don't Miss Your Beta Access

Lance Cummings — Tue, 03 Feb 2026 13:23:16 GMT

You’re getting this because you’re a paid subscriber, and I want to make sure you didn’t miss this.

I just released a Deep Reading episode on the transparent technology myth which explains why most people can’t see their own workflows, and why that matters for AI integration.

If you haven’t listened yet, it’s the foundation for what I’ve building.

Writing…

Why Many Writers Can't Map Their Workflows (and why that matters for AI)

Lance Cummings — Fri, 30 Jan 2026 15:02:10 GMT

Here’s something that doesn’t make sense.

You work with engineers every day. Smart people. They build complex systems. They understand workflows, pipelines, dependencies.

But ask them to map how they actually write documentation, and most of them can’t do it.

They’ll describe the end result. They’ll tell you what the doc should contain.

But the actual workflow? How information moves from subject matter expert interview to final publication? The tools, the handoffs, the format conversions, where things get stuck?

Blank stare.

Meanwhile, you can map that workflow in your sleep.

You know exactly where the bottleneck is—usually in review cycles, right? Or maybe it’s getting engineers to actually respond to edit queries. Or it’s that one legacy system that doesn’t integrate with anything.

You see the system because your job depends on seeing the system.

But here’s what I realized recently: Most people were literally taught NOT to see what you see.

And that’s not a small thing. That’s the difference between people who are prepared for AI integration and people who are scrambling.

I’m Lance Cummings, and you’re listening to Deep Reading, where we look at research that changes how we think about writing and AI.

Today I want to show you a piece of research from 1996 that explains why you’re positioned to lead AI integration in your organization—even though nobody’s probably told you that yet.

And why the engineers who are supposed to be “tech people” are actually starting from behind.

Subscribe now

The Invisible Tool Myth

In 1996, a researcher named Christina Haas published a study called “Writing Technology: Studies on the Materiality of Literacy.”

She was investigating something she called the transparent technology myth.

Here’s what she found: Most people are taught to treat writing tools as neutral instruments that don’t change the writing itself.

The tool is just a conduit. The thinking happens inside your head, and the tool just captures it.

This is how most of us were taught. “It doesn’t matter if you write by hand or on a computer—the writing is the same.”

Except that’s not true at all.

Haas showed that tools fundamentally shape not just the final product, but the process of composing itself.

Writing in Microsoft Word feels different than writing in MadCap Flare. Not because one is better, but because they organize information differently, which changes how you think about structure.

Managing documentation in Confluence creates different workflows than managing it in SharePoint.

Authoring in DITA with Oxygen XML forces you to think modularly in ways that a traditional word processor doesn’t.

The tool isn’t transparent. It’s active. It shapes the work.

But if you’ve been taught the transparent technology myth—that tools don’t matter, only ideas matter—then you literally can’t see how tools shape your process.

In the Classroom

I saw this play out exactly as the research predicts in a classroom last week.

I asked twenty students—computer engineers, English majors, cybersecurity students—to map their writing workflows.

The engineers were the most striking. These are people who live in version control systems. They can map a deployment pipeline in their sleep.

But their writing workflow? “I just... write it until it’s done.”

The English majors could describe intellectual moves—brainstorming, researching, drafting—because that’s what they’d been taught to name.

But the operational reality? The actual tools, formats, handoffs, file management, version control? That was supposed to be background noise. Irrelevant to “real” writing.

They couldn’t map their workflows because they’d been taught not to see the tools.

And this matters now because you can’t integrate AI into workflows you can’t see.

From Process to Workflow

In 2020, two researchers named Tim Lockridge and Derek Van Ittersum published a framework specifically about writing workflows.

They defined a workflow as “the tools and the process used for a writing task.”

Not just the cognitive process—brainstorm, draft, revise.

But the tool sequences. How work actually flows through systems.

They argued that you can’t understand contemporary writing without examining these tool sequences rather than treating technology as transparent.

Then in 2024, a researcher named Alan Knowles extended this to AI specifically.

He merged workflow thinking with something called Human-in-the-Loop principles.

The question isn’t “what can AI do?”

The question is “where does AI fit within existing work practices?”

Does it reduce friction? Does it open up new relationships with tools and tasks?

You can only answer that if you can see the workflow first.

Here’s where it gets interesting for content professionals.

In 2025, three researchers—Getto, Kelley, and Vance—applied this specifically to technical communication.

They pointed out something crucial: Technical communicators don’t operate under the transparent technology myth.

They never could.

Because technical communicators routinely attend to how tools shape content.

Style guide enforcement software. Content management systems. Structured authoring environments. XML editors. Publication pipelines.

Technical editors and content professionals have always had to think about where human judgment enters a production sequence and where automation can handle routine operations.

That’s the job.

You can’t manage content through production systems while pretending tools are transparent.

So when AI shows up, technical communicators are already prepared.

They already ask: Which tasks, at which stages, under what oversight conditions?

That’s exactly what Human-in-the-Loop AI collaboration requires.

Understanding AI Workflows

When most people try to integrate AI, they treat it like the transparent technology myth: just another neutral tool that captures thinking.

“Help me write this.”

But AI isn’t transparent. It’s deeply shaped by how you structure information, how you sequence prompts, what format you give it, what stage of the workflow it enters.

Getto, Kelley, and Vance describe the Human-in-the-Loop role as “manager of the process, validating outputs for whatever criteria they are aiming for.”

That’s familiar work if you’re a technical editor.

You already define tasks. Evaluate outputs against rhetorical criteria. Iterate based on results.

You already manage content through production systems.

AI is just applying the same analytical framework you’ve been using for structured authoring, content management, and publication workflows.

The question isn’t whether AI changes writing. Of course it does—tools always do.

The question is: Can you see well enough to manage that change strategically?

Here’s what this research tells us:

Most writers can’t map their own workflows because they were taught not to see tools as shaping work.

But content professionals—especially technical communicators and editors—were never trained that way.

You’ve always had to see the systems.

You understand that tools aren’t neutral. They shape how information flows, where friction happens, what’s easy and what’s hard.

You know where human judgment matters and where automation helps because you’ve been making those decisions for style guides, CMSs, and structured content for years.

That’s not a nice-to-have skill anymore. It’s the essential skill for AI integration.

Because you can’t integrate AI into workflows you can’t see.

And you can already see them.

The Value of Operational Thinking

The panic narrative says AI replaces writers.

But the research suggests something different: AI integration requires exactly the operational thinking that content professionals have been developing all along.

You’re not behind. You’re prepared.

You just might not have recognized workflow thinking as the strategic advantage it actually is. Because now that you can see the workflow, we need to understand what AI can actually do within it.

I’m Lance Cummings. Thanks for listening to Deep Reading.

If this changed how you think about AI and writing, share it with someone who needs to hear it.

And if you want the practical framework for mapping your workflows and integrating AI systematically, check out my newsletter Cyborgs Writing at http://www.isophist.com.

I’m also releasing a beta version of my Writing with Machines course soon for paid subscribers. For more information, click here.

The Skill That Makes Writers More Valuable in the AI Age

Lance Cummings — Tue, 20 Jan 2026 15:05:48 GMT

Image created by Gamma AI

Oftentimes, people are surprised that I teach writing … and AI. Isn’t that some kind of contradiction?

I often get this question: “What’s left for writers when AI can generate adequate text?”

The question itself reveals a misunderstanding about what makes writers valuable.

Last week I ran a workflow mapping exercise with twenty students from different disciplines—computer engineers, English majors, cybersecurity students, elementary education majors. I wanted them to map their actual writing workflows before we started integrating AI.

What happened revealed something important about the difference between describing a process and understanding a workflow.

The computer engineers struggled. They could explain their coding workflow in extraordinary detail. But their writing workflow?

“I just... do it until it’s done.”

The English majors did better initially. Brainstorming with clustering diagrams. Research with annotated bibliographies. Thesis development. Zero drafts for idea generation. Revision for argument structure.

In a world where AI supposedly makes writers obsolete, the ability to map operational workflows might be writers’ most valuable professional skill. Not just describing what you do, but understanding the entire system through which work flows.

But it became clear as we dug deeper that they were describing an idealized process or the intellectual steps they’d been taught. When I asked about the operational reality, they struggled too.

“I write in Google Docs, I guess?”

“My notes are... everywhere. Notebook, phone, scattered documents.”

“I don’t really have a system for organizing research.”

They could articulate the intellectual process. But the operational workflow remained largely invisible.

That distinction matters enormously for AI integration.

This pattern held across disciplines. Students from humanities backgrounds could articulate intellectual process, or the moves writers make.

Technical students could describe technical workflows, or the systems code moves through. But almost no one could clearly map their writing workflow, or the operational system through which their intellectual work actually happens.

This year I’m releasing beta access to my course, Writing with Machines, which helps writers and content professionals take a workflow approach to AI. Click below for more info.

Get Beta Access

The Irony of Technical Education

Those computer engineers already think in workflows for their technical work. They understand systems, information flow, version control, deployment pipelines, etc. They can map how code moves through their development environment.

But they haven’t applied that systems thinking to their intellectual work.

Meanwhile, English majors have been trained to articulate intellectual process, but typically in isolated academic contexts. They haven’t learned to map workflows in operational terms like tools, formats, handoffs, and information design.

Content professionals (or most professional writers) bridge both worlds. We understand intellectual process AND operational workflow.

We know that “writing” isn’t just thinking work. It’s information moving through systems.

Research lives in databases or note systems.
Drafts exist in specific tools with version histories.
Reviews happen through particular platforms.
Final outputs have required formats and distribution channels.

We already think in workflows because your work requires it.

When I research and collaborate with organizations like Motorola or Hitachi Energy on their content operations, the people who excel in operational roles typically come from content backgrounds. Not because they’re more creative (though they might be), but because they can see how information moves through a system and articulate what happens at each stage … in operational terms, not just intellectual ones.

That’s not a skill you automatically gain from learning to code or from studying literature. It’s a skill you develop from working in environments where content moves through systems and someone has to understand and improve those systems.

Why Workflow Mapping Matters Now

The ability to articulate intellectual process has always been valuable for writers. But AI integration requires workflow thinking … understanding the operational system, not just the thinking moves.

During the exercise, I asked students to identify friction points. Where do things slow down, where do you get stuck?

The English majors identified intellectual friction.

“I hate writing conclusions.”
“The transition from outline to draft feels like starting over.”
“Synthesizing research into coherent arguments is hard.”

The technical students identified task friction.

“Finding sources takes too long.”
“Bibliography formatting is tedious.”
“Blank page anxiety.”

But when I pushed them to think operationally, different patterns emerged.

One English major got specific about how she can write body paragraphs fine once she has her evidence organized. But she struggles with synthesizing everything into a conclusion that doesn’t just repeat what I already said.

This is process articulation. She knows the intellectual move that’s difficult (synthesis and elevation rather than summary). That’s valuable.

But then I asked: “Where are your body paragraphs when you’re trying to write the conclusion? What tool? What format? Can you see all your arguments at once, or are you scrolling?”

Long pause.

She then talked about how she writes each section separately, and by the time she gets to the conclusion, she’s forgotten what’s in the earlier sections … so she’s scrolling back and forth a lot.

Now we’re talking about workflow. The friction isn’t just intellectual. It’s operational. Information is organized in a way that makes synthesis difficult. The tool setup creates the problem as much as the intellectual challenge.

Compare that to “finding sources takes too long.” Also legitimate, but what’s the actual workflow friction? Is research scattered across multiple databases with different interfaces? Are you re-searching for things you’ve already found because notes aren’t organized? Is the problem identifying keywords, or is it that each source requires switching contexts and losing your train of thought?

Without mapping the actual workflow, AI integration becomes throwing technology at a vague sense of difficulty.

What AI Actually Needs From Writers

AI does one thing extraordinarily well: pattern-matching. It recognizes structures, suggests approaches, provides frameworks based on millions of examples.

But it can’t tell you which approach fits your specific situation unless you can map your actual workflow … not just the ideal process, but the operational reality.

The content professional who understands both the intellectual challenge (synthesis in conclusions) AND the workflow friction (information scattered across tools, requiring constant context-switching) can integrate AI strategically.

Maybe AI helps by consolidating key points from different sections. Maybe it suggests synthesis patterns. Maybe it’s not an AI solution at all. Maybe the workflow needs redesigning so information is visible when you need it.

The one who just says “help me write my conclusion” gets generic output because they haven’t mapped where the real friction lives.

This is why workflow mapping makes writers MORE valuable in AI environments, not less. Because effective AI integration requires:

Understanding how work actually flows through tools and systems
Identifying where in operational workflows AI can help (and where it can’t)
Distinguishing intellectual challenges from operational friction
Redesigning workflows when needed, not just automating bad processes
Making both intellectual work and operational systems visible for improvement

These are content operations skills. The kind of thinking you develop from working in environments where content moves through systems and someone has to understand, document, and improve those systems.

The organizations that integrate AI effectively have people who can bridge technical capability with operational thinking. They need people who can build the systems AND people who can articulate the workflows those systems support.

The engineers in my class will build better chatbots than the English majors in a functional sense. No question. But content professionals already understand how to map the operational reality of knowledge work, not just describe the idealized process. Both are needed.

The Mixed Group Revelation

By the end of class, the groups that produced the richest insights probably won’t be the homogeneous ones. They will be the mixed groups where an English major, an engineer, and an education major talked through their processes.

The English major can learn that version control isn’t just for code. It’s a strategy for managing drafts. The engineer can discover that concept mapping could organize technical documentation. The education major can realize her strategies for making content accessible to children were universal design principles.

They can teach each other to see their own processes differently.

This mirrors what I see in successful content operations teams. The organizations that integrate AI effectively have people who can bridge technical capability with operational thinking. They need people who can build the systems AND people who can articulate the workflows those systems support.

Content professionals bring that operational thinking. Writers understand modularity, that writing happens in chunks (research phase, outline phase, drafting phase) and that AI works best with chunked information and targeted requests. You know the difference between extending capability at a specific friction point versus replacing judgment in complex synthesis.

You already think in workflows. You just might not have recognized it as a marketable skill.

What This Means for Your Work

The panic narrative says AI replaces writers because it can generate text. But text generation was never the whole job.

The actual job includes:

Diagnosing what needs to be communicated and why
Mapping how information flows through operational systems
Identifying where automation helps versus where human judgment is essential
Designing workflows that support quality at scale
Maintaining meaning and accuracy across complex operations

These are content operations skills. Operational thinking applied to knowledge work. Workflow mapping, not just process description.

When organizations implement AI successfully, it’s because someone can map the operational workflows clearly enough to know where AI fits. When implementations fail, it’s usually because they’re throwing AI at undefined processes or, worse, automating workflows that were already broken.

The computer engineers in my class will learn to map their writing workflows this semester. But they’re starting from further back than content professionals and writers who already understand that content moves through systems.

We need to recognize that workflow mapping as the strategic skill that makes you valuable in AI environments.

What’s Next

I’m teaching this class all semester as a laboratory for systematic AI integration. The students are building an AI tool for disaster communication, which is high-stakes work where accuracy matters and hallucinations aren’t acceptable.

I’ll be sharing my thoughts as we move through the semester.

Next week I want to talk about the difference between AI pattern-matching and human reasoning. This is key to understanding AI workflows.

But you can’t integrate AI effectively into workflows you haven’t mapped.

If you want to move beyond casual AI use to systematic integration, I’m opening beta access to my Writing with Machines course for paid subscribers. It’s built around this exact challenge: mapping your actual workflows and integrating AI strategically at specific friction points.

The beta runs through this semester, giving you the complete framework while I refine it based on real-world feedback.

Learn more about the beta course → (paid subscribers only)

Subscribe now

P.S. Want to test this for yourself? Try mapping your own workflow for a typical content project. If you can articulate not just what you do but why you do it at each stage, you’re already ahead of most people trying to integrate AI. If you find it harder than expected—you’re in good company, and it’s worth figuring out.

Writing With Machines

Lance Cummings — Sat, 17 Jan 2026 19:52:24 GMT

Paid subscribers now get first access to something I’ve been developing for the past year: a complete course on systematic AI integration for content professionals and technical writers.

This isn’t just about casual ChatGPT tips or prompt hacks. It’s about building the operational framework you need to integrate AI strategically into your actual work …the kind that makes you more effective without replacing your judgment or expertise.

I’m making the beta version available to you paid subscribers, before the polished video course launches publicly. Here’s what that means and why it might interest you.

Subscribe now

AI works best when you can articulate your process clearly enough to identify exactly where it helps.

What This Course Actually Is

Writing with Machines teaches content professionals how to move from casual AI use to systematic integration. It’s built around a core insight: AI works best when you can articulate your process clearly enough to identify exactly where it helps.

The course covers ten chapters:

Understanding Your Workflow - Making your invisible intellectual process visible and identifying friction points
What AI Actually Does - Pattern-matching vs. reasoning, and why the distinction matters
The Five Information Types - Structuring content so AI can process it effectively
Grounding AI in Knowledge - Building reliable knowledge bases instead of hoping for accurate responses
Prompt Design - Moving beyond trial-and-error to systematic prompt design
Pairing Prompts with Process - Matching specific prompts to specific workflow stages
Style and Temperature - Controlling AI output characteristics deliberately
The Structured Principles - How content operations thinking improves AI integration
Building Your Prompt Taxonomy - Organizing prompts as operational assets
Workflow Redesign - The capstone where you transform an actual process using everything you’ve learned

This isn’t just theory. Each chapter includes practical exercises, templates you can adapt, and frameworks you’ll use immediately. Chapter 10 has you redesign a real workflow from your work. That’s your deliverable.

What Beta Access Means

The beta course is complete. All ten chapters are written and functional. But it’s developmental. It’s text-based with exercises and templates rather than polished video lessons. You’re getting the systematic framework and practical tools, not production value.

Here’s what you get as a beta participant:

The exchange is simple: You get early access to systematic AI integration frameworks. I get insights that make the final course better. Both of us benefit.

Why Beta Access Now

I could wait until everything is polished and perfect before releasing anything. But that’s not how I work, and frankly, it’s not how good courses get built.

The best educational materials come from real interaction with real learners.

You’re professionals doing actual content work. You have real workflows, real constraints, real stakeholders. Your friction points are different. Your applications will be different. Your feedback will be invaluable.

Also, I’m learning things from my research and teaching this semester that are making the course better week by week. The disciplinary differences I wrote about in this week’s newsletter? That came from watching students map their workflows. Those insights are already improving how I teach the concepts.

If you wait for the polished version, you pay more and you miss the opportunity to shape it. If you join the beta, you get the systematic framework now (which you can use immediately) and you influence what the premium version becomes.

Who This Course Is For

What About Cost?

If you’re already paying for Cyborgs Writing. This beta access is included in your subscription. No additional payment required.

The premium video course, when it launches, will cost significantly more. Early beta participants will get preferred pricing on that, but right now, your subscription covers this beta access completely.

If cost is a barrier: I’m willing to provide access to people who can’t afford or prefer not to pay for the subscription but are genuinely committed to the process and feedback. If that’s you, email me at Lance.cummings@hey.com and we’ll work something out. The goal is to build an excellent course, not to restrict access unnecessarily.

Access beta form below.

Content Modeling My 2025 and Beyond

Lance Cummings — Tue, 06 Jan 2026 15:30:20 GMT

Image generated by Gamma Ai.

I spent the last week of December doing something I hadn’t planned: modeling my own Substack archive as a knowledge graph.

I’ve been writing about AI-ready content, structured knowledge, and retrieval systems for two years now, and I’ve been wanting to see if I could structure my posts in a way that would make them more useful … to me, to readers, and eventually to AI systems that might help people navigate the archive.

What I didn’t expect was how much the modeling itself became a form of reflection.

So as my first post of 2026, I thought I’d give you some of my reflections on where I’m going with this Substack and how this knowledge graph exercise is helping me in the process.

The Year of Narrowing

First, some context. 2025 was a year of cutting away.

When I started writing about AI in education and technical communication, fewer people were doing it. That’s no longer true. Everyone writes about AI and plagiarism now. Everyone has opinions about classroom policy. The general conversation doesn’t need another voice.

And, well, if I’m honest, my ADHD brain kind of finds those discussions a bit boring these days.

So I began focusing on other things in information design by narrowing in on questions where my background in rhetoric and professional writing gives me something distinctive to say:

How does retrieval-augmented generation work from a compositionist’s perspective?
What can classical rhetorical frameworks tell us about prompt design?
How do we test content systems, not just prompts?

Posts like “Is Structured Prompting Dead?” and “Testing as Rhetorical Proof” came from this narrowing.

So did the Deep Reading podcast episodes on topoi and AI hallucination.

I wrote less often but went deeper when I did.

This focus also shaped my teaching and research. I’ve got a couple academic articles in the works, and I’m starting this semester with an interdisciplinary grant project, where students and faculty from English, Sociology, and Computer Engineering are building a knowledge graph for a crisis food communication tool.

The structured content work I’ve been exploring publicly is now something I’m building with students, in real time, with actual users (and trying to bring into scholarly conversations).

My goal as an online writer has always been to bridge the space between the workplace and academia. In the world of AI, this is more important than ever.

What the Model Revealed

Back to the knowledge graph experiment.

I used Claude along with Neo4j’s data modeling MCP server, which is essentially a tool that helps you design graph structures.

For those unfamiliar: a knowledge graph represents information as nodes (things) connected by relationships (how those things relate). Instead of storing content as documents, you store it as a web of connected entities.

I started by asking: What are the meaningful units in my archive? Posts, obviously. But what else?

The first interesting decision was distinguishing Concepts from Rhetorical Frameworks. Concepts are the ideas I write about, for example structured prompting, RAG, AI literacy, vibe coding. Rhetorical Frameworks are the lenses I use to interpret those ideas—kairos, topoi, stasis theory, rhetorical proof.

I could have lumped these together. They’re all “topics” in some sense. But separating them forced me to articulate how the classical rhetoric is the interpretive layer through which I read everything else.

Kairos informs how I think about vibe coding.
Topoi shapes my understanding of RAG.
Memory connects to knowledge graphs.

Mapping these relationships made explicit what had been implicit across dozens of posts.

The second decision was adding Claims. Most writers think of posts as being “about” topics. But when I added a node type for Claims—with properties like statement, claim type, and confidence level, each post became a collection of assertions at various stages of development. Some claims I state definitively. Others I mark as provisional. A few are speculative, ideas I’m testing rather than defending.

Here’s the knowledge layer of the model:

Claude generated graph in Mermaid....

This helps me understand that my archive isn’t just a collection of articles about topics. It’s a slowly developing argument, with claims that build on earlier claims, supported (or not yet supported) by evidence, attached to concepts that relate to each other in particular ways.

The EXTENDS relationship between claims might be the most useful part. My December post on rhetorical proof extends the prompt testing work from November.

Seeing that connection mapped changes how I understand what I’ve been doing. Not as isolated posts, but intellectual development over time.

When we create knowledge graphs for retrieval-augmented generation (and when we decide how to chunk content, what entities to extract, how to represent relationships) we’re doing philosophy whether we recognize it or not.

The Ontology Encodes the Epistemology

The usefulness of this activity goes beyond my own navel-gazing.

Knowledge graphs will become a key component to understand what we know (and what machines know) as AI becomes more ubiquitous.

Every knowledge graph encodes assumptions.

What becomes a node?
What becomes a relationship?
What properties matter?

These aren’t neutral technical decisions. They’re interpretive choices about what counts, what connects, what gets left out.

The model I built privileges argumentation—claims require evidence, ideas have lineages. It privileges rhetorical tradition as a distinct layer of interpretation.

Someone else modeling the same archive might structure it entirely differently. A computer scientist might emphasize technical concepts and tool relationships. A historian might organize by period and influence.

The structure reflects a worldview.

This has implications for the AI systems we’re all building. When we create knowledge graphs for retrieval-augmented generation (and when we decide how to chunk content, what entities to extract, how to represent relationships) we’re doing philosophy whether we recognize it or not.

The ontology (or how the graph is built) shapes what the system can know and how it can know it.

But as an academic, I also understand the impulse to structure knowledge exists alongside the recognition that any structure is partial. A knowledge graph doesn’t capture knowledge. It creates a frame for retrieval. What lies outside the frame matters too.

There’s something almost mystical about this. The more carefully you map what you know, the more visible the boundaries of your knowing become. Structure reveals mystery rather than eliminating it.

The graph doesn’t contain the territory—it just makes certain paths through the territory easier to find.

This is where our own cultural and religious backgrounds, many of which are invisible, become key to understanding how we set up AI systems.

I don’t have this fully worked out. It’s one of the threads I want to pull on this year.

How do philosophical and contemplative traditions inform how we design these systems?
What would it mean to build a knowledge graph that acknowledges its own limits?

I’ve been wanting to explore this for a while now. Focusing my work even tighter around these topics and arguments give me the opportunity to go deeper in 2026.

What’s Ahead

For 2025, a few things:

I’ll continue the work on structured content, knowledge graphs, and testing systems—but now with the grant project providing a concrete laboratory. Students will be building something real. I’ll be writing about what we learn.

The Deep Reading podcast will keep going. Research that connects AI systems to rhetoric, history, and context. Short episodes, but substantive.

And I’ll be developing my course, Writing with Machines, which takes the prompt operations material I’ve been sharing and structures it into a learning path that helps writers and teams integrate AI in ways that enhance expertise without taking away agency.

Paid subscribers get access to the beta version of this course. I’ll be testing new material with them before it becomes a more polished offering through Firehead Digital Communications. If you want to dig into the structured prompting work and help me think through what’s useful, that’s the way in. More to come on this soon.

The university doesn’t provide resources for this kind of public scholarship. No course releases, no dedicated funding. Paid subscriptions help me continue the work—and I’m genuinely grateful for every one of them.

Subscribe now

One last thought.

Building this knowledge graph was supposed to be an organizational project. It turned into something more like an examination of what I actually believe about my own work. The modeling surfaced assumptions I hadn’t articulated, connections I hadn’t named, and limits I hadn’t acknowledged.

Maybe that’s what structured content does at its best. Not capturing knowledge, but creating conditions for reflection—for ourselves, and eventually for the systems we build alongside us.

Here’s to a year of structuring, and of honoring what escapes the structure.

Metadata as Practice

One thing I’m committing to this year: tagging my own posts with the structure I’ve been writing about. Not just talking about AI-ready content—making it.

Here’s the metadata for this post:

Concepts: Knowledge Graphs, AI-Ready Content
Framework: Memory, ontology
Tools: Claude, Neo4j MCP
Builds on: “Is Structured Prompting Dead?”, “Testing as Rhetorical Proof”

Claims I’m making:

The ontology encodes the epistemology (definitive)
Building a knowledge graph is a form of reflection (provisional)
Structure reveals mystery rather than eliminating it (speculative)

The confidence levels matter.

I’m certain about #1.
I believe #2 but want more evidence.
#3 is something I’m testing—it might not survive contact with further thinking.

Over time, these tags will let me (and eventually you) trace how ideas develop across posts. That’s the hope, anyway.

What do you think?

Testing as Rhetorical Proof

Lance Cummings — Mon, 22 Dec 2025 15:15:18 GMT

Last semester, my students and I built a writing feedback chatbot for our technical communication course. In testing, it worked beautifully. Clear, specific feedback that maintained professional warmth. We deployed it.

Within two weeks, students started reporting inconsistent experiences. The same submission structure that earned detailed feedback on Monday produced superficial responses on Wednesday.

One student showed me screenshots. For example, the chatbot had praised her conclusion as “effectively synthesized” in the morning, then flagged the identical paragraph as “needing stronger connections” that afternoon. Same prompt. Same model version. Same student text.

This isn’t a bug. It’s just the way it is when working with language models. And it’s why prompt testing requires something more rigorous than “try it and see if it looks good.”

Evaluation as Craft

The ancient Greeks had a term for what we need: kritikē technē … Or the art of judgment. The word kritikē comes from krinō (to separate, to decide), and technē means a teachable craft. Together, they describe a disciplined practice for evaluating the worth, correctness, and fitness of language.

The grammarians who were a part of the library of Alexandria developed kritikē into a systematic discipline that created repeatable procedures for evaluating texts. Working with multiple manuscript copies of Homer, they faced a problem familiar to anyone testing AI outputs: variant versions of the same content, with no obvious way to determine which was best.

Their solution was to operationalize judgment. They established criteria, collected evidence across variants, applied standards consistently, and recorded their decisions so others could follow the reasoning. Judgment became a craft that could be taught, reproduced, and improved.

This is exactly what prompt testing requires. When we evaluate AI outputs, we’re not asking “does this sound good?” We’re asking whether the outputs meet specific criteria reliably enough for a particular purpose.

That question demands a method—articulated standards, systematic procedures, transparent documentation.

Image generated by Gamma.ai

Why Structured Prompting Demands Systematic Testing

When people claim structured prompting is dead, they’re usually working with single interactions or more dialogic collaborations. Ask a question, get an answer, move on (or continue working on that one instance). In that context, casual prompting often works fine.

But the moment you’re building something that needs to perform reliably across users, sessions, and contexts, you’re no longer in single-interaction territory.

This could be:

a classroom assistant
a documentation helper, or
a content generation workflow.

You’re building a system. And systems require consistency that casual prompting can’t guarantee.

My research on prompt format bears this out. When I tested the same complex task across four different structures, the outputs varied dramatically. Not just in efficiency (processing time ranged from 64 to 120 seconds) but in character. The unstructured prompt produced exploratory, wandering responses. JSON triggered mechanical, compliance-document prose. Natural structure with clear sections generated focused, efficient communication.

Each format created different statistical conditions for the model’s token prediction. JSON tokens co-occur with technical documentation patterns in training data, so generating JSON-formatted input increases the probability of formal, exhaustive output patterns. Unstructured conversational input co-occurs with exploratory discussion, so the model follows those statistical tendencies.

Structure gives you leverage over consistency, but only if you verify that your structure actually produces the consistency you need. And structure isn’t the only variable. Temperature settings, model selection, and context length all affect output character. (I explore temperature’s effects in a separate lesson.) Testing helps you understand how these variables interact for your specific use case.

Testing Prompts vs. Testing Code

When we test software, we’re verifying logical operations. Given input X, does the function return output Y? The relationship is deterministic. Run the test a thousand times, get the same result a thousand times.

Prompt testing operates on different principles. We’re examining rhetorical reliability, not verifying logic. Does the relationship we’ve established between human intention and machine interpretation remain stable across time, context, and varied inputs?

This distinction matters because it changes what we’re looking for. Code either passes or fails. Prompts exist on a spectrum of reliability, and our job is to understand where on that spectrum a given prompt sits for a given purpose.

The writing feedback chatbot didn’t “fail” in any binary sense. It produced plausible feedback every time. The question was whether that feedback remained consistent enough to be pedagogically useful … And whether students could trust that the evaluation criteria were being applied reliably rather than arbitrarily.

That’s a question of judgment, not logic. And answering it requires a method for judgment.

What We’re Judging

When you test a prompt systematically, you’re evaluating three aspects of the human-AI collaboration you’ve created.

Stance stability. Every prompt establishes a rhetorical stance, or a position from which the AI speaks. “You are a writing tutor who provides constructive feedback focused on argument structure and evidence use” isn’t just an instruction. It’s establishing a consistent voice and perspective.

But does that stance actually persist?

With our classroom chatbot, testing revealed that the constructive-tutor stance held firm for the first few exchanges in a conversation, then gradually drifted toward generic encouragement.

This could happen because the model’s training data contains patterns where tutoring interactions soften over time, or because earlier instructions lose influence as the context window fills with conversation.

Whatever the mechanism, the effect was measurable: stance drift under extended use. Testing helped us identify where drift occurred so we could add stabilizing elements—periodic reinforcement of the evaluative criteria, structural markers that maintained the rigorous-feedback pattern.

Interpretive framework reliability. Your prompt doesn’t just tell the AI what to do. It shapes how inputs get processed. When our chatbot prompt said “evaluate based on the technical communication rubric criteria,” we were creating conditions where rubric-related language would influence the output. But those conditions had gaps we didn’t anticipate.

The rubric worked well for standard assignments because the model had seen similar patterns. But when students submitted creative approaches, like an infographic, the statistical patterns broke down. The model couldn’t match rubric language to unfamiliar input formats, so it defaulted to surface-level observations about grammar and formatting. Testing with diverse input types revealed these blind spots. The fix wasn’t clarifying instructions—it was providing examples of the rubric applied to non-standard formats, giving the model patterns to match against.

Collaborative boundaries. Every prompt creates what I think of as a collaborative space, or the zone where human intention and machine capability overlap productively. Testing maps the edges of this space.

For the classroom chatbot, we needed to know: What types of student writing produce useful feedback? Where does the feedback quality drop off? What submission characteristics cause confusion or generic responses? Which edge cases does the prompt handle gracefully, and which break it entirely?

These boundaries aren’t obvious from the prompt text. They emerge only through running varied inputs through the system and observing where reliability holds and where it fractures.

Knowing what to judge is only half the challenge. The Alexandrian grammarians understood this. They didn’t just identify what made a text authentic or well-formed. They also developed procedures for making those judgments systematically: comparing variants, marking uncertainties, documenting reasoning.

Prompt testing requires both dimensions. We need criteria for evaluation—what counts as stable stance, reliable interpretation, appropriate boundaries. And we need procedures for applying those criteria.

The Rhetorical Appeals as Evaluation Criteria

The Alexandrian grammarians faced a problem we might recognize: they had no original to compare against. When scholars assessed a line of Homer, they weren’t checking it against some authoritative master copy—none existed. Homer was oral tradition committed to writing centuries after composition, and every manuscript was a copy of copies, each with its own variants and corruptions.

So how did these scholars develop criteria for judgment? By immersion in the corpus itself. They studied patterns across many manuscripts, inferring what Homeric diction typically looked like, identifying metrical conventions from the poems themselves, developing a sense of stylistic consistency through deep familiarity with the work. They’re standards emerged from the body of texts, then got applied back to evaluate individual passages.

We’re doing something similar with AI outputs. There’s no “ideal response” to compare against—just multiple outputs from which we infer what “good” looks like for a particular purpose. Our criteria emerge from examining what works, identifying patterns that characterize successful responses, and then applying those standards to evaluate new outputs.

But rhetoric offers a framework that accelerates this process: the three appeals. Aristotle identified ethos (credibility), pathos (emotional engagement), and logos (reasoning) as the fundamental dimensions of persuasive communication. These aren’t just persuasion techniques—they’re categories for evaluating whether communication works.

We’ve applied them to speeches, text, digital media … And now AI outputs.

When we adapt them for prompt testing, they become three lenses for examining output quality.

Ethos Testing: Can the Output Be Trusted?

Ethos in classical rhetoric establishes the speaker’s credibility and character. For AI outputs, we’re not assessing whether the model has credibility (it doesn’t, inherently), but whether the outputs are trustworthy enough for the intended purpose.

Trustworthiness breaks down into two components: consistency and accuracy.

Consistency asks whether the same prompt produces comparable outputs across multiple runs. This matters because inconsistent outputs can’t be trusted for systematic use. If a documentation prompt generates comprehensive coverage on one run and superficial summaries on the next, you can’t build a workflow around it.

Testing for consistency is straightforward: run the same prompt with the same input multiple times and compare the outputs. But “same” doesn’t mean identical. The question is whether variation falls within acceptable bounds for your purpose.

Consider a blog title generator. Testing the same article summary five times might produce five different titles—but if all five maintain brand voice, include relevant keywords, and target the right audience, that variation is a feature for brainstorming purposes. The prompt has sufficient ethos for generating options.

Contrast that with a product description prompt. If testing reveals 30% variation in which technical specifications get mentioned, the prompt lacks the consistency required for that task. Product descriptions need completeness, not creativity. The prompt would need explicit checklists and verification steps until testing shows reliable coverage of required elements.

Accuracy asks whether the outputs are factually correct and appropriately grounded. This is particularly critical for prompts that draw on domain knowledge or make claims that could be verified.

Testing for accuracy requires reference points—either human expert review or comparison against known-correct information. For our classroom chatbot, we tested accuracy by having instructors evaluate whether the AI’s feedback aligned with how they would assess the same submissions. Where the AI and instructors diverged significantly, we examined whether the prompt’s criteria were unclear or whether the model was introducing its own evaluation standards.

The ethos question for any prompt is: Can I trust this output enough to use it for its intended purpose? Testing answers that question with evidence rather than hope.

Pathos Testing: Is the Emotional Register Appropriate?

Pathos in classical rhetoric involves emotional appeal—engaging the audience’s feelings appropriately for the context. For AI outputs, we’re testing whether the tone and emotional register remain appropriate across different inputs and contexts.

This matters more than many practitioners realize. Tone inconsistency can undermine otherwise solid content. A customer service prompt that sounds helpful for simple questions but becomes condescending for complex ones will damage relationships regardless of how accurate the information is.

Imagine an automated feedback system for student writing. The prompt might maintain an encouraging tone when reviewing strong work but shift to patronizing reassurance for weaker submissions. Phrases like “You tried your best” and “Don’t worry, writing is hard” appearing only in responses to struggling students would unintentionally signal that the system had already judged them as less capable.

In this scenario, the prompt’s ethos could be fine—consistent, accurate feedback. But its pathos would be off, treating different students with different levels of respect based on submission quality.

Testing pathos requires diverse inputs that trigger different emotional contexts. For a feedback system, this means testing with:

Strong submissions (does it avoid excessive praise that might seem hollow?)
Weak submissions (does it maintain respect while identifying problems?)
Frustrated student language (does it respond with patience rather than matching the frustration?)
Confused questions (does it clarify without condescension?)

For a customer service prompt, you’d test across complaint types, customer tones, and issue severity. For documentation, you might test whether the prompt maintains appropriate professional distance when explaining both mundane features and exciting new capabilities.

The pathos question is: Does the emotional register remain appropriate across the full range of likely inputs? Testing reveals where tone calibration breaks down.

Logos Testing: Is the Reasoning Sound?

Logos in classical rhetoric involves logical argument, or the structure and validity of reasoning. For AI outputs, we’re testing whether the logical framework established in the prompt actually governs how outputs get generated.

This goes beyond checking factual accuracy (that’s ethos). Logos testing examines whether the prompt’s stated priorities, decision rules, and evaluation criteria actually shape the output—or whether they get overridden by other patterns in the model’s training.

Consider a documentation prompt that claims to prioritize accuracy but consistently chooses simpler explanations over precise ones. The prompt might include both “maintain technical accuracy” and “explain in accessible language.” In practice, accessibility could win out every tim—the AI sacrificing precision for readability without letting the user know.

This wouldn’t be a failure of the model. It would be a logical contradiction in the prompt that testing reveals. “Accurate and accessible” sounds reasonable until you encounter cases where accuracy requires technical precision that isn’t accessible. Without guidance for resolving that tension, the model could default to patterns from its training data, where accessible explanations are more common than technically precise ones.

Testing for logos means deliberately creating inputs that force your prompt’s priorities into conflict:

If your prompt says “be concise but thorough,” test with topics that can’t be covered both concisely and thoroughly. Which wins?
If your prompt prioritizes “user benefit” and “technical accuracy,” test with features where the accurate description doesn’t sound beneficial. What happens?
If your prompt establishes an evaluation hierarchy (“first check X, then Y, then Z”), test with inputs where X and Y suggest different conclusions. Does the hierarchy hold?

The logos question is: When the prompt’s instructions compete, does the output resolve conflicts the way I intend? Testing surfaces hidden contradictions and reveals which instructions actually govern behavior.

Combining the Three Lenses

Most prompt testing requires all three lenses, but their relative weight depends on purpose.

For a research summarization prompt, logos dominates. You need the reasoning structure to govern output reliably. Ethos matters for accuracy, but pathos is less critical since emotional register in research summaries is relatively narrow.

For a customer-facing chatbot, pathos may matter most. Users will forgive minor inconsistencies or occasional reasoning gaps if the tone feels right. They won’t forgive condescension or inappropriate cheerfulness when they’re frustrated.

For a compliance documentation prompt, ethos is paramount. Consistency and accuracy are non-negotiable. Pathos and logos matter, but trustworthiness is the threshold requirement.

When designing your testing approach, identify which appeals are critical for your use case and weight your testing accordingly. A prompt can have strong ethos but weak pathos (consistent and accurate but tonally inappropriate), or strong logos but weak ethos (sound reasoning but inconsistent execution). Testing across all three reveals the full picture.

From Criteria to Procedures

Knowing what to evaluate doesn’t tell you how to evaluate it. The Alexandrian grammarians understood this. They developed not just standards for judgment but systematic procedures for applying those standards: methods for comparing variants, marking uncertainties, and documenting reasoning so others could follow or challenge their conclusions.

These procedures translate surprisingly well to prompt testing. The Alexandrians were solving a version of our problem: multiple variant texts, no definitive original, and the need for judgments that could be taught, reproduced, and defended.

Recension: Multi-Run Comparison

The Alexandrians framed this work as diorthōsis and ekdosis. They collated multiple manuscript “witnesses” to identify variants, marked doubtful lines, and recorded their comparative reasoning in commentaries. Rather than trusting any single copy, they corrected the text and then issued a stabilized edition, choosing readings based on consistent patterns across the evidence.

For prompt testing, this becomes multi-run comparison. Never evaluate a prompt based on a single output. Run the same prompt with the same input multiple times and compare results.

This sounds obvious, but it’s surprisingly rare in practice. Most prompt development follows a pattern: write prompt, test once, adjust if the output looks wrong, test once more, deploy. That’s like Alexandrians looking at one manuscript and declaring it authoritative.

Multi-run comparison reveals what single tests hide. When I tested my four prompt formats, I didn’t just run each once. Each variation ran under controlled conditions, with metrics tracked across runs. The patterns emerged from comparison, not from any single output.

For practical testing, I recommend a minimum of three runs for informal evaluation and five or more for anything you’ll deploy in production. Compare outputs looking for:

Structural consistency. Do the outputs follow the same organization? If your prompt specifies a format, does that format hold across runs, or does it drift?

Coverage variation. Do all runs address the same key points, or do some outputs omit information that others include? For the blog title generator, variation in titles is fine. For product descriptions, variation in which features get mentioned is a problem.

Tonal range. Do all outputs stay within the same emotional register, or do some runs produce noticeably different tones? This is your pathos check.

Priority adherence. When the prompt contains competing instructions, do all runs resolve the conflict the same way? This is your logos check.

The goal isn’t identical outputs. That’s neither possible nor desirable. The goal is understanding the range of variation your prompt produces and determining whether that range falls within acceptable bounds for your purpose.

Athetesis: Marking Uncertainty

When Alexandrian grammarians encountered lines they suspected were spurious or corrupted, they didn’t simply delete them. They marked them with an obelos—a horizontal line indicating doubt. This kept them visible in the text. Future scholars could see the judgment, assess the reasoning, and reach their own conclusions.

This practice, called athetesis, prioritized transparency over tidiness. A marked line told readers: “This is questionable, but I’m preserving it so you can evaluate my judgment.”

For prompt testing, this becomes uncertainty flagging. When you identify problems in AI outputs, mark them explicitly rather than silently fixing them or discarding the output entirely.

This matters for two reasons. First, patterns of uncertainty reveal prompt weaknesses. If you’re consistently flagging the same type of problem, you’ve identified where your prompt needs revision. Silent fixes hide these patterns.

Second, flagged outputs become training data for your own judgment. Over time, a collection of marked outputs teaches you (and your team) what to watch for. The Alexandrians built scholia (or commentary traditions) around their marked texts. You can build similar institutional knowledge around flagged AI outputs.

A simple flagging system might include markers like:

H for hallucination (unsupported claims or fabricated details)
T for tone problems (inappropriate emotional register)
I for incompleteness (missing required elements)
C for contradiction (conflicts with prompt instructions or internal inconsistency)
D for drift (departure from established stance or format)

The specific markers matter less than consistent use. Pick a system and apply it across all your testing so patterns become visible.

Scholia: Documenting Your Reasoning

The Alexandrians didn’t just mark problems—they explained their judgments. Marginal notes called scholia documented why a line was suspect, what alternatives existed, and how the editor had reasoned through the decision. These annotations accumulated over generations, creating a scholarly conversation around the text.

For prompt testing, this becomes documented evaluation. Don’t just record that an output passed or failed—record why.

This is where most testing falls apart. Teams run prompts, glance at outputs, make a gut judgment, and move on. Nothing gets written down. A month later, no one remembers why certain prompt versions were rejected or what problems the current version was designed to solve.

Documented evaluation doesn’t require elaborate systems. A simple log capturing the following for each test proves valuable:

The input used. What specific content did you feed the prompt? Save it so tests can be reproduced.

The output received. Keep the full output, not just a summary or judgment.

Your assessment. Did it pass or fail on ethos, pathos, logos? What specific problems did you identify? Use your flagging system.

Your reasoning. Why did you judge it this way? What would have made it better? This is the scholia—the part that teaches future evaluators (including future you) how to think about the prompt’s performance.

When you revise a prompt based on testing, document what you changed and why. Link the revision to the specific test failures that motivated it. This creates a trail that makes prompt development cumulative rather than circular.

Putting Procedures into Practice

These Alexandrian procedures provide the methodological foundation. But you still need practical workflows for implementing them. The approach you choose depends on your technical resources and scale.

What the Ancient Art of Organized Thinking Says About AI Hallucinations

Lance Cummings — Mon, 08 Dec 2025 16:12:38 GMT

Welcome to Deep Reading. I’m Lance Cummings from Cyborgs Writing, and today we’re exploring a question that might sound simple but the deeper you dig, the more complicated it gets.

What happens when AI gets “confused”?

I recently discovered a metric called semantic entropy.

Before your eyes glaze over at the word “entropy,” let me explain why its important.

Semantic entropy measures how much an AI’s responses vary in meaning when you ask it the same question multiple times.

High entropy means the model generates different meanings each attempt—it doesn’t have stable knowledge, so it improvises. Low entropy means consistent responses.

This is one reason why AI hallucinates.

For this podcast, I’m going to try to bring this concept down to earth and make it actionable through the eyes of ancient rhetoric.

From an ancient rhetoric perspective, high semantic entropy is when your AI model is walking through a house with no rooms.

Let me explain.

For those reading, this is a transcript of the podcast, which can be listened to above or in your favorite podcast player.

Recent research into semantic entropy

A few months ago, a paper came out in Nature called “Detecting hallucinations in large language models using semantic entropy.”

They had developed and refined a way to measure when AI seems confused, even when it sounds confident.

Here’s how it works.

You ask an AI the same question multiple times. For example, “What are the installation steps?”

And you get back five different answers. Now, those answers might use different words, but do they mean the same thing?

If answer one says “First, power down the system” and answer two says “Begin by turning off power”—that’s low semantic entropy. Different words, same meaning. The AI got the response right.

But if answer one says “power down first” and answer three says “leave power on during installation,” then you’ve got high semantic entropy. The meanings contradict, and the AI is improvising. It probably isn’t building its answer on solid information.

This happens even when the AI seems confident. It’s not hedging with “maybe” or “possibly.” It’s just... making stuff up to fill the gap.

The researchers showed that semantic entropy can predict hallucinations with pretty good accuracy. When entropy is high, you’re about to get unreliable information.

Why this matters

Now, you might be thinking, “Okay, that’s interesting from a computer science perspective. But I’m a writer, a professor, a content developer. What does this have to do with me?”

Everything.

Because while this semantic entropy research is newer, a broader principle has been established across other studies: how you structure source content directly affects AI performance.

Research on RAG systems, or the technology most organizations use for AI-powered search and question-answering, shows that chunking strategy can impact performance as much as or more than the choice of AI model itself.

Think about what causes high entropy. The AI generates variable meanings because it doesn’t have stable grounding in what the source material actually says. In a way, it’s uncertain or guessing.

And what causes that “uncertainty”? The research suggests it’s often the source material. When documents are poorly organized, the AI does what a confused human reader would do. It fills gaps. Makes assumptions. Creates different interpretations.

The semantic entropy metric gives us a way to measure this instability. But the underlying principle isn’t new: structure matters for machine comprehension just like it matters for human comprehension.

I should add a note here. The AI model isn’t actually getting uncertain and that’s really part of the problem. The knowledge it’s working from is uncertain, but the AI is trained to be confident, and in the end, that’s what causes semantic entropy.

So you need confident information or knowledge behind your model to match the actual training to be confident.

Why Ancient rhetoric is still important

This problem isn’t new.

Ancient rhetoricians figured this out thousands of years ago.

They had to create speeches on the fly. In the Assembly, in the courts, at public ceremonies. No time to prepare.

Just, “here’s your topic, now speak.”

How did they do it? They used something called topoi.

The word literally means “places” or “rooms.” They organized their knowledge like a house with clearly labeled rooms.

Need to define something? Go to the definition room.
Need to compare two things? The comparison room.
Need to trace cause and effect? That room.

Having these stable mental spaces, or patterns, meant they could reliably find what they needed and construct coherent arguments quickly.

In 1984, Carolyn Miller wrote what became one of the most cited paper in rhetorical studies, called “Genre as Social Action.” And she argued that this is how all communication works. We recognize recurrent situations, and we reach for typified patterns of response.

When the situation recurs in recognizable form, we know what to do. We have stable knowledge structures to draw from.

When it doesn’t, we improvise. We hedge. We contradict ourselves across attempts.

Topoi in machine rhetorics

High semantic entropy is the computational version of lacking stable topoi.

When you ask an AI the same question multiple times and get semantically different answers, the model is doing exactly what a rhetor without proper topoi would do. It’s improvising under uncertainty. It lacks the organizational patterns, or the “rooms,” where specific types of knowledge reliably live.

But … you can create those rooms through content structure.

When you write a procedure with clear steps, properly labeled, you’re creating the “procedure room.”

When you write a concept explanation with a definition, characteristics, and examples in consistent order, you’re creating the “concept room.”

When you use consistent terminology throughout, you’re making sure the rooms have clear labels.

This is what structured content does. It is creating stable topoi for machines.

Low semantic entropy means the AI knows which room it’s in and what that room contains. It’s not guessing. It has reliable patterns to draw from.

What does this mean for you?

So what do you do with this?

First, understand that structure isn’t just about making content look organized. Structure is a signal. It’s how you communicate to both humans and machines.

“This is what kind of information this is, and here’s how to use it.”

Second, recognize that the same principles that help human readers help AI systems. Clear headings. Focused chunks. Consistent terminology. Explicit organization.

Third, start thinking of yourself not just as a writer but as an information designer. Your job isn’t just to explain things clearly. It’s to create reliable knowledge structures that work across contexts, including computational ones.

The content professional who understands this is going to be incredibly valuable as AI becomes more central to how information gets used.

Challenge

So here’s my challenge to you: Next time you’re creating content, ask yourself: Am I building a house with clearly labeled rooms? Or am I creating an unmarked space where readers, human or machine, have to guess what goes where?

Because high semantic entropy isn’t just an AI problem. It’s a content problem.

And content problems? Those are solvable.

Are you wondering how we might test for semantic entropy. Well, stay tuned. More on that soon!

Until then, I’m Lance Cummings. Keep reading deeply.

And if you’re testing this stuff in your own work, I want to hear about it. Find me on LinkedIn or drop a comment on the newsletter.

Talk to you next time.