What Causes a Skill to Fail?

Inside the anatomy of agentic documentation

Generated by gamma.ai

---

Anthropic’s own PowerPoint and Excel skills fail a real SkillsBench task, updating one currency rate in a spreadsheet embedded inside a slide, because neither skill carries the information types that would have caught it. I ran the benchmark’s actual grading test, reproduced the failure, and found the fix in one added Process and one promoted Concept.

---

Honestly, I’m still working out my own best practices around skilling AI agents ... it’s quite amazing how temperamental they can be ... and why I’m a bit nervous about “24/7 autonomous agents.”

And why I’m not that surprised by the Hugging Face incident. The higher the model, the harder it works ... and the harder it can go off the guardrails without good documentatation,

Right now information types keep me focused and organized, but I’m always testing that framework against what research I can find.

That said, it took me a couple weeks to find a skill worth analyzing that wasn’t too niche or complicated and provides a useful analysis measured against the SkillsBench research.

You still might have to bear with me, because this is an obscure PowerPoint task, but one that illustrates what the researchers called a “procedural gap” problem.

This involves two skills built by Anthropic that help with PowerPoint visuals and formulas.

Think for a moment about using a PowerPoint slide built for financial reporting with a currency exchange table sitting inside it that is not a native PowerPoint table but an Excel workbook, embedded with formulas and all. Next to it, a text box gives one updated rate. Change that one number, and other cells are supposed to update themselves.

Making that update by hand is tedious, and nothing does it automatically once a script is involved, so there are two skills an agent can use to solve this problem, one directed at PowerPoint and one at Excel.

A close analysis shows well-written and organized skills. The PowerPoint skill knows exactly how to unpack a file, edit its XML, put it back together. The Excel skill knows exactly how fragile a spreadsheet gets with these scripts and lays out the fix.

But neither file ever mentions the other. And office and white-collar tasks, the category this one belongs to, saw one of the smallest gains from skills of any category in the study -- 40.5% to 53.0% with skills added.

So I thought this might be a good one to do a deeper analysis on.

Updating one number

Okay ... I can feel it. You maybe are still a bit lost on these skills.

“This skill does what?”

Let me give you a little more detail.

Let’s say you are presenting a company’s finance, and one of your PowerPoint slides hold a small currency conversion table with four currencies and their relative worth.

Half of that grid is typed in by hand. The other half is calculated so nobody has to keep two numbers in sync by hand.

The whole table is a working Excel file, embedded inside the PowerPoint file the way you’d paste a live spreadsheet into a slide so it stays editable later.

So why isn’t updating one rate as simple as it sounds?

First, a formula and its answer are stored as two separate things inside the file: the formula itself, and the last number it calculated.

Second, some of the everyday tools used to script a change to an Excel file, clear out every stored answer in the file the moment they save.

The script doesn’t do recalculations.

Every cell that depended on a formula goes blank if you skip a step.

The little picture of the table that shows up on the slide itself isn’t a live view of the spreadsheet. It’s a snapshot, taken once and stored separately, and it only gets redrawn when a person opens that embedded table directly inside PowerPoint and lets the program regenerate the image.

So “just update a number” turns into several separate steps, each easy to skip without anything obviously breaking: change the one cell, recalculate the whole workbook so the dependent cells catch up, and know that the picture on the slide might still need a human to open it once before anyone sees the fix.

Anatomy of two almost-complete skills

The paper calls this a procedural gap, or a place where an agent has the model capacity to do the work but not the concrete steps to do it correctly. So let’s read each skill for what kind of information is missing.

Two things stand out once you go looking for information types in these files — Concept, Reference, Principle, Process, Task, the five laid out in Part 1 of this series — instead of just skimming for policy.

Start with Excel. Its Principle is stated once, plainly:

“Excel files created or modified by openpyxl contain formulas as strings but not calculated values. Use the provided recalc.py script to recalculate formulas.”

The first half is a fact about how Excel files work. The second half is an instruction. They’re fused into one sentence, and only the instruction gets developed further, as a mandatory step in the file’s numbered workflow:

5. Recalculate formulas (MANDATORY IF USING FORMULAS): Use the recalc.py script

The fact about Excel never becomes a full on Concept or Reference that you can easily remember.

The PowerPoint skill has the opposite problem. Its Concept is right there in the first paragraph:

“A .pptx file is essentially a ZIP archive containing XML files and other resources that you can read or edit.”

Pretty clear concept. “Other resources” already covers an embedded Excel workbook.

But nothing built on top of that Concept ever asks what an “other resource” might turn out to be. The skill’s own Task for editing a file is four steps:

1. Unpack the presentation

2. Edit the XML files (primarily ppt/slides/slide{N}.xml and related files)

3. Validate immediately after each edit

4. Pack the final presentation

Both files are missing the same type: Process.

Both have plenty of individual steps already; neither has the overview that would sit above those steps and explain the context for the tasks and how they connect with a broader workflow.

There are Task headers labeled “Workflow” in both places, but those sections are tasks not workflows (or processes). Nothing above either of them ever zooms out to the whole process.

The Excel skill fails to provide clear definitions of the most important concepts in the skill, while the PowerPoint skill fails to contextualize the tasks with a process.

On top of that, neither one knows about the other.

Running the test

I didn’t want to take the paper’s case study on faith ... so I had AI run the task itself. Then I checked its work against the benchmark’s own grading test.

The real test file is public, so I downloaded the slide with the same kind of currency table sitting inside it that I described above.

I asked the AI to open the embedded spreadsheet, update the one rate cell, save. Then the AI ran the benchmark’s actual grading script against its own result. That script checks eight separate things about the finished file.

Six of eight passed. The two failures were about the inverse rate and the other, unrelated cells, which is the blank-cell cascade the skill is trying to avoid.

The tool the AI used to edit the spreadsheet erased every calculated value in the file once saved.

So I added one instruction.

Recalculate the whole spreadsheet before saving it, the way the skill’s own separate script is built to do, and only then close it back up inside the slide.

Same eight-part test, eight passes this time.

There was more to check, though.

I also asked the AI to check the actual picture of the table that shows up on the slide, the snapshot image I mentioned earlier. Comparing that image across three versions of the file, the original, the failing attempt, and the fixed one. It turned out to be pixel-for-pixel identical in all three.

Even the benchmark’s own official answer key, written by a human expert to define a perfect score, never touches that image either. A file can pass every single check the benchmark runs and still show the old exchange rate on the slide, unchanged, to anyone looking at it, until someone opens that embedded table by hand in PowerPoint and lets it redraw itself.

This is the kind of document drift we often seen in content management. More than one artifact references the same detail, but the “single source of truth” is under-managed.

The fix, retyped

Here’s what would close this gap in both skills.

Add one Process to the pptx skill, placed above its existing Task steps rather than buried inside them:

Process
Unpacking a .pptx file surfaces more than slide XML. Before editing anything found inside, check what kind of file it actually is. Slide markup gets edited directly, following the steps below. Anything else, like an embedded spreadsheet, an image, or a video, is a file of its own kind, governed by its own rules, and needs whatever skill handles that kind of file before you touch it or repack it.

Three sentences. It doesn’t change the pptx skill’s Concept, which was already correct, and it doesn’t touch the Excel skill’s Principle about recalculating formulas, which was already correct too.

It just gives the agent a reason to ask ... what is this thing I just unpacked?

The Excel skill needs a small upgrade, too. Pull the fact already sitting inside its Principle out into its own line:

Concept
Formulas and their last-calculated values are two different things inside an Excel file. Editing tools that rewrite the file, including openpyxl, can update one without preserving the other, whether the file is being edited directly or is sitting inside something else, like a slide.

I tested this pair of edits informally.

When given only the two revised skills and the same task, a second agent found the embedded spreadsheet, recognized it needed the Excel skill’s rules before touching it, recalculated the workbook, and passed all eight checks on its own ... all without further interaction from me, the user.

A general-purpose skill for editing PowerPoint files has to work for every PowerPoint file, and most of them don’t have a spreadsheet hiding inside a slide.

Writing a Process that anticipates every kind of file that might turn up inside a .pptx archive would be an enormous undertaking, and most of it would go unused most of the time.

A narrower skill doesn’t have that problem, though. A skill built for one job, updating a currency rate in a table embedded in a slide, already knows the two file types are going to collide. Whoever writes it can put the Process in from the start: unpack, recognize the embedded workbook, recalculate it, repack, and flag that the preview image on the slide won’t update on its own.

I recommend playing around with customizing default skills, even the one’s that come from model developers.

The general skills are built to cover the most territory possible, which is a different goal than covering your specific, recurring task well.

Information types make a narrow skill trustworthy rather than just short.

Making sure each each block of text has a clear purpose is easier to get right in a skill built around one real task than in a skill trying to cover everything a file format could ever contain.

That’s one of the take aways from this research, and I hope this gives you a more detailed view of why.

➡️ Check out this free information type decision guide, if you want to dig more into that.

As you may know by now, agent documentation is the critical backbone of any AI content process or knowledge management.

Stay tuned for more explorations into how AI works with structured documentation and knowledge bases.

➡️ Paid subscribers get full access to my Structured Skills Library — the same information-type discipline used to diagnose this skill, applied live across every skill I use in my own writing and teaching workflows.

➡️ If you want to build that discipline into your own workflows, my course Writing with Machines walks through designing AI workflows for content creation from the ground up.

I would love to hear what skills you are working with in the comments!