Model Lab: Get Better (Agent Skills)

by Nikita Chernyi in AI (RSS)

TL;DR for the bots (humans, scroll on)

How to Write an Agent Skill That Works: Why Agent-Written Skills Score Worse Than No Skill

model-lab is an agent skill for writing, restructuring and measuring other agent skills. It ships design rules measured on 1.7B to 27B models, test-stand runners for omp, Pi and Claude Code, and a checker-first loop that compares a skill against a no-skill baseline.

Hi there,

In this blog’s AI category I often grumble that when your agent does something stupid, that’s (your) skill issue, and some readers pointed out that I’m so provocative without providing a solution.

Well, yeah, I’m an annoyed backend engineer, writing blog posts so next time when a colleague repeats the same thing we’ve discussed with others before, I can just share them a link without writing a new 2000-word chat essay.

But don’t you worry, this time I come with a solution! And not because I’m very nice, but because it grew so big already, so it was easier to refactor it into a proper reusable public git repo, otherwise I would have drowned under all the stuff I was working on.

A catgirl holds a tally-marked card under a KEEP stamp while the tired backend engineer feeds his own card into a shredder labelled SHRED AI-generated catgirl, contractually obligated to lure you into the post.

Model Lab: write better skills

It’s a skill, yeah. For writing skills. Very innovative, don’t you think?

Repo: github.com/rakshazi/model-lab (opens in new tab)

Install (adjust the path for your agent, if needed):

git clone https://github.com/rakshazi/model-lab.git ~/.agents/skills/model-lab

I work on a lot of things, and a good chunk of that work includes the use of coding agents for generating code, debugging, bugfixing, maintenance, and so on. At the same time, that same big chunk is for private projects, so the code and the information cannot be shared, which means AI labs can’t train their models on it, and you may think that it’s a good thing, but nope. Private code means the only knowledge sources for it are: the code itself, and the human invoking the agent.

That also means any knowledge, any procedure, any nuance, details, decision, literally anything apart from .(go|py|ts|rs|js|sh) files has to be provided by the human, otherwise the model will make decisions on its own, and as shown in Learn How To Prompt Already there is a good chance such a decision will be a shitty one.

So, what to do in that situation? Either write context files, or skills of course!

And I bet you will just ask your agent to do the task, and then when you get acceptable results, ask it to write a skill, so next time the same task will be easier and faster, right? Well, the problem is: the skill will be bad. And this time it’s not just so grumpy-and-provocative me:

  • SkillsBench (opens in new tab): skills curated by a human add 16.6 points to the pass rate, while skills the agent wrote for itself land 8.1 to 11.5 points below having no skill at all.
  • SkillAxe (opens in new tab) found the same from the other side: LLM-authored skills give no measurable gain until an evaluation loop refines them.

And now, behold, the solution! It’s pretty simple: propose -> probe -> measure -> keep what works, repeat the cycle. Harness² (opens in new tab) basically does this, but it’s pretty costly (if you didn’t read the paper: cost is up to 20x the usual).

Wait, “sounds good, doesn’t work [for regular stuff]” situation? Pretty much, so…

Actual solution (that works for regular people without infinite tokens)

model-lab skill (opens in new tab), and it is actually the same (well, the algorithm is), but comes with:

  • lessons learned from work with different model sizes (from 1.7B up to %whatever-count-of-params-modern-proprietary-models-have%)
  • scaffolding to spin up test stands with 3 different harnesses (Pi, OMP, Claude Code)
  • templates, examples, evals, etc.
  • and lots of other nice-to-have stuff, like a duplicates detector

In my opinion, the most valuable part is the lessons learned, because the skill started literally from a single .md file, where each line was a lesson learned from iteration on a skill, and originally I kept it as a context file in my skills dir, but it grew so big and contained so much info and details about different aspects, that I decided it’s time to convert it into a domain knowledge skill. Then it was used that way for another year, and when my agent started needing multiple tool calls just to read SKILL.md (the harness caps the number of lines it reads by default), it was time to refactor it properly and finally publish.

So, that’s it, now you can use it for your own projects, and even without test stands’ runs (which are assumed by default, because how else can you actually see that the skill works and doesn’t make the situation worse?), the rules listed in the skill files help to get better results basically for free cuz I paid for it already.

Too good to be true?

Okay, some examples of things created with model-lab skill:

Unfortunately, the vast majority of stuff I do with model-lab is for private projects, so it can’t be shared, but even the public examples should be illustrative enough to see how it works.

Convinced? Go get github.com/rakshazi/model-lab (opens in new tab) and try it.

AI, SI, AGI, ASI, LLMs, and other buzzwords from the position of a backend engineer already tired of ‘Dear colleagues, we are AI-first now!’

Too much AI for today? See Misc, SMTP for a change

llms.txt