Learn How To Prompt Already.
Hi there,
I revived this blog to rant about why you don’t need frontier LLMs, and why your permission policy is a joke, then I switched to how to do good with Krea 2 prompting, and now I’m ready to give you the best of both worlds: ranting and showing cool things at the same time!
So, multiple people poked me with comments like
“hey, you are telling with a straight face that you have to prompt model correctly, but
look, I can tell my %whatever model of whatever provider of whatever size% does the job
even with simple Do the job, make no mistakes prompt, and it just works!”
Yeah-yeah, I know, I’m reviewing your “works” after that, and I won’t even argue that it “works”.
The question is how good it actually works, and I’m tired of pretending “good” is even on the table.
So, I decided to prepare this post with an example of an actual small feature, within an actual small project. Why “small”? Because I want you to read what’s going on and not “sure, will read it on %never%”. The examples are from actual planning sessions, btw, because supposedly reading markdown slop is easier than code diff slop.
Setup: same harness, model, configuration, parameters, project, etc, with the only difference being the prompt. And yes, of course it is a single example to illustrate the point instead of lab research with 100500 experiments under ideal conditions.
The Content Starts Here: The Prompts
Job: in a small project, implement SMTP email sending alongside the existing Postmark email sending for simple emails (mostly plaintext markdown, no fancy HTML or images).
2 approaches, copy-pasted from the actual session transcripts I ran:
Make No Mistakes
Plain and straightforward.
Implement SMTP mail sending in addition to postmark in internal/mail
Design Prompt
Naming design, constraints, and structure.
Implement SMTP mail sending in addition to postmark in internal/mail/, some pseudocode below:
1. add common interface Sender { Send(ctx context.Context, req *Postmark.Email) error } into internal/mail/mail.go
2. decouple postmark logic into internal/mail/postmark.go:
a. type PostmarkSender
b. func NewPostmark(token string) Sender
c. func (p *PostmarkSender) Send(ctx context.Context, req *postmark.Email) error
3. add smtp logic into internal/mail/smtp.go:
a. type SMTPSender
b. func NewSMTP(host, port, login, password string) Sender
c. func (s *SMTPSender) Send(ctx context.Context, req *postmark.Email) error
4. refactor internal/mail/mail.go:
a. New(from, to string, sender Sender) *Client
b. refactor (*client) Send() to use sender
5. Add config, main.go, README.md changes, and of course, tests
The slop results
And here are the result plans, around 13kb and 21kb. I’ve put them on separate links so you could easily skip them and jump to conclusions!
“Make No Mistakes” plan (13kb)
Context at the end of the planning session: 140k tokens.
Context at the end of the planning session: 81k tokens.
So… what’s the difference?
(Playing slop advocate) Both plans do the job, and in the end both work, so…
So one produces AI slop and the second one produces somewhat maintainable code (I won’t pretend it’s perfect, tho).
This is what I meant about domain knowledge in previous posts - the model you use, despite how awesome it is, doesn’t know anything about your project and your code conventions. It assumes based on whatever it was trained on, and commonly this is not what you want to get in your codebase.
“Design Prompt” comes with interfaces, because in that project interfaces are a thing, yet unless explicitly specified, the result will be an abstraction-free mess. Of course, with 2 sender options you may say “hey, whatever, not so much to worry about!” - yeah, except for the config gates, policy defaults, and silent fallbacks that the second sender drags into what used to be a single send call.
“Design Prompt” comes with a mention of documentation (README.md), and of course the plan contains proper additions there, but “Make No Mistakes” never mentions it, so neither does its plan.
“Design Prompt” comes with naming specific files, so the result plan contains logically placed code that will be easy to maintain in the future, also the files are small enough to be easily reviewed and understood, while the “Make No Mistakes” plan just shoves everything into a single place - exactly the way to 30k loc files in the future.
“Design Prompt” comes with exact signatures, and they pay off in tests: the Postmark constructor takes options, so its test swaps in a local test server and exercises the whole send path, and the SMTP message gets asserted header by header. “Make No Mistakes” needs a test-only NoTLS escape hatch just to talk to its own stub, and leaves the Postmark path untested. Guess which plan catches a regression a year from now?
The funny part: “Make No Mistakes” plan invents attachment support nobody asked for, but drops the X-Tag header the project actually uses. No mistakes, yup.
An overlooked “bonus” part
When you tell the model “do the thing” and let it roam free on your codebase, it spends its attention and context budgets on figuring out your project, because again - it has no idea what it is. And both budgets are limited, with the attention one being the most valuable, because “My model can handle 1 million tokens of context” doesn’t mean the model’s “smartness” is up to 1 million tokens.
For example, Matt Pocock (the “grilling skill guy”) in one of his talks mentions that he considers the model’s “smart zone” (opens in new tab) to be around 125k tokens of context, after which the model just becomes dumber - it misses more nuances, hallucinates more, and so on. And this was about frontier SOTA models, the most it-is-so-dangerous-it-could-do-anything kind.
I won’t claim this (or any other) specific number is definitely accurate, but I can say from my experience that degradation can be noticed with the naked eye around 400k tokens (40%) of context - the model just starts to forget things it got at the start of the context.
So, when the model reads arbitrary files of your codebase, it fills the context quite fast, and the “dumb point” is reached pretty quickly.
When your prompt is specific, names constraints and design decisions, it helps to keep the model in the “smart zone” and focus on the nuances of the task in hand.
By the way, in case you skipped the plans, the “Make No Mistakes” plan took 140k tokens of context, while with the “Design Prompt” it took 81k tokens.
The (repeated) conclusion
The best way to get the job done with LLMs is to prompt it with clear and specific instructions: your head is the best knowledge source, so give the model the decisions you already know and name constraints explicitly. “Make no mistakes” is not a specification, you know.
The model you use may be the most powerful thing in the whole universe, but it knows nothing about your project. Vague “do the thing” will produce a mess - yes, it will probably technically work, but an LLM is a big amplifier: a small “not good” thing missed today gets generalized and poured over the whole codebase, and a year later you’re debugging the resulting nightmare.
Why do people forget they’ll need to maintain the thing they generated today in 6 months or a year?