Demo-to-production is the hardest gap in AI development right now.
The demo works every time. Production fails in ways you didn't
predict, at times you didn't expect, with inputs you never tested.
Here are the mistakes I see most often.
MISTAKE 1: NO OUTPUT VALIDATION
You asked the model for JSON. It returned JSON — except this time
it wrapped it in a markdown code block. Or added a sentence before
it. Or returned a slightly different key name.
If your code does JSON.parse() on raw model output without
sanitizing it first, it will eventually throw an error in production.
Fix: Strip markdown formatting before parsing. Validate the schema
after parsing. Have a fallback for when validation fails.
This is not optional.
MISTAKE 2: SYNCHRONOUS API CALLS BLOCKING THE UI
LLM calls take 2–15 seconds. If your backend makes a synchronous
API call and the frontend waits for a response, users are staring
at a frozen UI.
Fix: Make the call async. Return a job ID immediately. Let the
frontend poll or use websockets to get the result when it's ready.
Or stream the response so users see output appearing in real time.
MISTAKE 3: NO RATE LIMIT HANDLING
OpenAI and Anthropic both have rate limits. Hit them and your
app throws a 429 error. If you haven't handled this, that error
reaches your user.
Fix: Catch 429 responses. Implement exponential backoff — wait,
retry, wait longer, retry again. For high-volume use cases,
implement a queue so requests don't all fire simultaneously.
MISTAKE 4: PROMPTS THAT WORK ON CLEAN DATA ONLY
You built your prompt with neat, well-formatted test data.
Real users send incomplete forms, weird characters, very long
inputs, and edge cases you never imagined.
Fix: Test your prompts with messy data from day one. Add
instructions for handling missing fields. Set a max token limit
on inputs. Sanitize user input before it reaches the prompt.
MISTAKE 5: NO LOGGING
Something goes wrong in production. You have no idea what the
model was sent or what it returned. You can't reproduce it.
You can't fix it.
Fix: Log every API call — the prompt, the response, the latency,
the model used, and any errors. Not forever, but long enough to
debug issues when they come up. A simple database table or
a log file is enough to start.
MISTAKE 6: TREATING THE MODEL AS INFALLIBLE
Models hallucinate. They return confident wrong answers.
They occasionally ignore instructions.
If your product relies on model output being correct 100% of
the time without any verification layer — it will eventually
fail in a way that matters.
Fix: Add human review for high-stakes outputs. Use confidence
scoring where possible. Design your UX so that wrong outputs
are catchable before they cause real damage.
None of this is complicated. It's just the engineering discipline
that separates a prototype from a production system.
If you need an LLM integration built properly — not just working
in demo, but reliable in production — that's what I specialise in.
Get in touch.
