Rob Peter to Pay Paul: When Prompt Engineering Starts Breaking Your Custom GPT
- ukrsedo
- May 31
- 3 min read
One of the biggest misconceptions about prompt engineering is the belief that adding more instructions automatically improves a Custom GPT.
In reality, every new instruction competes with the previous ones. Eventually, the model starts redistributing attention rather than improving its capabilities. You fix one problem and quietly create three more.
That is exactly what happened while I was building a new custom GPT for my CIPS IT Procurement Foundations training.
Originally, the GPT had a relatively simple role: to reliably retrieve course content, expose the uploaded knowledge, avoid hallucinations, and explain procurement implications. It worked surprisingly well. The GPT consistently returned structured topics such as Kraljic Matrix, Buygrid, Relational Contracts, and other concepts directly from the knowledge file.
Then came the dangerous intent: “Why won't I add one more capability?” So, I decided to introduce practical case analysis. On paper, it sounded like a quick win:
case introductions
student scoring
coaching feedback
challenge questions
course topic linkage.
The GPT did not isolate these instructions to “case mode.” Instead, it started applying the new behaviour to everything.
The moment I added concise coaching logic, the GPT began compressing normal topic retrieval as well. The moment I instructed it to “briefly explain” cases, it started summarising retrieved course content instead of exposing it faithfully.
The degradation was immediate:
related topics disappeared,
examples vanished,
external links were skipped,
nested bullets flattened,
formatting collapsed,
sections merged.
Nothing was technically “wrong” with the instructions themselves. The problem was the cumulative interaction between them.
At some point, the GPT begins optimizing against itself.
One instruction says: “Be concise.”
Another says: “Retrieve the full topic block.”
Another says, “Explain the concept.”
Another says, “Do not summarize.”
The model tries to satisfy all of them simultaneously, and the result becomes unpredictable.
Ironically, the final solution looked very similar to solving a procurement governance problem. We had to separate operational modes:
Topic Retrieval Mode,
Case Coaching Mode,
Assessment Mode.
Then another issue appeared: the knowledge files themselves.
Long PDFs, inconsistent formatting, duplicated explanations, and presentation-oriented layouts created semantic retrieval ambiguity. The GPT started blending sections because the source material itself was not retrieval-friendly.
The eventual fix was brutally simple: rebuild the entire knowledge base into rigid retrieval blocks with standardised structures, explicit delimiters, and normalised formatting.
In other words, the AI problem was not really an AI problem. It was an architecture problem.
That may ring a bell for some organisations rushing into AI adoption today.
Most failures are not caused by weak models. They are caused by poorly governed complexity layered on top of otherwise capable systems.
That's why it always amuses me when people post "use my prompts", which sounds to me like "skip some time on a diagnosis, take my medicine."
P.S. If these kinds of practical IT procurement and AI governance topics are relevant to your work, my IT Procurement Foundations course (try this CIPS link) explores:
SaaS economics
TCO of technology services
Kraljic Matrix in IT sourcing
Information asymmetry and Game Theory in negotiations
Incomplete and relational contracts, and many more.
Type "list" for the list of key topics covered in the course. Type "case #1" to "case #6" to check out practical exercises.
Because modern IT procurement is no longer just about buying technology - it is about managing complexity instead of collapsing under its weight.
Comments