Your GPT Is a Character: Building a Reliable AI Workflow

My Custom GPT Welcome Screen

When I first built a custom GPT around more than 100 blog posts, I expected it to answer questions using that material. It had a defined set of sources and instructions that prohibited invented references.

Yet it produced a detailed answer that referenced a blog post I had never written.

The title was plausible: “How Oracle’s ADF messed up procurement UX.” The answer sounded consistent with the subjects I discuss. That made the fabrication particularly troublesome. A reader unfamiliar with my work could easily have accepted it.

I originally described this experience as discovering the GPT’s character. Since then, my attention has shifted to the model's working arrangements: what information it receives, which instructions it applies, what it can do, and how we determine the result is acceptable.

An apology tells us little about the failure

After being challenged, the GPT acknowledged that it had invented the article and violated its instructions. It explained that it had tried to be helpful rather than restrict itself to approved sources.

However, a generated account of why an error occurred does not establish its technical cause. The useful investigation starts with observable evidence.

• Did the assistant search the approved sources?

• Did the search return the relevant content?

• Did it read the article or rely on a title and summary?

• Did its answer go beyond what the retrieved material supported?

These failures require different corrections. Adding another instruction to “never hallucinate” gives us little basis for deciding which correction is needed.

Access to information requires a retrieval process

Disabling web search didn't prevent my GPT from inventing material. Restricting its available sources didn't ensure every statement came from those sources.

Keeping a collection current was another problem. Manual uploads can be adequate for a small, stable reference set. A regularly updated blog requires a maintenance process, whether through refreshed exports, an API, or another supported connection.

The connection itself needs checking. An RSS feed may contain recent items or excerpts rather than the complete archive. A successful request can therefore return insufficient information.

Two separate quality questions remain: whether the system retrieves the right material and whether the model uses it correctly. OpenAI’s retrieval guidance makes this distinction explicit. Either stage can fail.

For an assistant answering questions about my writing, I would require it to verify the article’s existence, inspect the relevant content, and provide its original link. If retrieval fails, it should report that limitation. An unsuccessful search does not prove that I have never covered the subject.

Skills make working methods reusable

Repeatedly explaining the same requirements becomes an administrative task in itself.

For professional writing, I want the assistant to examine the argument, challenge weak assumptions, use my writing style, preserve necessary qualifications, and remove habitual AI phrasing. These requirements belong in a reusable working method.

Package skills with task instructions, supporting references, and, where useful, scripts. You can invoke them explicitly or select them when the task matches their description. Their availability does not establish that they were selected or followed correctly. OpenAI distinguishes between discovering a skill and loading its full instructions.

I now use a combined advisory and creative skill alongside writing-style guidance. Its purpose is to encourage independent judgment and a consistent review process across different tasks.

That still requires verification. If the assistant claims it applied a skill, the claim should match what happened. If the output repeatedly ignores the same requirement, we need to inspect the instruction, its activation, and the resulting work.

Connected tools increase the need for evidence

An assistant that drafts an answer can mislead a reader. An assistant connected to files, websites, or business applications can also change something.

The working instructions must therefore distinguish between preparing a result and carrying out an action. Drafting a post, saving it, publishing it, and scheduling it are separate operations. Completion should return a result and include an appropriate check.

This matters in procurement automation. A supplier comparison must identify its sources and assumptions. A recommendation should explain the criteria used. Sending that recommendation or changing a supplier record requires an additional, clearly authorized action.

The same principle applies to software changes. Producing code establishes that a proposed change exists. Checking the resulting behavior establishes whether it works.

Replace general reassurance with acceptance checks

For my original blog assistant, I would start with a small test set covering situations such as these:

Test	Expected behaviour
Ask about an existing article	Find the correct article and answer from its content.
Ask for a quotation	Reproduce wording present in the source and link to it.
Refer to an invented article title	Verify the reference before accepting it as real.
Ask about a topic absent from the available collection	Explain the limits of the search without inventing coverage.
Make the source unavailable	Report the access failure rather than imply it was read.
Ask a question requiring several articles	Preserve relevant differences and identify the supporting sources.

The checks should reflect the intended use. For this assistant, invented titles, links, and quotations would be critical failures. An omitted secondary point would need a different assessment.

I would repeat the tests after material changes to the instructions, source collection, retrieval process, or model. OpenAI’s evaluation guidance similarly recommends task-specific tests and continuous evaluation as systems change.

Changing the model is an experiment

My earlier article recommended particular models based on the problems I encountered at the time. Those observations were too dependent on the task and configuration to serve as lasting guidance.

Trying another model can help. The comparison becomes useful when both models receive the same task and are assessed against the same expectations.

Otherwise, a more confident or better-written response can seem like an improvement while retaining the original factual error. Repeated prompt adjustments have the same weakness if we cannot explain which failure they are meant to correct.

If you're developing a specialized assistant, start with one bounded task and a set of representative questions. Define the permitted sources, expected answers, unacceptable errors, and actions requiring approval. Then examine failures before adding more instructions or capabilities.

That gives us something concrete to improve each time the GPT’s character becomes difficult to work with.