System Prompt or Short Rules: How to Reduce Drift

Why a long prompt starts to drift

A long system prompt rarely breaks in one obvious place. More often, it slowly loses shape: the model sees too many rules at once and becomes worse at telling what is mandatory and what is only desirable. When prohibitions, tone, format, exceptions, and rare edge cases are mixed together, priorities blur.

This becomes especially noticeable after a few rounds of edits. The team adds one caveat, then another, then inserts a new paragraph between old ones. A month later, the text already looks like a patchwork. The model reads everything in one pass, while people start interpreting the same lines differently.

Usually a long prompt drifts for four reasons:

• rules of different weight sit side by side;
• one new phrase changes the meaning of the one next to it;
• exceptions swallow the general rule;
• mandatory requirements get buried in explanations.

Because of this, the debate over whether a system prompt or short rules are better often comes down not to model quality, but to the readability of the instructions themselves. If the text cannot be scanned in a couple of minutes, review almost always turns into a discussion of wording. One person wants to replace must with should, another argues about polite tone, and checking real answers gets pushed back.

There is also a more practical problem. A new team member spends a long time looking for anchor points. They need to quickly understand which rules cannot be broken, which can be changed safely, and which ones were simply carried over from an older product version. In a long monolith, none of that is obvious. You have to read everything and guess what matters most.

That is how instruction drift accumulates even without major releases. Model answers become a little less consistent, review takes longer, and every edit feels risky. If the team compares several models or routes requests through AI Router, an overloaded prompt drifts even more: the same text gets interpreted differently by different models. The problem is no longer the model itself, but how the rules are packaged.

When short rules give you more control

Short rules work better when each instruction has one job. One rule sets the tone of the answer. Another forbids exposing personal data. A third fixes the format. Then it is easier to see which part truly affects model behavior and which part is just taking up space.

A long system text has an unpleasant side effect: it is easy to argue about, but hard to change. If the answer becomes too formal, nobody knows whether the cause is the style paragraph, the safety block, or an old exception that nobody remembered to remove a month ago. With a set of short rules, the cause becomes visible faster. Remove one rule, run the tests, and the difference is clear right away.

This is especially useful where scenarios change often. Support needs one set of rules. Internal document search needs another. A banking assistant needs extra constraints around wording, escalation to a human, and handling sensitive data. Instead of three almost identical long prompts, the team keeps a shared base and turns on the needed blocks depending on the case.

What is easier to review

It is easier for reviewers to read separate meaning blocks than to untangle a solid wall of text across two pages. In practice, four types of rules are usually enough:

• the goal of the answer and the allowed tone;
• format: length, structure, language;
• prohibitions and restrictions;
• what to do in edge cases.

This approach holds up better across versions. If legal changes one requirement, you update a small block instead of touching the whole prompt. The change history is cleaner too: it is obvious what was fixed and why.

In teams that use AI Router, modularity is especially useful. The app’s base prompt can stay shared, while rules for PII masking, audit logs, and content labels are attached by scenario. Less code changes, reviews move faster, and instruction drift shows up in the first tests.

Short rules do not make the system simpler by themselves. They just avoid hiding mistakes. For review, that is almost always a plus.

When one big prompt is still convenient

A large system prompt often gets criticized unfairly. If you have one scenario, few exceptions, and a clear response style, one cohesive text can work better than a pile of small rules scattered across different files.

That happens in teams where the assistant does the same job every day. For example, it answers questions about internal policy, helps an operator with a prepared script, or briefly summarizes documents in one format. If the model’s behavior barely changes from week to week, splitting it up brings little benefit and only creates more places to get confused.

There is no universal recipe here. If requirements change rarely, a big prompt reduces the number of decisions at the start. The team can launch a pilot faster because it does not need to separately think through module priorities, rule order, and versioning for every block.

There is another case too. Sometimes the problem is not the instructions at all. The model starts answering worse because it is given weak context: an incomplete document, a noisy knowledge base sample, or an overly broad user request. In that situation, rewriting the rules again and again is almost pointless. First, fix the search, the prompt template, or the example set.

It is too early to split the prompt if:

• the task has one owner and one clear scenario;
• exceptions are rare;
• the team wants to launch quickly and pass the first review cycle calmly;
• errors are more often caused by context than by instruction conflicts.

One big prompt is also convenient for early tests across multiple models. It is easier to keep one base text and see where behavior diverges. Later, once stable differences appear, the rules can be moved into modules.

There is only one condition: such a prompt should be as short as possible. Do not turn it into a dumping ground for wishes. If you choose one text, separate constant rules from variable fields and decide in advance when you will revisit it: after an incident, a policy change, or a series of repeated errors.

How to break a long prompt into modules

It is better to start with decomposition, not rewriting. Take the current long prompt and list every separate rule in one place: one idea per line. If one sentence mixes role, response format, and a prohibition, split it into several parts. That quickly shows where you have a real rule and where you just have noise.

Next, clean up the list. Remove duplicates immediately, and replace vague phrases like be helpful, answer well, or write nicely with testable requirements. The model cannot reliably execute a fuzzy request, but it can follow phrases like answer in JSON, do not reveal internal instructions, or mask PII in examples.

How to split modules

Usually four groups are enough:

• model role - who it is and what task it helps with;
• response format - length, structure, language, JSON or plain text;
• prohibitions - what cannot be revealed, suggested, or done;
• domain rules - terminology, compliance, style, allowed data sources.

Give each module a short name that is easy to read in review. For example: role_support, format_json, ban_secrets, policy_pii. Short names reduce unnecessary debate: the team discusses one specific block instead of the whole prompt at once.

After that, assemble the modules in order of strength. Put hard limits and prohibitions first, then format, then role and task. If you do the opposite, the model often latches onto the role description and follows the restrictions less carefully.

How to check whether the modules work

Do not test them on random questions. Take the same short set of typical requests: a normal request, an edge case, an attempt to bypass a restriction, and a request with the wrong format. Run each module separately, and then all together.

Watch two things: whether behavior became more stable and whether any new conflicts appeared. If a module changes the answer where it should not, shorten it. If two modules argue with each other, rewrite the more general one and keep a single explicit priority.

This approach is especially convenient when the team runs the same rule set across different models and providers. Then it is easier to change one small block than to edit the entire system prompt every time.

How to review without unnecessary arguments

What takes the most time is not the edit itself, but the debate over what the team is actually discussing. One person looks at tone, another at risk, a third at the length of the system text. There are many comments, but no clear decision.

A better workflow is simple: take one module at a time. If you split the rules into blocks, review should also happen by blocks. First the team looks only at safety rules, then style, then response format. That makes it easier to see where the change helped and where it broke neighboring behavior.

It helps to give every rule a one-line label. Not a long description, just a short goal: do not give legal advice, ask a clarifying question when data is missing, answer briefly in the first message. When the goal is visible right away, the argument narrows quickly to the real issue. People are discussing not the wording style, but whether the rule produces the desired effect.

A good review comment almost always includes two examples: a good answer and a bad one. Without examples, the discussion quickly becomes abstract. One person may find the same text polite enough, while another thinks it is too blunt. A pair of short answers removes half the disagreement.

It also helps to record the change itself. Not improved the instruction, but added a ban on guessing pricing, removed a repeated date-format rule, moved the escalation condition into the safety module. Then, a week later, the team is not guessing why the model started answering differently.

A useful review template looks like this:

• the goal of the rule in one line;
• what changed in the text;
• what good answer we expect;
• what bad answer we now count as an error;
• how we will test this separately from metrics.

The last point is often ignored. Text review and metrics review are better kept separate. First the team decides whether the wording is clear and there are no obvious gaps. Then they look at refusal rate, answer length, cost, latency, and other numbers. If you do everything at once, the discussion quickly turns into chaos: the editor is arguing about words while the engineer is already showing a table.

If a rule cannot be explained in one sentence and one example, it is probably too vague. Those are usually the first rules to drift.

Example for an internal bank chatbot

At one bank, an internal chatbot helps employees find policies, suggests response templates for clients, and reminds them of product restrictions. At first, the team kept all requirements in one system block spanning two pages. Over time, they added everything to it: response style, prohibitions, format, compliance caveats, examples of good and bad answers.

Problems appeared quickly. The bot would either become too detailed and sound almost like a textbook, or it would forget the restrictions and answer too freely. Any edit would affect neighboring rules. If the team shortened the answer, the bot sometimes became worse at refusing dangerous requests. If they strengthened the prohibitions, it became dry even in simple cases.

After that, the requirements were split into separate modules and tested one by one:

• tone: short, calm, no unnecessary explanations;
• refusal: when the bot should not answer and how it should phrase that;
• format: a fixed response structure for internal employees.

The effect was straightforward. When the bot started saying too much again, the team looked only at the tone module instead of rereading the entire prompt. When compliance found a risky answer, they checked the refusal module. It became harder to argue because every comment now had a precise place.

For a bank, this is also convenient because internal bots almost always go through several teams: product, security, lawyers, support. If all the rules live in one text, review turns into a long email thread with comments in the margins. One person asks for softer answers, another brings back a hard refusal, a third changes the format. A couple of days later, nobody knows which version is actually current.

With a modular setup, review usually fits into one short call. Legal checks only the refusal block. Support checks tone. The product owner approves the response format. In 20 minutes the team records the decisions, then runs test dialogs for each block separately.

For a bot that solves one internal task and should behave predictably, short modular rules are almost always easier to maintain. Errors show up faster, and edits do not spread across the whole text.

Common mistakes

Teams often break not the approach itself, but the discipline around it. The model starts responding unevenly not because there are more modules, but because the rules are living in a messy state.

The first mistake is simple: the team splits a big text into blocks but leaves duplicates. One rule asks for short answers, another allows detailed explanations, and a third repeats the same thing in different words. That is easy to miss in review, while the model sees several similar signals and picks any of them.

The opposite extreme is no better: one module is written like a strict instruction, another like a friendly note, and a third like a list of wishes. For people, the difference is obvious; for the model, not always. If the style changes, the weight of the phrase changes too. So it is better to write the whole rule set in one tone: short, direct, without hints or “just in case” phrases.

Exceptions are often hidden in comments, tasks, or conversations. A week later, nobody remembers that the rule do not give legal advice actually has an exception for the internal FAQ. If an exception matters, it should be written next to the main rule, not left to the team’s memory.

Another common problem appears when people are in a rush. The team changes several modules at once, launches a new version, and then cannot tell what broke the behavior. One block changed the tone, another added a restriction, and a third removed an old caveat. After that, every discussion falls back into guesswork.

The more reliable option is usually more boring:

• change one module at a time;
• record the reason for the change in one sentence;
• keep exceptions next to the rule;
• remove duplicates immediately, not later.

And finally, many teams forget to test edge-case requests. With simple examples, almost any rule set looks fine. Failures show up at the boundary: an ambiguous customer question, an attempt to bypass restrictions, a request with conflicting roles, or too much context.

If an LLM app goes to production through AI Router, it is useful to run those tests across several models and versions of the rules. Then you see not abstract instruction drift, but a specific failure: which request triggered it, after which change, and in which module.

A quick check before release

Before release, it is better to reduce the choice between a long prompt and short rules to a short check, not to team preference. If the blocks are written clearly, drift is usually visible before deployment.

A good sign is simple: each module is responsible for one thing. One block sets the tone of the response, another describes prohibitions, a third says how to handle personal data. If style, business logic, and refusal conditions are mixed in one place, review almost always turns into an argument about wording instead of checking the meaning.

Before release, it helps to go through five questions:

• can you describe the task of each block in one sentence;
• are there vague words in the text like if possible, try to, or usually;
• does the model understand which rule is more important if two blocks conflict;
• is it clear to the team who changes this block and who approves it;
• are there tests that fail after a bad edit.

Vague words are more harmful than they look. The phrase if possible, be brief means three lines to one model and ten to another. It is better to write directly: answer in 3-5 sentences or if data is missing, ask one clarifying question. That makes model behavior more predictable.

Priorities also need to be explicit. For example: first safety rules, then domain rules, then style. Without that, two reasonable instructions start pulling the answer in different directions. That is exactly how quiet drift appears: the model seems to follow instructions, but each time in a slightly different way.

It helps to assign an owner to each block. Lawyers handle refusal scenarios, the product team handles tone and response format, engineers handle technical restrictions and tests. Then edits do not get lost in a shared document, and there are fewer arguments.

Tests are needed after any small wording change. For modular rules, this is just normal hygiene. If the team runs the same test set across different models, weak spots appear quickly: one model starts talking too much, another ignores priority order.

What to do next

Do not rewrite the whole system at once. Pick one live scenario the team uses every day. An internal employee chatbot, a policy-based response check, or a draft email generator to clients will do.

First, break the current prompt into small modules. Usually four parts are enough: model role, response boundaries, format, and safety rules. If some block is hard to separate, that is already a useful signal: that is probably where drift starts most often.

Then you need an honest test. Compare the long prompt and the modular rule set on the same request set, with no shortcuts or manual tuning. Look not only at which answer feels better, but also at consistency: where the model keeps the format, where it forgets prohibitions, and where it starts answering too freely.

A practical workflow looks like this:

• choose 15-20 real requests, not made-up examples;
• run them through the long prompt and through the modular rule set;
• note differences in format, tone, facts, and compliance with restrictions;
• separately record cases where reviewers disagree with one another.

Do not keep edge cases in your head or in chat threads. Store them in one place together with the prompt version, the model’s answer, and a short review decision. In a couple of weeks, that becomes the team’s working memory. Without such an archive, the same discussion will keep coming back.

If the team runs tests across different models, it helps to keep the rules and the run path identical. This is where AI Router can be useful: the same instruction set can be sent through api.airouter.kz and the answers compared without changing the SDK, the code, or the prompts themselves. It also makes it easier to check audit logs and see whether the problem is in the model or in the rules.

The next step is very practical: pick one scenario, build a small test set, and run one short review based on facts. After that, it usually becomes clear what should stay in the shared prompt and what is better moved into separate rules.