Reranking Russian-Kazakh Search with Query Pairs

Search across Russian and Kazakh documents does not fail only when the system completely misunderstands the language. It fails earlier: the right document appears among the candidates, but the reranker places higher a text with matching words, the wrong procedure, or a convenient Russian paraphrase. An average metric across the whole corpus almost always smooths over this failure.

Testing this kind of search with a set of parallel translations is not enough. You need a set in which the judge answers a different question: does this document match the query's intent even when the query and document are written in different languages, and terminology, abbreviations, or part of the phrase shift from one language to the other? In production, this matters more than a model's ability to match two carefully translated news articles.

The test set should measure ordering, not translation similarity

Russian-Kazakh search reranking should be evaluated on small result sets containing one or more genuinely useful documents and several dangerously similar competitors. A pair consisting of a Russian query and its Kazakh translation measures semantic similarity. A pair consisting of a Russian query and a Kazakh document that solves the user's task measures search. These are different tests.

For example, a user might write: «how do I request unpaid leave». A Kazakh document may contain the required procedure, conditions, application process, and exceptions. Nearby there should be plausible but incorrect documents about paid annual leave, sick leave, and business trips. If you place random materials next to it, almost any modern reranker will look good because the task has become too easy.

Keep the distinction between three properties clear:

• Translation equivalence. Two fragments say nearly the same thing in different languages.
• Search relevance. A document helps the user fulfill the query's intent, even when the wording differs substantially.
• Ranking preference. Among several useful texts, the system puts first the one that answers the question more precisely and does not lead the user toward a wrong action.

Teams often mix these properties together. Then they get a high score on bilingual parallel sentences and wonder why their RAG system answers from an outdated version of a policy. The model may match «өтініш» and «application» well but fail to distinguish an application for leave from an application to change personal data.

In the MuSeCLIR work, the authors specifically point out the weakness of sets dominated by named entities: these can often be handled through simple transliteration, so the test barely requires resolving ambiguity. This is especially problematic for a Russian-Kazakh set. Organization names, order numbers, and English abbreviations give the model easy points.

First define what «relevant» means in your domain

Annotators should judge not the quality of the translation or the document's general topic, but whether it lets the user solve the original task. Write this rule down before collecting pairs. Otherwise, one judge will mark a document relevant because it is «about leave», while another will reject it because it concerns a different type of leave.

Create an intent card for every query. Five fields are enough:

1. What the user wants to do or find out.
2. Which fact, condition, or action the answer must contain.
3. Which documents count as a complete solution.
4. Which related documents do not count as a complete solution.
5. What language form of the query is expected in real traffic.

The card for «қайтарымды қалай рәсімдеймін» should not say «a document about returns». That is too vague. Be specific: the user wants to return a product and needs the deadline, contact channel, and return conditions. A returns policy is fully relevant. A page about warranty repair is partially relevant. A text about refunding payment for a service is irrelevant if it describes a different process.

This contract removes a common mistake: using any document containing the desired word as the reference. In corporate knowledge bases, this often happens with words such as «өтініш», «келісім», «төлем», «жоба», and «есеп». A word may refer to the right entity while the document describes a different route, status, or department.

For legal, medical, and financial materials, add a currentness flag to the card. An old document may be well written, accurately translated, and topically exact, but it must not be ranked above the current version. Ranking without version awareness creates an operational defect rather than a language problem, and it is especially easy to miss in bilingual testing.

A language matrix exposes the imbalance hidden by the average

The minimum matrix has four directions: a Russian query against a Russian document, a Russian query against a Kazakh document, a Kazakh query against a Kazakh document, and a Kazakh query against a Russian document. All four directions belong in the same set and should have comparable difficulty.

If you keep only two cross-language directions, you will not know whether the new model has become worse at same-language search. If you keep only Russian queries, you measure the convenience of a Russian-speaking team rather than real quality for Kazakh-speaking users.

On top of this matrix, add two separate slices:

• Mixed query: «eGov анықтамасын скачать», «жеке кабинетте пароль ауыстыру», «договорға қол қою тәртібі».
• Mixed document: a Russian main section with Kazakh fields, a bilingual table, or a policy whose section names use both languages.

Mixed text cannot be reduced to a «fifth language». What matters is the boundary inside the phrase. A user may state the action in Kazakh but leave the form name in Russian because that is how it appears in the interface. A document may contain a legal condition in one language and instructions for the employee in another. A model that identifies the language of the entire line and then selects one processing path often loses exactly these cases.

Do not make the pairs mechanical mirror images. If every Russian query has only a literal Kazakh translation, the system will quickly learn the pattern. Instead, collect equivalent intents in different natural forms. For the same action, these might be «where can I get a certificate», «справка қайдан алынады», «анықтаманы жүктеу керек», and «личный кабинеттен анықтама таба алмай отырмын». They do not have to be mutual translations, but they must lead to the same set of relevant materials.

Research on Arabic-English RAG showed that switching the language between a query and its supporting document causes a clear drop specifically at the ranking stage, not only during answer generation. The authors also tested every combination of query and document languages rather than one convenient cross-language diagonal. The same design is worth repeating for Russian-Kazakh evaluation, even if the domain and language pair differ.

Collect documents around the user's task, not around the language

Build the source corpus around practical tasks: returns, access, HR procedures, tariffs, application requirements, payment statuses, internal instructions, and support answers. Then choose materials in Russian and Kazakh. If you start with the languages, you will get two tidy shelves of documents that rarely compete for the same query.

A good evaluation unit consists of a query, a candidate pool, and a label for every candidate. Do not store only one positive pair. One positive document does not test ordering because the model has nothing to choose between.

A practical JSONL format might look like this:

{
  "query_id": "hr_014_kk_ru",
  "query": "демалысқа өтінішті қайда жіберемін",
  "query_language": "kk",
  "intent": "leave_request_submission",
  "candidates": [
    {
      "doc_id": "hr_leave_request_ru_v3",
      "document_language": "ru",
      "text": "Заявление на отпуск сотрудник направляет через кадровый портал...",
      "grade": 3,
      "reason": "Описывает нужный канал подачи заявления"
    },
    {
      "doc_id": "hr_leave_policy_kk_old",
      "document_language": "kk",
      "text": "Еңбек демалысының жалпы тәртібі...",
      "grade": 1,
      "reason": "Тема совпадает, но нет способа подачи и редакция устарела"
    },
    {
      "doc_id": "hr_sick_leave_ru",
      "document_language": "ru",
      "text": "Лист нетрудоспособности передается руководителю...",
      "grade": 0,
      "reason": "Другой тип отсутствия"
    }
  ]
}

The reason field is not decorative. It turns evaluation from a table of numbers into material for error analysis. A month later, you can find every case where the model confuses «a document about a general policy» with «a document describing a specific action», instead of rereading hundreds of rows by hand.

Segment documents the way people read them. A ten-page policy often contains the needed rule as well as five unrelated procedures. If your real product ranks passages, label passages. If users open entire documents, add a second document-level assessment. You cannot fairly compare a passage reranker with a reference set in which the entire file is marked positive.

Hard negatives determine the value of the test

The most useful negative documents are so similar to the correct text that a person hesitates at first. You do not need to search for them artificially in unrelated topics. They are already next to the correct materials: a neighboring procedure, a different application status, another service recipient, an outdated version, or an exception to the rule.

For each query, try to include several types of competitors, but do not copy the same template throughout the set:

• The same object but a different action: cancellation instead of submission.
• The same action but a different subject: for an employee instead of a customer.
• The right term with the wrong condition: a different deadline, limit, or contact channel.
• A document in another language that looks like a translation but describes a related procedure.
• A document containing a valid old rule that has been replaced by newer text.

A completely off-topic negative is useful only as a basic sanity check. It shows whether everything has broken at once. It says almost nothing about reranking quality.

The most dangerous case is a false friend in bilingual documentation. Suppose the Russian document «refund of funds» concerns a discontinued service, while the Kazakh fragment «тауарды қайтару» concerns returning a product. A translator may bring them close together, and a reranker with shallow semantics may place them next to each other. Your set should contain such a pair because it reveals whether the model understands the object and the action together.

Do not use a machine translation of a positive document with one word replaced as your only negative example. It is useful for an isolated sensitivity test, but quickly becomes a synthetic pattern. The reranker will learn to win on a generation artifact rather than on an understanding of domain language.

A 0-3 scale is better than binary «found or not found»

A four-level scale provides enough distinction for search without turning annotation into a dissertation. I usually define it as follows:

• 3 - the document lets the user complete the task without another primary source.
• 2 - the document is useful but does not cover an important condition or action.
• 1 - the topic is related to the query, but the reader cannot solve the task from this text.
• 0 - the document does not match the intent or leads to a wrong action.

Grade 2 prevents the system from being penalized for offering a reasonable alternative while keeping that alternative below a direct answer. Grade 1 separates topical overlap from real usefulness. Without it, you get an argument over «relevant or not» where both annotators are right in different senses.

Two independent judges should first label a small shared batch and then discuss disagreements using the intent cards. Do not try to achieve identical taste. Aim for an identical rule. If one person gives 3 to a general policy description while another gives it 1, the problem usually lies in the definition of a complete answer, not in their language skills.

Do not hide the document language from the annotator if users see the original in the product. User usefulness includes the ability to read the result. But do not let language preference automatically lower relevance. A document in another language may be the ideal source, especially if the interface can show a translation or brief explanation.

The work by Parton, Habash, and McKeown on cross-language search clearly shows why this distinction matters. The authors compared the relevance of original and translated results and found a drop in average precision caused by machine translation errors. In other words, translating a result can distort the user's assessment even after a successful search.

First separate candidate loss from reranking error

A reranker cannot fix a document that is absent from its input list. For every query, therefore, store two results: whether a grade-3 document appears among the first-stage candidates and where it lands after reranking.

The check is straightforward. For the same query set, save the IDs of the top 100, or whatever the actual pre-reranking limit is. Then calculate Recall@K for grade-3 documents. After that, calculate nDCG@10 and MRR on the reordered pool. If Recall@100 is low for the «Kazakh query - Russian document» direction, the problem is in embeddings, lexical search, query translation, or result merging. If Recall@100 is high but nDCG@10 is low, the reranker or its input format is at fault.

A minimal diagnostic script needs no special framework. It takes a list of grades in the actual result order:

import math

def dcg(grades, k=10):
    return sum(
        (2 ** grade - 1) / math.log2(rank + 2)
        for rank, grade in enumerate(grades[:k])
    )

def ndcg(grades, k=10):
    ideal = sorted(grades, reverse=True)
    denom = dcg(ideal, k)
    return 0.0 if denom == 0 else dcg(grades, k) / denom

def reciprocal_rank(grades):
    for rank, grade in enumerate(grades, start=1):
        if grade == 3:
            return 1 / rank
    return 0.0

observed = [1, 0, 3, 2, 0, 0]
print(f"nDCG@10={ndcg(observed):.3f}")
print(f"MRR@all={reciprocal_rank(observed):.3f}")

For this example, the output will have the following form:

nDCG@10=0.590
MRR@all=0.333

Do not treat MRR as the only metric. It is useful when a query has one clearly best answer. But if two grade-3 documents complement each other or several good results matter, MRR stops describing the user experience. In those cases, nDCG@10 is more honest because it distinguishes positions and relevance grades.

Do not combine metrics into one «overall score» before examining the slices. The report should show the direction matrix, mixed texts, domains, short and long queries, and cases involving outdated documents separately. Average nDCG may rise because the model improved on easy Russian queries while declining on rare but critical Kazakh procedures.

Query translation is useful as a baseline but dangerous as the only path

The strategy «translate the query into the other language and run ordinary search» is popular for an obvious reason: it quickly uses an existing monolingual index. Keep it in the experiment as a baseline. But do not call it the reference approach until you have compared it on your difficult pairs.

In research by Saleh and Pecina on medical cross-language search, query translation outperformed document translation overall under their conditions. That is a good reason to test both strategies fairly, not a reason to choose one universally. A medical corpus, its terminology, and its translation quality are not the same as those of Kazakh corporate documents.

For every query, run at least three variants:

1. Direct search and reranking on the original query.
2. Search using a translation of the query into the other language.
3. Merge candidates from both runs and apply one reranker afterward.

Compare more than overall nDCG. Take queries where the strategies disagree and read the top five results. The cause usually becomes clear quickly: the translation lost an abbreviation, replaced a legal term with a conversational word, chose the wrong sense of an ambiguous word, or erased the language mixing.

The query «қызмет көрсету актісін жіберу» should not be blindly turned into «send the act». In different organizations, «акт» may mean a work completion certificate, an acceptance certificate, a reconciliation statement, or an internal confirmation. If one variant is fixed in the English field name and Russian documentation while the Kazakh query is broader, translation increases the risk of ranking the wrong procedure.

The reranker's input should contain exactly the context the user sees

Feed the model the same data on which you expect it to make decisions. If production gives it a title, passage, and version metadata, do not evaluate it on the cleaned full document. If users can access only a passage in the results, do not grade that passage using an important paragraph that falls outside the chunk.

A good input usually includes the title, the main passage text, language, document type, and date or version when these affect the choice. But metadata must not act as a hidden answer. is_current=true is acceptable if the live search uses it too. A manual «correct document» label inside the input is obviously not acceptable.

Also check the order and length of fields. If you concatenate text like this:

[ru] Заголовок: Порядок возврата товара
Тип: Регламент
Текст: ...

apply the same format to Russian and Kazakh text. A frequent error occurs when Russian documents have detailed titles while Kazakh documents enter the index under a technical file name. The model then ranks content preparation quality rather than languages.

The set should include documents where the answer is not in the first sentence. Otherwise, you are testing the ability to compare titles. That is useful for FAQ search, but it does not replace testing policies, contracts, instructions, and knowledge bases with dense conditions.

A run without an error log gives you nothing to improve

After every model comparison, record not only metrics but also the first wrong results for each slice. For each error, it is enough to record the query, query language, incorrectly positioned document, document that should have ranked higher, grades, error type, and a short decision.

Keep error types stable: missed condition, wrong object, wrong action, ambiguous term, old version, lost mixed-language fragment, weak candidate retrieval, and poor segmentation. After several iterations, you will see where to spend time. If half of the errors come from missing candidates, changing the reranker's prompt for weeks is pointless.

Do not train or tune the system on one test set alone. Keep a private query batch that the team does not inspect during tuning. The open set is for diagnosis; the private set shows whether you have learned to fix only known formulations. This is especially important when an LLM is involved in generating queries, synthetic negatives, or error analyses.

If your infrastructure already uses a unified OpenAI-compatible gateway, you can run this as a separate offline task: save the model version, input template, candidate list, and final output to an audit log. In AI Router, you can use unified API access to different models for this purpose without changing client code when comparing options.

A test set does not prove that you «support two languages». It shows something more practical: when the wording is Russian, Kazakh, or mixed, the system raises the document that lets a person take the needed action. Until the report makes this distinction, a high average score remains a convenient number rather than a reason to trust the search.