Building Nepali-language AI: test the language your users actually use

A Nepali-language application should be tested on more than a few well-formed Devanagari sentences. Real messages may mix Nepali and English, use Romanized spelling, include Nepali digits or refer to a local place without explaining it. Instead of asking whether a model “supports Nepali,” define the exact tasks and language patterns your product needs to handle. Then evaluate those tasks with people who understand the context.

Collect a small, representative language set

For an illustrative shop assistant, include “यो सामान उपलब्ध छ?”, “yo saman available cha?” and “Is the blue one available?” Add messages with spelling variation, omitted punctuation and product names in English. These are proposed test inputs, not the results of a model benchmark.

Ask a fluent reviewer to record the intended meaning and acceptable behavior for each case. A product-availability question may require a stock lookup; a request about an unspecified “blue one” should trigger clarification. Fluency alone is not enough if the system confidently answers the wrong question.

Normalize carefully and preserve the original

Unicode allows some visually similar text to have different underlying representations. NFC normalization can make canonically equivalent text consistent. It does not translate Romanized Nepali, fix every spelling difference or establish that two messages mean the same thing.

Keep the original input for display and review. Use a normalized copy for a clearly defined processing step, and test how your search system treats it. Avoid stripping all non-ASCII characters: that would remove the script your users are trying to use.

import unicodedata

original = "यो सामान उपलब्ध छ?"
normalized = unicodedata.normalize("NFC", original)

# Convert a known digit set only when the field requires digits.
nepali_to_ascii = str.maketrans("०१२३४५६७८९", "0123456789")
quantity_text = "१२"
quantity = int(quantity_text.translate(nepali_to_ascii))

assert quantity == 12
assert original == "यो सामान उपलब्ध छ?"
print(normalized, quantity)

Reference: Unicode Standard — Normalization forms ↗ · Python documentation — unicodedata ↗

Keep script, meaning and facts separate

An application may need a script conversion step, a semantic search step and a factual lookup. Treat them as separate responsibilities. If someone writes a Romanized place name, do not silently assume a single location when several records match.

The same principle applies to dates and quantities. Preserve whether a date came from a Bikram Sambat or Gregorian field. Use a tested conversion library or service when conversion is required, and show the calendar explicitly. A language model’s fluent explanation is not a date-validation routine.

Design retrieval around actual queries

Create paired Devanagari and Romanized questions that should reach the same approved document. Also add near misses: a similar product name, a location with a similar spelling and a query that has no supporting document. Record retrieval failures independently from answer-writing failures.

If you introduce multilingual embeddings or transliteration, compare them against a keyword baseline on the same data. Do not assume a model’s advertised language coverage establishes performance on your catalog, vocabulary or informal chat messages.

Use benchmarks as context, then test your product

The Belebele dataset includes multilingual reading-comprehension material, including Nepali. It can help you understand one kind of evaluation, but reading comprehension is not the same task as answering a live stock question or interpreting Romanized customer messages.

Build a product-specific report with separate results for script, task and failure type. Review whether answers preserve names, numbers and uncertainty. Before release, let fluent users try the workflow and explain where an answer sounds unnatural or changes the intended meaning.

Reference: Meta — Belebele multilingual reading comprehension dataset ↗