RLHF preference labeling and human annotation in Indian languages

A managed annotation team for model builders who need human judgement in Indian languages, not machine-translated English guidelines.

What's included

• Pairwise and ranked preference judgements on model responses, with written rationales
• Response rating against your rubric for helpfulness, accuracy, safety and tone
• Text classification, named-entity tagging and transcription review
• Guideline localisation and a calibration round before production starts
• Per-batch quality report with gold-question accuracy and inter-annotator agreement

How buyers use it

• Reward-model and DPO training data for Indic and multilingual assistants
• Safety red-teaming and evaluation in regional languages
• Labelled sets for intent detection and entity extraction in customer-service bots
• Human evaluation of machine translation and summarisation

Coverage and methodology

Kathana employs 1,800 trained annotators, including graduates in law, medicine and engineering for expert tasks, and works inside your annotation tool or its own. Each project starts with a five-day pilot of up to 2,000 items. Production batches include hidden gold questions, 10% double annotation and senior review of disagreements; typical agreement reaches a Cohen's kappa of 0.7 or higher on preference tasks. Prompts and outputs you send stay in an isolated workspace, are not reused for other clients and are deleted at project close. You own the resulting labels and may use them for model training and commercial products.