Indic and English text corpus for LLM pre-training and fine-tuning

Clean, licensed text for teams building language models that need to work well in Indian languages, where web-crawled data is thin and noisy.

What's included

• 62 billion tokens across 48 million documents in 22 Indian languages and Indian English
• Domains: news, government and legal documents, educational material, literature and encyclopaedic text
• A licence tag, source type and publication year on every document, plus a chain-of-title register for audit
• A quality score from a per-language classifier, toxicity flags and near-duplicate cluster IDs
• Optional instruction-tuning subsets: 1.8 million human-written prompt and response pairs

How buyers use it

• Continued pre-training of foundation models to improve Indic language performance
• Fine-tuning domain models for legal, government and education use
• Building tokenizers and evaluation benchmarks for Indian scripts
• Academic research on low-resource language modelling

Coverage and methodology

Text comes from publishers that have signed AI-training licences with Kathana, from government open-data portals under open licences, and from public-domain books that Kathana digitises with OCR and human correction. Kathana does not include text scraped from sites that prohibit it. Every document passes MinHash deduplication, language identification, PII scrubbing for phone numbers, email addresses and ID numbers, and classifier-based quality filtering. Character accuracy of corrected OCR is above 97% on audited pages. New material is added quarterly, and the licence covers model training and commercial deployment of the resulting models.