For the makers, artists, and builders of the future, the barrier to creating truly inclusive AI is no longer just code-it is the language in which that code is discussed. While English dominates the global conversation, the reality of open-source collaboration is far more diverse. To help creators build tools that actually work for non-English speaking developers, GitHub has released the GitHub Multilingual Repositories Dataset. This open resource allows you to locate public repositories containing natural-language content in languages other than English, ensuring your next AI model understands the full spectrum of human instruction.
The release addresses a critical gap: language distribution varies wildly depending on where developers communicate. Analysis of the data reveals that Korean is the dominant non-English language in issue threads, yet it ranks only fifth in README files. By contrast, Portuguese leads the README list with over 3 million repositories. This dataset is a direct outcome of a pledge made in 2025, as part of Microsoft’s European Digital Commitments, to democratise access to multilingual data for the open-source AI community.
What’s inside the dataset
This is not a raw dump of repository files. Instead, it is a structured metadata tool designed to help researchers and developers pinpoint where multilingual collaboration is occurring. The collection spans over 80 million classification rows across more than 40 million repositories. For each public repository, the dataset provides:
- Language classifications for the README, the most-commented issue, and the most-commented pull request. The input sample consists of the first 150 characters, excluding any text under 20 characters.
- Classifications from three distinct sources: fastText, gcld3, and lingua-py. Each entry includes a confidence score, and the dataset only retains classifications with a score greater than 0.5.
- Repository metadata including creation timestamp, disk usage, stars, forks, primary programming language, SPDX license, and counts for issues and pull requests.
We have deliberately avoided collapsing these three classifiers into a single label. Different models have varying coverage and confidence calibration, particularly for lower-resource languages. By exposing all three, you retain the flexibility to define your own precision. Need a high-precision Greek subset? Require all three classifiers to agree. Looking for broad recall to study Romance languages? One classifier may suffice.
Practical applications for builders
This dataset is tailored for tasks that general web text cannot solve:
- Discover repositories likely to contain developer documentation or collaboration in specific languages.
- Study how non-English communities utilise issues, pull requests, and READMEs.
- Build evaluation sets for AI coding tools, documentation generators, or review assistants that must perform reliably across languages.
- Encourage decision-makers to expand language coverage for new developer tools and AI features using data-backed arguments on the rich multilingual diversity of developers.
- Measure the representation of European and other underrepresented languages in open source.
Important caveats
Language identification in software repositories is notoriously difficult. Repository text is often short and noisy, containing badges, templates, installation commands, code snippets, usernames, or mixed-language content. A 150-character sample may not represent the whole repository. Furthermore, classifiers vary in coverage and calibration, especially for lower-resource languages.
Consequently, this dataset should not be treated as a ground-truth benchmark for language identification. It is designed as a transparent discovery tool. Users can inspect classifications, confidence scores, and sources to choose the precision and recall trade-offs that fit their specific workflow.
Additionally, the dataset must not be used to infer sensitive attributes about repository owners, contributors, or communities. The signals are repository-level metadata, not person-level attributes.
Why open multilingual data matters
Currently, many European languages remain underrepresented in the online text used to train and evaluate AI systems. This creates a risk that AI tools function well for some developers while leaving others behind. Open data can help bridge that gap. We built this dataset because developer content is distinct from general web text. READMEs, issues, and pull requests contain the specific language of software collaboration: installation instructions, bug reports, feature requests, review comments, and community norms. That context is essential for building AI systems that truly understand how developers work.
By making multilingual developer-content signals easier to find and analyse, this dataset gives researchers, open source developers, and model builders another tool for studying language representation in software development. It can help identify gaps, support better evaluation, and inform more inclusive AI tools for developers across Europe and beyond. It also reflects a broader principle: Building AI for developers must include the communities, languages, and workflows developers actually use.
What’s next
We will be discussing the dataset and the broader importance of open data for multilingual AI at the Open Innovation Dialogue Hub in Strasbourg on June 16. The event is co-organised by the Microsoft Open Innovation Center, the Council of Europe, and GitHub. It will bring together policymakers, researchers, cultural institutions, and open innovation leaders to discuss AI, linguistic diversity, cultural heritage, and open data.
Multilingual AI requires multilingual developer communities. We hope this dataset helps more people study, support, and build for them. By releasing it under CC0-1.0 on GitHub, we are inviting researchers, open source maintainers, and model builders to use it, critique it, extend it, and build evaluation sets and tools on top of it.
Key takeaways
- The GitHub Multilingual Repositories Dataset provides metadata for over 40 million repositories, highlighting where non-English collaboration occurs in READMEs, issues, and pull requests.
- Language distribution varies significantly by context; for instance, Korean dominates issues but ranks lower in READMEs, while Portuguese leads the latter.
- The dataset exposes three distinct classifiers to allow users to balance precision and recall based on their specific research or development needs.
- Released under CC0-1.0, this open resource aims to close the gap for underrepresented European languages in AI training data.
