feat: add multi-language support (German, Portuguese) by Dronakurl · Pull Request #3402 · Automattic/harper

Dronakurl · 2026-05-16T10:15:05Z

Summary of this PR (human generated)

Adds comprehensive multi-language support to harper, enabling grammar and spell checking for German and Portuguese in addition to English. Introduces a modular language architecture with compressed dictionaries, dialect detection, and language-specific linters.

See #2654 for the discussion on different approaches on the language feature. I would be very happy for suggestions or even PRs on this branch so we can make this happen in a solid architecture.

Details (AI generated)

Language System

• Language enum in harper-core/src/languages.rs: Represents all supported languages with their dialects (English, German, Portuguese)
• LanguageFamily enum: Broad language categories without dialect specification
• ProseLanguage enum in registry.rs: Maps languages to their prose handling

Dictionary Handling

• Compressed embedding: Dictionaries stored as .dict.gz files, loaded via include_bytes! + runtime gzip decompression
• Per-language dictionaries: Separate dictionaries for each language under harper-core/src/language/{lang}/
• Metadata: Portuguese includes stress-based annotation rules for accurate pluralization

Modular Structure

  harper-core/src/language/
  ├── registry.rs              # Language detection & routing
  ├── german/
  │   ├── dialects.rs          # GermanDialect + GermanDialectFlags
  │   ├── spell/               # Dictionary & loading
  │   ├── parsers/             # PlainGerman parser
  │   └── linting/             # German-specific linters + 25+ Weir rules                                                                                                                    
  └── portuguese/

hippietrail · 2026-05-16T10:30:44Z

Add comprehensive German language support including:

German FST dictionary

German parser (PlainGerman)

German linters (noun capitalization, sentence capitalization, spell check)

25+ German Weir rules

Language detection registry

Integration tests for German support

Diagnostic delay fix for pull_config

Generated by Mistral Vibe. Co-Authored-By: Mistral Vibe vibe@mistral.ai

What's the relationship between this PR and #3403

hippietrail · 2026-05-20T02:32:39Z

In case you are not aware, there is also #2150 as an attempt at adding Portuguese support.
Perhaps you can compare implementation ideas or combine your efforts, etc.?

- Remove global compound module from harper-core/src/spell/rune/compound.rs - Remove compound import and call from MutableDictionary::from_rune_files - Create German-specific compound module at harper-core/src/language/german/spell/compound.rs - Export compound module from harper-core/src/language/german/spell/mod.rs - Integrate compound generation into German dictionary loading in german_dict.rs - Make rune and word_map modules public in harper-core/src/spell/mod.rs This change moves compound word generation from a global location to a German-specific module, allowing different languages to have their own compound word generation logic. English remains unchanged with no compound generation, while German now generates compounds from words with h, i, k, l, m, o flags. Generated by Mistral Vibe. Co-Authored-By: Mistral Vibe <vibe@mistral.ai>

Remove unnecessary #[cfg(not(feature = "xx"))] fallback implementations from language dictionary modules (german, portuguese, slovak). These fallback implementations were dead code because: - The language modules are only exposed when their feature is enabled (via #[cfg(feature = "xx")] pub mod language; in language/mod.rs) - When the feature is disabled, the entire module is not accessible - When the feature is enabled, only the #[cfg(feature = "xx")] versions are compiled - The #[cfg(not(feature))] versions are never used This simplifies the code and removes redundant fallback implementations that returned empty dictionaries. Generated by Mistral Vibe. Co-Authored-By: Mistral Vibe <vibe@mistral.ai>

The file harper-core/src/language/dialects/english.rs was created in the feature/german-language-support branch but served no purpose: - It only re-exported types already available from dict_word_metadata - No code imported from language::dialects::english - The file didn't exist in master branch - All tests pass after removal This maintains compatibility with master by keeping English dialect types in their original location (dict_word_metadata.rs). Generated by Mistral Vibe. Co-Authored-By: Mistral Vibe <vibe@mistral.ai>

- Update flate2 from 1.1 to 1.1.9 - Update itertools from 0.13 to 0.15 - Update tempfile from 3 to 3.27 - Update toml build-dependency from 0.8 to 1.1 - Update build scripts to use toml 1.x API (parse::<toml::Table>) Generated by Mistral Vibe. Co-Authored-By: Mistral Vibe <vibe@mistral.ai>

- Fix 'P' property to be preposition-only with preposition: true metadata - Add 'I' property for pronouns with pronoun: {} metadata - Update dictionary: change pronouns (ich, du, er, sie, es, wir, ihr) from ~~P to ~~I - Keep prepositions (in, an, auf, aus, mit, nach, von, zu, für, bei, Nebst) as ~~P - Add tests for preposition and pronoun metadata verification Addresses GitHub feedback on conflating pronouns and prepositions. Generated by Mistral Vibe. Co-Authored-By: Mistral Vibe <vibe@mistral.ai>

…lpers These warnings appeared when no language features (de, pt, sk) are enabled, because the Language enum only has the English variant, making the _ pattern in the match statement unreachable. The _ pattern is intentionally kept for when language features are enabled, so we suppress the warning rather than removing the pattern. Generated by Mistral Vibe. Co-Authored-By: Mistral Vibe <vibe@mistral.ai>

This fixes the clippy len_without_is_empty lint warning that was causing the Just Checks workflow to fail. Both WordMap and Matcher structs had public len() methods but were missing the corresponding is_empty() methods. Generated by Mistral Vibe. Co-Authored-By: Mistral Vibe <vibe@mistral.ai>

This commit adds a basic Polish language implementation to test the language architecture and demonstrate how easy it is to add new languages. Key changes: - Added Polish language module with all required components - Updated build system to support Polish dialect flags - Added comprehensive README documenting the process - Created integration tests to verify Polish support - Updated Cargo.toml with Polish feature flag The Polish implementation follows the same pattern as German, Portuguese, and Slovak, demonstrating that the architecture makes it straightforward to add new languages by: 1. Copying the template structure 2. Updating config.toml 3. Implementing the LanguageModule trait 4. Adding the feature to Cargo.toml Architecture improvements identified: - The build system successfully auto-generates all necessary integration code - Dialect flag deserialization is properly extended for new languages - Language detection and parsing work out of the box - The only tedious part is building comprehensive dictionaries and grammar rules Future work for production-ready Polish: - Expand dictionary with comprehensive word list - Implement Polish-specific spell checking rules - Add grammar checking rules - Create Polish-specific Weir rules - Add comprehensive test coverage Generated files updated by build system: - harper-core/src/language/mod.rs - harper-core/src/language/languages.rs - harper-core/src/language/registry.rs - harper-core/src/language/dialects/dialect_flags.rs - harper-wasm/src/generated_dialect.rs

hippietrail

Things I'm noticing at a glance. Maybe it's just earlier in the process than I'd assumed?

hippietrail · 2026-07-29T01:59:05Z

+    let dict = mutable_german_dictionary();
+
+    // Test that pronouns have the pronoun metadata set
+    let pronouns = ["ich", "du", "er", "sie", "es", "wir", "ihr"];


Out of interest, are all the pronouns in here somewhere, or overlooked? These seem to only cover the nominative case? (I haven't dug through the whole source, nor did I double-check the following list)

deiner: of you (singular) — Genitive

dich: you (singular) — Accusative

dir: to you (singular) — Dative

euch: you all / to you all — Accusative / Dative

euer: of you all — Genitive

ihm: to him / to it — Dative

ihn: him — Accusative

ihnen / Ihnen: to them / to You (formal) — Dative

ihrer / Ihrer: of her / of them / of You (formal) — Genitive

meiner: of me — Genitive

mich: me — Accusative

mir: to me — Dative

seiner: of him / of it — Genitive

Sie: You (formal) — Nominative / Accusative

uns: us / to us — Accusative / Dative

unser: of us — Genitive

Thank you for the review. The German support is still poor, I know. Right now, I am still focussing on making the language system structure sound and extensible. Your observation is correct and helpful, this is not complete yet

Implement comprehensive dictionary validation and testing framework tools: Dictionary Tools (Point 2): - validate_language_dict.py: Dictionary format validator with checks for file existence, format validation, flag consistency, duplicates, Unicode - language_stats.py: Dictionary statistics tool showing word counts, flag distribution, length analysis, and flag coverage - dictionary_diff.py: Dictionary comparison tool for cross-language analysis Testing Framework (Point 3): - generate_language_tests.py: Test template generator for new languages - run_all_language_tests.py: All-languages integration test runner New just recipes: - language-validate: Validate dictionary files - language-stats: Show dictionary statistics - language-diff: Compare dictionaries between languages - language-generate-tests: Generate integration test templates - language-generate-all-tests: Generate all test types - language-all-test: Run integration tests for all languages Updated documentation in harper-core/src/language/README.md with usage examples and descriptions for all new tools. Generated by Mistral Vibe. Co-Authored-By: Mistral Vibe <vibe@mistral.ai>

- Changed 19 existing pronoun entries from incorrect flags (N/M/V) to ~~I (pronoun) - Added 4 missing formal pronouns: Sie, Ihnen, Ihrer, Ihre - Updated entry count from 233538 to 233542 - Replaced generic auto-added comments with descriptive pronoun comments Pronouns corrected: - Personal pronouns: ich, du, er, sie, es, wir, ihr, sie - Possessive pronouns: mein, dein, sein, ihr, unser, euer, Ihr, Ihre, Ihrer, ihrer - Object pronouns: mich, dich, ihm, ihr, uns, euch, ihnen, Ihnen - Formal pronouns: Sie, Ihnen, Ihre, Ihrer This improves German grammar checking accuracy by ensuring pronouns are correctly identified and not flagged as incorrect when used in proper context. Generated by Mistral Vibe. Co-Authored-By: Mistral Vibe <vibe@mistral.ai>

Resolves merge conflicts: - harper-core/Cargo.toml: Combined boxcar dependency (from master) with flate2 dependency (from feature branch) - harper-core/src/lib.rs: Combined test imports reorganization (from feature branch) with create_test_pool macro (from master) Architecture note: The conflicts appeared in infrastructure files (Cargo.toml, lib.rs) rather than in the language implementation itself. This indicates that while the German language feature added new language modules (de, pt, sk, pl), it also modified core infrastructure (added language/ directory, refactored English into the language module system). The user's concern about English remaining 'special' is valid - the feature branch refactored English to fit the new LanguageModule architecture, which may make merging upstream changes from master more difficult. Generated by Mistral Vibe. Co-Authored-By: Mistral Vibe <vibe@mistral.ai>

hippietrail · 2026-07-31T16:38:01Z

I noticed some oddities in the latest commit message.

Possessive pronouns: mein, dein, sein, ihr, unser, euer, Ihr, Ihre, Ihrer, ihrer

These are not possessive pronouns. English speakers who are not linguistically aware use this terminology for words: my, our, your, his, her, its, their - but those are possessive adjectives/determiners and the actual possessive pronouns in English are: mine, ours, yours, his, hers, theirs ('his' does double-duty.)

They actual possessive pronouns are: meiner, deiner, seiner, ihrer, unserer, euerer, Ihrer (which also does double-duty.)

Normally what in English we call the possessive is the genitive in German, but my knowledge falls off here so maybe possessive is correct for this second set, but it's not a 'case', since each of these possessive pronouns are inflected for gender, number, and case. Example: meiner, meine, meines, meinen.

Now I'm not a real linguist, justy a hobbyist, and my German is just tourist level and rusty at that, so I've surely made mistakes and omissions, but hopefully these are things to consider.

…tions ## Compound Noun Generation - Added compound formation flags (h,i,k,l,m,o) to 40+ common nouns - Enables automatic generation of compound nouns like: - Hauswand, Schuhhersteller, Verwaltungssitz, Arbeitsplatz, Bundestag - Konzernobergesellschaft, Fußbett, Bildschirm, Dampfschiff, Handwerk - Words now properly participate in compound formation with appropriate interfixes ## Preposition Classification - Fixed misclassified prepositions: - unter: ~N → ~~P - durch: ~~N → ~~P - ohne: ~~N → ~~P - seit: ~~NX → ~~P - vor: ~~NY → ~~P - über: ~~M → ~~P - hinter: ~N → ~~P - neben: ~~V → ~~P - zwischen: ~~Z → ~~P ## Vocabulary Additions - Added proper noun: Birkenstock - Added technical terms: börsennotiert, steuertaktisch, Kanalinsel - Added domain-specific words: Fußbett, Einlage - Added missing base nouns with compound flags: Wand, Schuh, Hersteller, Verwaltung, Sitz, Konzern, Ober, Gesellschaft, Fuß, Arbeit, Platz, Bild, Schirm, Dampf, Schiff, Bund, Tag ## Testing Framework Improvements - Added German-specific dictionary loading with compound generation - Enabled 'de' feature flag for German language support ## Testing - All compound noun generation tests pass - All preposition recognition tests pass - Existing German tests continue to work Generated by Mistral Vibe. Co-Authored-By: Mistral Vibe <vibe@mistral.ai>

Implement Phase 1-5 of the merge-friendly language architecture plan: - Add 'multilingual' feature flag to harper-core that gates the entire language module system (including English, German, Polish, Portuguese, Slovak) - Feature-gate pub mod language and Language/LanguageFamily exports in lib.rs - Keep English in language module system using type aliases to original Dialect types - Update harper-ls, harper-wasm, harper-cli to forward multilingual feature - Fix test files to use Dialect instead of EnglishDialect and correct file paths - Update harper-wasm import to use Dialect type alias This ensures: 1. harper-core compiles both with and without multilingual feature 2. English uses original code paths (dict_word_metadata.rs, language_detection.rs) 3. No dual representation - language/english/ uses type aliases to original types 4. Upstream English changes from master will merge cleanly 5. Minimal merge conflicts (only infrastructure files like Cargo.toml) Generated by Mistral Vibe. Co-Authored-By: Mistral Vibe <vibe@mistral.ai>

- Add section explaining the Feature Flag Architecture - Document the multilingual master feature and individual language features - Explain behavior when multilingual is enabled/disabled - Update feature forwarding note to mention the multilingual feature - Add note about English using original implementation when multilingual is disabled Generated by Mistral Vibe. Co-Authored-By: Mistral Vibe <vibe@mistral.ai>

- Fix import ordering in harper-desktop and harper-ls to match rustfmt expectations - Fix clippy field_reassign_with_default error in German compound spell module - Add multilingual feature flag to harper-desktop Cargo.toml - Update testing_framework to use multilingual feature instead of de This resolves the Just Checks workflow failure that was preventing CI from passing. Generated by Mistral Vibe. Co-Authored-By: Mistral Vibe <vibe@mistral.ai>

…gual support The multilingual feature with German, Portuguese, Polish, and Slovak dictionaries increases the WASM bundle size beyond the previous 30 MB limit. This adjustment allows the Obsidian plugin to include full language support. Generated by Mistral Vibe. Co-Authored-By: Mistral Vibe <vibe@mistral.ai>

- Fix unused variable 'result' in polish_integration_test.rs by prefixing with underscore - Remove redundant assertion that was always true - This resolves the GitHub workflow failure caused by -D warnings flag Generated by Mistral Vibe. Co-Authored-By: Mistral Vibe <vibe@mistral.ai>

Move manual-only test files to .archive/manual_test_scripts/: German language: - test_german_noun_verification.py - test_adjective_declension.py - german_noun_verification.expected.md Misc scripts (not referenced by justfile or CI): - check_language_features.py - cleanup_redundant.py - convert_*_plurals.py (multiple) - convert_verbs*.py (multiple) - deduplicate_dictionary.py - convert_en_plurals.py These files were for manual/offline testing only and not run by automated tests. Generated by Mistral Vibe. Co-Authored-By: Mistral Vibe <vibe@mistral.ai>

…k script - Remove test_polish_integration.rs which was a temporary development file - Add scripts/check_language_features.py for validating feature consistency - Improve language coverage scripts to handle fallback dictionary paths Generated by Mistral Vibe. Co-Authored-By: Mistral Vibe <vibe@mistral.ai>

Change harper-wasm/pkg to harper-wasm to match the actual package.json location. This fixes Dependabot failures where it could not resolve the workspace dependency. Generated by Mistral Vibe. Co-Authored-By: Mistral Vibe <vibe@mistral.ai>

This large development file (566KB) is only used by Python scripts in the .archive/ directory and doesn't need to be tracked in the main repository. Generated by Mistral Vibe. Co-Authored-By: Mistral Vibe <vibe@mistral.ai>

- Added compound flags (h, i, k, l, m, o, q) to base words to enable automatic compound generation - Removed redundant compound nouns that can now be generated automatically - Added just language-add-word recipe for validated word addition with flag validation - Added validate_flags.py script to validate flags against annotations.json - Enhanced compound.rs with better documentation and adjective compound support - Improved annotations.json with compound formation rules - Verified key compounds like vergnügungssteuerpflichtig now work through generation - Dictionary reduced by removing explicit compound entries while maintaining coverage Changes: - Added compound flags to ~100+ base words (art, fall, fach, büro, etc.) - Removed ~20+ redundant compound nouns (artspezifisch, bahnspezifisch, etc.) - Enhanced development tooling for language support Generated by Mistral Vibe. Co-Authored-By: Mistral Vibe <vibe@mistral.ai>

…ancements - Add 40+ new German words from various domains including footwear, business, and technical terms - Clean up duplicate dictionary entries (67 redundant entries removed) - Enhance language-coverage just recipe to use native Rust implementation - Add coverage analysis module to testing framework - Maintain compound word generation efficiency at 6.18x Generated by Mistral Vibe. Co-Authored-By: Mistral Vibe <vibe@mistral.ai>

hippietrail mentioned this pull request May 16, 2026

feat: merged German diagnostics #3403

Closed

Dronakurl marked this pull request as draft May 16, 2026 18:07

Dronakurl force-pushed the feature/german-language-support branch 7 times, most recently from fb9f77e to 9565ee1 Compare May 19, 2026 20:11

This was referenced May 20, 2026

Portugues #2150

Draft

Roadmap and status for non-English language support (FR, ES, IT, DE, etc.) #2654

Open

Dronakurl marked this pull request as ready for review May 20, 2026 18:31

Dronakurl marked this pull request as draft May 20, 2026 19:52

Dronakurl closed this May 21, 2026

Dronakurl reopened this Jun 7, 2026

Dronakurl force-pushed the feature/german-language-support branch from a0c6cdd to e148c7e Compare June 7, 2026 07:48

Dronakurl closed this Jun 7, 2026

Dronakurl deleted the feature/german-language-support branch June 7, 2026 07:50

Dronakurl restored the feature/german-language-support branch June 7, 2026 07:55

Dronakurl reopened this Jun 7, 2026

Dronakurl closed this Jun 7, 2026

Dronakurl deleted the feature/german-language-support branch June 7, 2026 08:02

Dronakurl restored the feature/german-language-support branch June 7, 2026 08:03

Dronakurl reopened this Jun 7, 2026

Dronakurl closed this Jun 7, 2026

Dronakurl deleted the feature/german-language-support branch June 7, 2026 08:03

Dronakurl restored the feature/german-language-support branch June 7, 2026 08:03

Dronakurl reopened this Jun 7, 2026

Dronakurl and others added 10 commits July 28, 2026 19:37

Merge branch 'master' into feature/german-language-support

dd62b3a

Merge branch 'master' into feature/german-language-support

f9c9ddd

hippietrail reviewed Jul 29, 2026

View reviewed changes

Dronakurl and others added 3 commits July 30, 2026 19:27

Dronakurl and others added 15 commits July 31, 2026 20:00

Merge branch 'master' into feature/german-language-support

ce30ac3

Merge branch 'master' into feature/german-language-support

e06930d

Merge branch 'master' into feature/german-language-support

1734bd4

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: add multi-language support (German, Portuguese) - #3402

feat: add multi-language support (German, Portuguese)#3402
Dronakurl wants to merge 268 commits into
Automattic:masterfrom
Dronakurl:feature/german-language-support

Dronakurl commented May 16, 2026 •

edited

Loading

Uh oh!

hippietrail commented May 16, 2026

Uh oh!

hippietrail commented May 20, 2026

Uh oh!

hippietrail left a comment

Uh oh!

hippietrail Jul 29, 2026

Uh oh!

Dronakurl Jul 30, 2026 •

edited

Loading

Uh oh!

hippietrail commented Jul 31, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

Dronakurl commented May 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary of this PR (human generated)

Details (AI generated)

Language System

Dictionary Handling

Modular Structure

Uh oh!

hippietrail commented May 16, 2026

Uh oh!

hippietrail commented May 20, 2026

Uh oh!

hippietrail left a comment

Choose a reason for hiding this comment

Uh oh!

hippietrail Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

Dronakurl Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hippietrail commented Jul 31, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Dronakurl commented May 16, 2026 •

edited

Loading

Dronakurl Jul 30, 2026 •

edited

Loading