Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions .editorconfig
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
root = true

[*.py]
profile = black
indent_style = space
indent_size = 4
src_paths = src,tests

3 changes: 3 additions & 0 deletions .isort.cfg
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
[settings]
profile=black
src_paths=src,tests
27 changes: 26 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
.PHONY: help run build preview aggregate aggregate-repo aggregate-update-repo aggregate-repo-single aggregate-update-repo-single test-aggregate-local clean clean-projects clean-aggregated-git test test-unit test-integration check spelling linkcheck woke
.PHONY: help run build preview aggregate aggregate-repo aggregate-update-repo aggregate-repo-single aggregate-update-repo-single test-aggregate-local clean clean-projects clean-aggregated-git test test-unit test-integration check spelling linkcheck woke glossary glossary-check format

help:
@echo "Garden Linux Documentation Hub - Available targets:"
Expand All @@ -18,6 +18,8 @@ help:
@echo " spelling - Check spelling with codespell"
@echo " linkcheck - Check links with lychee"
@echo " woke - Check inclusive language with woke"
@echo " glossary - Process glossary links in documentation"
@echo " glossary-check - Validate glossary structure"
@echo ""
@echo " Documentation Aggregation:"
@echo " aggregate-local - Aggregate from local repos (file:// URLs in repos-config.local.json)"
Expand Down Expand Up @@ -48,6 +50,10 @@ build: install clean aggregate
preview:
pnpm run docs:preview

format:
black src/ tests/
isort src/ tests/

# Testing
test: test-unit test-integration
@echo "All tests passed!"
Expand Down Expand Up @@ -76,6 +82,25 @@ woke:
@echo "Running inclusive language checks..."
@pnpm run docs:woke

glossary:
@echo "Processing glossary links..."
@python3 -c "import sys, importlib.util; \
from pathlib import Path; \
spec = importlib.util.spec_from_file_location('auto_glossary', 'src/aggregation/auto_glossary.py'); \
mod = importlib.util.module_from_spec(spec); \
spec.loader.exec_module(mod); \
mod.process_glossary_links(Path('docs'))"

glossary-check:
@echo "Validating glossary structure..."
@python3 -c "import importlib.util; \
from pathlib import Path; \
spec = importlib.util.spec_from_file_location('auto_glossary', 'src/aggregation/auto_glossary.py'); \
mod = importlib.util.module_from_spec(spec); \
spec.loader.exec_module(mod); \
mod.AutoGlossary(Path('docs/reference/glossary.md')); \
print(' ✓ Glossary structure valid')"

# Documentation Aggregation
aggregate-local:
@echo "Aggregating from local repositories (relative paths)..."
Expand Down
211 changes: 211 additions & 0 deletions docs/contributing/documentation/auto-glossary.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,211 @@
---
title: "Documentation Auto Glossary"
description: "Automatically link common idioms to their glossary entry"
order: 13
related_topics:
- /contributing/documentation/documentation_workflow.md
- /contributing/documentation/writing_good_docs.md
- /contributing/documentation/adding-repos.md
- /contributing/documentation/working-locally.md
- /contributing/documentation/ci-architecture.md
- /contributing/documentation/ci-workflows-reference.md
- /contributing/documentation/configuration.md
- /contributing/documentation/technical.md
- /contributing/documentation/testing.md
- /contributing/documentation/vitepress-features.md
---

# Auto Glossary

The documentation system includes an automatic glossary linker that converts marked terms into links pointing to the glossary page.

## Quick Start

Mark terms in your documentation using the marker format:

```markdown
Deploy {glossary:Gardenlinux} on {glossary:AWS} using {glossary:KVM}.
```

After aggregation, this becomes:

```markdown
Deploy [Gardenlinux](/reference/glossary#gardenlinux) on [AWS](/reference/glossary#aws) using [KVM](/reference/glossary#kvm).
```

## Function

During aggregation (`make aggregate`), the system:

1. Parses `docs/reference/glossary.md` to extract all level-3 headers as terms
2. Extracts aliases from terms with parenthesized expansions (e.g., `ADR (Architecture Decision Record)`)
3. Scans all markdown files for glossary markers
4. Replaces markers with markdown links to glossary anchors
5. Preserves code blocks, inline code, and existing links

## Marking Terms

Use the format where the term name matches a glossary entry:

```markdown
{glossary:AWS}
{glossary:Garden Linux}
{glossary:ADR (Architecture Decision Record)}
```

Term matching is case-insensitive but preserves your original formatting:

- Input: `{glossary:aws}` produces `[aws](/reference/glossary#aws)`
- Input: `{glossary:AWS}` produces `[AWS](/reference/glossary#aws)`

::: details Developer Information
The auto glossary system is designed to accept a new custom `entry_format` through which
the shortcode `{glossary:*}` can be replaced with a new custom pattern.

Check the source code for more information.
:::

## Protected Regions

The system leaves certain content unchanged:

- Fenced code blocks (triple backticks)
- Inline code (single backticks)
- Existing markdown links

Glossary markers inside these regions are not processed.

## Adding Glossary Terms

To add a new term:

1. Edit `docs/reference/glossary.md`
2. Add a level-3 header:

```markdown
### New Term Name

Definition of the term.
```

The anchor is generated automatically (lowercase, hyphens replace spaces). Reference it with the marker format in your documentation.

## Alias Support

Terms with parenthesized expansions create aliases automatically.

Given this glossary entry:

```markdown
### ADR (Architecture Decision Record)
```

The system creates:
- Main term: `ADR (Architecture Decision Record)`
- Alias: `Architecture Decision Record`
- Alias: `ADR`

All three can be used in markers and link to the same glossary entry.

## Auto-Linking



Auto-linking creates links for glossary terms without requiring explicit markers. This feature is disabled by default.

With auto-linking enabled, input text:

```
Deploy Gardenlinux on AWS using KVM virtualization.
```

Becomes:

```markdown
Deploy [Garden Linux](/reference/glossary#garden-linux) on [AWS](/reference/glossary#aws) using [KVM](/reference/glossary#kvm) virtualization.
```

Auto-linking rules:
- Links only the first occurrence of each term
- Matches longer terms first (`Garden Linux` before `Linux`)
- Case-insensitive matching
- Respects word boundaries
- Preserves code blocks and inline code
- Does not modify existing links

::: warning
This feature is disabled by default as it is highly experimental and intended for research work only.

**Contributors must not rely on this feature when contributing documentation.**
:::

## Makefile Targets

Process glossary links manually:

```bash
make glossary
```

Validate glossary structure:

```bash
make glossary-check
```

## Warnings

**Term not found:**

```
[Warning][auto-glossary] Term 'xyz' not found in glossary (referenced in file.md)
```

The marker remains unchanged. Add the term to the glossary, fix the spelling, or remove the marker.

**Anchor collision:**

```
[Warning][auto-glossary] Anchor collision detected:
'Term 1' and 'Term 2' both generate anchor 'term-1-2'
```

Rename one of the terms in the glossary.

## Anchor Generation

Terms are converted to VitePress-compatible anchors:

- Convert to lowercase
- Replace spaces and special characters with hyphens
- Collapse consecutive hyphens
- Strip leading and trailing hyphens
- Normalize Unicode to ASCII

Examples:
- `AWS` becomes `aws`
- `Garden Linux` becomes `garden-linux`
- `ADR (Architecture Decision Record)` becomes `adr-architecture-decision-record`

## Technical Reference

Module location: `src/aggregation/auto_glossary.py`

Main components:
- `AutoGlossary` class for glossary processing
- `process_glossary_links()` function for batch processing
- `GLOSSARY_ENTRY_FORMAT` constant defining the default marker format

Integration points:
- Runs as part of `src/aggregate.py` after release notes generation
- Makefile provides `glossary` and `glossary-check` targets

## Testing

Run the test suite:

```bash
python3 -m pytest tests/unit/test_auto_glossary*.py tests/unit/test_generate_anchor.py -v
```

Test coverage includes format validation, anchor generation, alias extraction, auto-linking, and integration tests.
2 changes: 2 additions & 0 deletions requirements.txt
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
codespell==2.4.2
black
isort
pytest
pyyaml
# glrd @ git+https://github.com/gardenlinux/glrd.git@v4.2.0
Expand Down
20 changes: 13 additions & 7 deletions src/aggregate.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,13 +11,13 @@
import tempfile
from pathlib import Path

from aggregation import (DocsFetcher, copy_targeted_docs, load_config,
save_config)
from aggregation.structure import verify_internal_links
from aggregation import DocsFetcher, copy_targeted_docs, load_config, save_config
from aggregation.auto_glossary import process_glossary_links
from aggregation.flavor_matrix import generate_flavor_matrix_docs
from aggregation.github_api import GitHubAPIError, list_repo_releases
from aggregation.release_notes import generate_release_notes_docs
from aggregation.releases import generate_release_docs
from aggregation.structure import verify_internal_links


def transform_repo_docs(
Expand Down Expand Up @@ -227,9 +227,7 @@ def main() -> int:
repo_names = {repo.name for repo in repos}
if args.repo not in repo_names:
if args.single:
parser.error(
f"Repository '{args.repo}' not found in config"
)
parser.error(f"Repository '{args.repo}' not found in config")
else:
print(f"WARNING: Repository '{args.repo}' not found in config")

Expand Down Expand Up @@ -308,7 +306,9 @@ def main() -> int:
)
return 1
existing_gh_tags = {r["tag_name"].lstrip("v") for r in gh_releases}
print(f" Fetched {len(gh_releases)} GitHub release(s), {len(existing_gh_tags)} unique tag(s)")
print(
f" Fetched {len(gh_releases)} GitHub release(s), {len(existing_gh_tags)} unique tag(s)"
)

# Generate release documentation from GLRD
print(f"\n{'='*60}")
Expand All @@ -322,6 +322,12 @@ def main() -> int:
print(f"{'='*60}\n")
generate_release_notes_docs(docs_dir, gh_releases)

# Process glossary links
print(f"\n{'='*60}")
print("Processing glossary links...")
print(f"{'='*60}\n")
process_glossary_links(docs_dir)

# Summary
print(f"\n{'='*60}")
print("Documentation aggregation complete!")
Expand Down
15 changes: 11 additions & 4 deletions src/aggregation/__init__.py
Original file line number Diff line number Diff line change
@@ -1,16 +1,20 @@
"""Aggregation package for docs-ng documentation aggregation."""

# Re-export commonly used functions for backward compatibility with tests
from .auto_glossary import AutoGlossary, process_glossary_links
from .config import load_config, save_config
from .fetcher import DocsFetcher
from .flavor_matrix import generate_flavor_matrix_docs
from .models import AggregateResult, RepoConfig
from .releases import generate_release_docs
from .sphinx_builder import build_sphinx_markdown
from .structure import (copy_targeted_docs, process_all_markdown,
verify_internal_links)
from .transformer import (ensure_frontmatter, parse_frontmatter,
quote_yaml_value, rewrite_links)
from .structure import copy_targeted_docs, process_all_markdown, verify_internal_links
from .transformer import (
ensure_frontmatter,
parse_frontmatter,
quote_yaml_value,
rewrite_links,
)

__all__ = [
# Models
Expand All @@ -36,4 +40,7 @@
"generate_flavor_matrix_docs",
# Sphinx builder
"build_sphinx_markdown",
# Auto Glossary
"AutoGlossary",
"process_glossary_links",
]
Loading
Loading