Improve the default HTML parser and document normalizer extension

.gitignore

@@ -21,3 +21,9 @@
 # IDE and silo-local state
 /.idea/
 /.silo.local.yml
+
+# Docs site build output (the docs branch holds the source)
+/docs/.astro/
+/docs/dist/
+/docs/node_modules/
+/.pnpm-store/

AGENTS.md

@@ -143,7 +143,10 @@ except across blank lines. Every tag must pick one of two forms when
 
 `spec/integration/markbridge/renderers/discourse/html_mode_contract_spec.rb`
 enforces this structurally: every registered tag is rendered in
-`html_mode` and the output is checked for raw Markdown sigils.
+`html_mode` and the output is checked for raw Markdown sigils. The
+check ships as a shared RSpec example (`require "markbridge/rspec"`,
+then `it_behaves_like "an html_mode safe tag"`), so consumer projects
+can run it against their own tags.
 
 **See `examples/` for complete examples.**
 

docs/extending.md

@@ -466,6 +466,39 @@ library.register(AST::Spoiler) do |element, interface|
 end
 ```
 
+### Testing Custom Tags Against the html_mode Contract
+
+Every tag must also render sensibly when `interface.html_mode?` is
+true — inside a CommonMark HTML block (for example the table HTML
+fallback), raw Markdown output would show up as literal text. A tag
+has two valid forms there: a raw HTML fragment, or its normal Markdown
+wrapped in `\n\n…\n\n` (a Markdown island; the blank lines let
+CommonMark parse the inner content).
+
+Markbridge ships this check as a shared RSpec example. Require it from
+your spec setup and give it your tag and a sample element:
+
+```ruby
+require "markbridge/rspec"
+
+RSpec.describe MyQuoteTag do
+  it_behaves_like "an html_mode safe tag" do
+    let(:tag) { described_class.new }
+    let(:element) do
+      element = MyQuote.new
+      element << Markbridge::AST::Text.new("body *with* sigils")
+      element
+    end
+  end
+end
+```
+
+Give the element children whose text contains Markdown sigils — a tag
+that passes them through unprotected is then caught. Markbridge runs
+its own tags through the same example. When the tag needs a customized
+renderer to resolve its children, override `markbridge_renderer` with
+your configured renderer inside the block.
+
 ## Plugin Pattern
 
 Create reusable plugins that bundle parser and renderer extensions:

docs/parsers/html.md

@@ -17,10 +17,9 @@ This guide explains how the HTML parser converts standard HTML into the Markbrid
 The HTML parser (`Markbridge::Parsers::HTML::Parser`) uses Nokogiri to convert HTML markup into AST. It provides a simpler alternative to the BBCode parser when working with HTML content.
 
 **Key Features:**
-- Leverages Nokogiri's battle-tested HTML parser (libxml2 on MRI/TruffleRuby, Xerces/NekoHTML on JRuby)
+- Uses Nokogiri's HTML parser (libxml2 on MRI/TruffleRuby, Xerces/NekoHTML on JRuby)
 - Handles malformed HTML gracefully
 - Stateless handler API (simpler than BBCode)
-- Lambda handler support for quick customization
 - Void element detection (self-closing tags)
 
 **Dependencies:**
@@ -70,6 +69,14 @@ end
 | `<code>` | `AST::Code` | Inline or block code |
 | `<pre>` | `AST::Code` | Preformatted code block |
 
+The language for syntax highlighting is read from the first of: a
+`language-*` class on the element itself or on its direct `<code>` child
+(the CommonMark convention, `<pre><code class="language-ruby">`), the
+`lang` attribute, or a lone class used as-is. Styling classes like
+`class="hljs codeblock"` are ignored, and the result must be a single
+clean token (letters, digits, `_`, `+`, `-`) so it is safe on a fence
+line.
+
 ### Link Tags
 
 | HTML Tag | AST Node | Notes |
@@ -89,6 +96,7 @@ end
 | HTML Tag | AST Node | Notes |
 |----------|----------|-------|
 | `<p>` | Paragraph handling | Adds spacing between paragraphs |
+| `<h1>` – `<h6>` | `AST::Heading` | Heading with matching level |
 | `<br>` | `AST::LineBreak` | Line break |
 | `<hr>` | `AST::HorizontalRule` | Horizontal rule |
 
@@ -160,14 +168,17 @@ class CustomHandler < Markbridge::Parsers::HTML::Handlers::BaseHandler
     @element_class = element_class
   end
 
-  def process(element:, parent:, processor:)
+  def process(element:, parent:)
     # element: Nokogiri::XML::Element (complete DOM element)
-    # parent: AST::Element (where to add children)
-    # processor: Parser (for processing children)
+    # parent: AST::Element (where to add the new node)
 
     ast_element = @element_class.new
     parent << ast_element
-    processor.process_children(element, ast_element)
+
+    # Return the created element and the parser processes the DOM
+    # element's children into it. Return nil to skip the children
+    # (for example when the handler consumed the content itself).
+    ast_element
   end
 
   attr_reader :element_class
@@ -187,11 +198,11 @@ registry.register(["b", "strong"], handler)
 
 #### RawHandler
 
-For code blocks that preserve content:
+For code blocks that preserve content. Takes the AST class to create:
 
 ```ruby
-handler = RawHandler.new
-registry.register(["code", "pre"], handler)
+handler = RawHandler.new(Markbridge::AST::Code)
+registry.register(["code", "pre", "tt"], handler)
 ```
 
 #### UrlHandler
@@ -212,20 +223,13 @@ registry.register(["ul", "ol"], ListHandler.new)
 registry.register("li", ListItemHandler.new)
 ```
 
-### Lambda Handlers
+#### SelfClosingHandler
 
-For simple cases, use lambda handlers:
+For void elements that map to a childless AST node:
 
 ```ruby
-registry.register("hr", ->(element:, parent:, **) {
-  parent << AST::HorizontalRule.new
-  nil # Return nil to skip processing children
-})
-
-registry.register("br", ->(element:, parent:, **) {
-  parent << AST::LineBreak.new
-  nil
-})
+registry.register("br", SelfClosingHandler.new(Markbridge::AST::LineBreak))
+registry.register("hr", SelfClosingHandler.new(Markbridge::AST::HorizontalRule))
 ```
 
 ## Configuration

lib/markbridge/parsers/html.rb

@@ -17,6 +17,7 @@
 require_relative "html/handlers/list_handler"
 require_relative "html/handlers/list_item_handler"
 require_relative "html/handlers/quote_handler"
+require_relative "html/handlers/heading_handler"
 require_relative "html/handlers/paragraph_handler"
 require_relative "html/handlers/table_handler"
 require_relative "html/handlers/table_row_handler"

lib/markbridge/parsers/html/handler_registry.rb

@@ -132,6 +132,7 @@ def self.default
             registry.register("a", Handlers::UrlHandler.new)
             registry.register("img", Handlers::ImageHandler.new)
             registry.register("blockquote", Handlers::QuoteHandler.new)
+            registry.register(%w[h1 h2 h3 h4 h5 h6], Handlers::HeadingHandler.new)
             registry.register("br", Handlers::SelfClosingHandler.new(AST::LineBreak))
             registry.register("hr", Handlers::SelfClosingHandler.new(AST::HorizontalRule))
             registry.register(%w[ul ol], Handlers::ListHandler.new)

lib/markbridge/parsers/html/handlers/heading_handler.rb

@@ -0,0 +1,29 @@
+# frozen_string_literal: true
+
+module Markbridge
+  module Parsers
+    module HTML
+      module Handlers
+        # Handles <h1> through <h6> by creating a Heading element with the
+        # matching level and processing children into it.
+        class HeadingHandler < BaseHandler
+          # @param element [Nokogiri::XML::Element] the heading element
+          # @param parent [AST::Element] the parent AST node
+          # @return [AST::Heading] the created heading, so children get processed into it
+          def process(element:, parent:)
+            level = element.name.delete_prefix("h").to_i.clamp(1, 6)
+            heading = AST::Heading.new(level:)
+            parent << heading
+
+            heading
+          end
+
+          # @return [Class] AST::Heading
+          def element_class
+            AST::Heading
+          end
+        end
+      end
+    end
+  end
+end

lib/markbridge/parsers/html/handlers/raw_handler.rb

@@ -6,6 +6,12 @@ module HTML
       module Handlers
         # Handler for raw/preformatted tags that preserve content as-is
         class RawHandler < BaseHandler
+          # A language must be one clean token — the renderer splices it
+          # into the code fence line, so a class attribute with spaces or
+          # other markup characters must not end up there.
+          LANGUAGE_PATTERN = /\A[a-z0-9][a-z0-9_+-]*\z/
+          private_constant :LANGUAGE_PATTERN
+
           def initialize(element_class)
             @element_class = element_class
           end
@@ -14,10 +20,7 @@ def process(element:, parent:)
             # Get the inner text content
             content = element.inner_text
 
-            # Extract language from class or lang attribute
-            language = element["class"] || element["lang"]
-
-            ast_element = @element_class.new(language:)
+            ast_element = @element_class.new(language: language_for(element))
             ast_element << AST::Text.new(content) unless content.empty?
             parent << ast_element
 
@@ -26,6 +29,43 @@ def process(element:, parent:)
           end
 
           attr_reader :element_class
+
+          private
+
+          # The language of a code block, from the strongest signal to the
+          # weakest: a `language-*` class on the element itself or on its
+          # direct <code> child (the CommonMark convention for fenced code,
+          # `<pre><code class="language-ruby">`), then the `lang` attribute,
+          # then a lone class used as-is. A lone class ranks below `lang`
+          # because a class can be pure styling (`hljs`, `prettyprint`).
+          def language_for(element)
+            code_child_classes = element.at_xpath("./code")&.[]("class")
+
+            prefixed_language(element["class"]) || prefixed_language(code_child_classes) ||
+              attribute_language(element["lang"]) || single_class_language(element["class"]) ||
+              single_class_language(code_child_classes)
+          end
+
+          def prefixed_language(classes)
+            classes
+              &.split
+              &.filter_map do |name|
+                name.delete_prefix("language-").downcase if name.start_with?("language-")
+              end
+              &.find { |language| LANGUAGE_PATTERN.match?(language) }
+          end
+
+          def attribute_language(value)
+            language = value&.strip&.downcase
+            language if language&.match?(LANGUAGE_PATTERN)
+          end
+
+          def single_class_language(classes)
+            names = classes&.split
+            return unless names&.length == 1
+
+            attribute_language(names.first)
+          end
         end
       end
     end

lib/markbridge/renderers/discourse.rb

@@ -7,6 +7,7 @@
 require_relative "discourse/markdown_escaper"
 require_relative "discourse/identity_escaper"
 require_relative "discourse/html_escaper"
+require_relative "discourse/html_block_safety"
 require_relative "discourse/postprocessor"
 
 # Builders

lib/markbridge/renderers/discourse/html_block_safety.rb

@@ -0,0 +1,31 @@
+# frozen_string_literal: true
+
+module Markbridge
+  module Renderers
+    module Discourse
+      # Decides whether a rendered fragment is safe to splice into a
+      # CommonMark HTML block (spec §4.6). Inside such a block the
+      # content passes through as raw HTML; Markdown is only parsed
+      # again across blank lines. Safe output is therefore: a raw HTML
+      # or plain-text fragment without Markdown sigils, or a
+      # +\n\n…\n\n+ wrap — a deliberate Markdown island.
+      #
+      # Used by the html_mode contract check that ships in
+      # +markbridge/rspec+ and by this repo's own contract spec.
+      module HtmlBlockSafety
+        # Markdown sigils that would surface as literal text inside an
+        # HTML block: emphasis (`*`, `_`, `~`) and link middles (`](`).
+        MARKDOWN_SIGILS = /[*_~]|\]\(/
+        private_constant :MARKDOWN_SIGILS
+
+        # @param output [String] a tag's html_mode render result
+        # @return [Boolean]
+        def self.safe?(output)
+          return true if output.start_with?("\n\n") && output.end_with?("\n\n")
+
+          !output.match?(MARKDOWN_SIGILS)
+        end
+      end
+    end
+  end
+end

lib/markbridge/renderers/discourse/tags/code_tag.rb

@@ -9,6 +9,10 @@ def render(element, interface)
             child_context = interface.with_parent(element)
             content = interface.render_children(element, context: child_context)
 
+            # An empty element renders to nothing — a bare `` pair or an
+            # empty fence would only add noise to the output.
+            return "" if content.empty?
+
             if interface.block_context?(element)
               if interface.html_mode?
                 render_html_block(content, element.language)

lib/markbridge/renderers/discourse/tags/details_tag.rb

@@ -25,12 +25,16 @@ class DetailsTag < Tag
 
           def render(element, interface)
             child_context = interface.with_parent(element)
-            content = interface.render_children(element, context: child_context)
+            content = interface.render_children(element, context: child_context).strip
+
+            # A details block with nothing to show renders to nothing —
+            # an empty [details] shell would only add noise to the output.
+            return "" if content.empty?
 
             return render_html(element.title, content) if interface.html_mode?
 
             opener = element.title ? %([details="#{element.title}"]) : "[details]"
-            "\n\n#{opener}\n#{content.strip}\n[/details]\n\n"
+            "\n\n#{opener}\n#{content}\n[/details]\n\n"
           end
 
           private

lib/markbridge/renderers/discourse/tags/spoiler_tag.rb

@@ -11,6 +11,10 @@ def render(element, interface)
             child_context = interface.with_parent(element)
             content = interface.render_children(element, context: child_context)
 
+            # A spoiler with nothing to hide renders to nothing — an empty
+            # [spoiler] shell would only add noise to the output.
+            return "" if content.empty?
+
             return render_html(element.title, content) if interface.html_mode?
 
             if element.title