DocsV3: Update plugin docgen tooling to work with new MD files and structure

[karenzone]

Current stack-versioned plugin docs tooling generates plugin docs for inclusion in the Logstash Reference (LSR). I broke plugin doc generation (for main and 9.0) when I replaced asciidoc source files in the logstash repo with the new MD source files. Tooling is looking for logstash/docs/index.asciidoc, but it's no longer there.

Line 1574 Can't find --doc /home/runner/work/logstash-docs/logstash-docs/logstash/docs/index.asciidoc

Update:
We have work-arounds that let us generate plugin docs PRs after:

• GH: Add action to build 9.x series plugin docs logstash-docs#1829 + elastic/logstash-docs@dec9b88
• Research: Plugin docgen without docbuild logstash-docs#1839 +Fix yaml format in GH action logstash-docs#1842
  The output will still fail docs-ci because of differences between ADOC and MD, but we have output to work with, cutting out several manual steps outlined in https://github.com/elastic/logstash-docs-md/issues/16..

Decisions

• Where should conversion happen? Source? Enrichment? End-of-pipeline?
• When to migrate source content?
• We'll likely need a phased approach (short-term, mid-term, long-term) to deal with complexity and volume. What does that look like?

Considerations

• Need ability to support both MD (for 9.0+) and ASCIIDOC (for 8.x and earlier).
• File location and Include path settings are different for 9.0+ and 8.x.
• Link format varies between ADOC and MD, and so far, has required manual cleanup

Tooling affected

• LSR plugin docgen:
  • https://github.com/elastic/docs-tools/blob/main/plugindocs.rb
  • GitHub action in logstash-docs
• Note that tooling for the Versioned Plugin Reference (VPR) is unaffected because it's self-contained. That is, it doesn't rely on any files in the logstash repo.

Related and relevant

• Build LSR Plugin docs from this repo logstash-docs#1817

Requirements

• Stack-versioned docs tooling should be modified to generate plugin docs for main 9.0, and 9.x branches
• Stack-versioned docs tooling should maintain existing (or equivalent) functionality to build plugin docs for 8.x series and earlier branches.

[karenzone]

Idea for consideration

• Merge Brandon's PR to get LSR building from logstash-docs.
  We'll also need to add docset.yml and index before we can update the build command.
• Clone the LSR GitHub action. Modify the line where we build the LSR to point to docset.yaml in logstash-docs. Rename new GitHub action to "Reference Doc gen for specified 9.x version."
• Rename existing GitHub action to "Reference Doc gen for specified pre-9.0 version."

[karenzone]

Plugin docgen status

Summary: Generating plugins and then failing on the docbuild step.

• Line 1571: INFO:build_docs:Building HTML from /doc/logstash-docs/docs/plugins/index.asciidoc

• Erroring on relative file paths for :include: files

• Sample: Line 1574 ERROR: integrations/aws.asciidoc: line 21: include file not found: /doc/logstash/docs/include/plugin_header.asciidoc

  • ^^ Building from logstash-docs, but still looking in logstash repo for include files

Observations

• Appear to be no errors on plugins in docs/plugins/static/core-plugins. Why?

• Build command (../docs/build_docs --asciidoctor --respect_edit_url_overrides --doc docs/plugins/index.asciidoc --chunk 1 --open) run locally from /logstash-docs/ finds include files and builds without error

• I cleaned up some directory attributes, just in case we were leaking directory assignments, but still see the same errors.

[karenzone]

cc:/@jsvd

[karenzone]

Update from @jsvd:

ok I got it. the issue is that logstash-docs main branch builds fine, but when bundle exec ruby plugindocs.rb --skip-existing --output-path ../logstash-docs ../logstash/plugins_version_docs.json runs, it picks up versions of plugins not yet in main, that have an incompatible include_path, e.g. https://github.com/logstash-plugins/logstash-filter-elasticsearch/blob/v4.1.1/docs/index.asciidoc?plain=1#L10

[karenzone]

Progress! Running the new GitHub Action for 9.0 generates only a few errors in seven files:

INFO:build_docs:
INFO:build_docs:asciidoctor: ERROR: integrations/jdbc.asciidoc: line 21: include file not found: /doc/logstash/docs/include/plugin_header.asciidoc
INFO:build_docs:asciidoctor: ERROR: inputs/elasticsearch.asciidoc: line 21: include file not found: /doc/logstash/docs/include/plugin_header.asciidoc
    INFO:build_docs:asciidoctor: ERROR: inputs/elasticsearch.asciidoc: line 654: include file not found: /doc/logstash/docs/include/input.asciidoc
INFO:build_docs:asciidoctor: ERROR: inputs/jdbc.asciidoc: line 22: include file not found: /doc/logstash/docs/include/plugin_header-integration.asciidoc
    INFO:build_docs:asciidoctor: ERROR: inputs/jdbc.asciidoc: line 691: include file not found: /doc/logstash/docs/include/input.asciidoc
INFO:build_docs:asciidoctor: ERROR: inputs/syslog.asciidoc: line 21: include file not found: /doc/logstash/docs/include/plugin_header.asciidoc
    INFO:build_docs:asciidoctor: ERROR: inputs/syslog.asciidoc: line 219: include file not found: /doc/logstash/docs/include/input.asciidoc
INFO:build_docs:asciidoctor: ERROR: filters/elasticsearch.asciidoc: line 20: include file not found: /doc/logstash/docs/include/plugin_header.asciidoc
    INFO:build_docs:asciidoctor: ERROR: filters/elasticsearch.asciidoc: line 553: include file not found: /doc/logstash/docs/include/filter.asciidoc
INFO:build_docs:asciidoctor: ERROR: filters/jdbc_static.asciidoc: line 21: include file not found: /doc/logstash/docs/include/plugin_header-integration.asciidoc
    INFO:build_docs:asciidoctor: ERROR: filters/jdbc_static.asciidoc: line 630: include file not found: /doc/logstash/docs/include/filter.asciidoc
INFO:build_docs:asciidoctor: ERROR: filters/jdbc_streaming.asciidoc: line 21: include file not found: /doc/logstash/docs/include/plugin_header-integration.asciidoc
    INFO:build_docs:asciidoctor: ERROR: filters/jdbc_streaming.asciidoc: line 319: include file not found: /doc/logstash/docs/include/filter.asciidoc
INFO:build_docs:asciidoctor: WARNING: invalid reference: plugins-inputs-elasticsearch-add_field

Now I know which plugins have updates, and can cross-reference those plugins with the 9.0 lock.release file to get the version numbers we need to update for 9.0.

Plugin	Prev version	New version	PR
integration-jdbc input-jdbc filter-jdbc_static filter-jdbc_streaming	5.5.2	5.5.3	logstash-plugins/logstash-integration-jdbc#181 Doc change to filter-jdbc_static
input-elasticsearch	5.0.0	5.0.2	logstash-plugins/logstash-input-elasticsearch#218 Doc change logstash-plugins/logstash-input-elasticsearch#223
input-syslog	3.7.0	3.7.1	logstash-plugins/logstash-input-syslog#78
filter-elasticsearch	4.1.0	4.1.1	logstash-plugins/logstash-filter-elasticsearch#191

Note that all files will need updates to version, release date, and changelog link, regardless of if they have doc changes.

This is not a sustainable workflow, but will get us where we need to be for the release.