fix(sitemap): reject HTML error pages instead of publishing them as the sitemap

[ahmadreza-deriv]

Context

We recently moved the staging source from staging.deriv.com to staging-webflow.deriv.com. The live sitemap at https://urls.deriv.com/sitemap.xml broke around that change. This PR fixes the mechanism that allowed the breakage to be published, and makes the same class of failure loud instead of silent.

What is actually live right now

https://urls.deriv.com/sitemap.xml is not a sitemap. It is a Next.js "Page not found" HTML page with an XML declaration on top and a </urlset> tag glued to the bottom — 14,464 bytes, where a real sitemap is ~15 MB. It ends like this:

    </body>
</html>
</urlset>

Root cause

The old host staging.deriv.com no longer serves a sitemap after the migration. It returns a Next.js 404 page — with an HTTP 200 status and content-type: text/html.

Four things then lined up:

1. curl -o ... || exit 1 cannot catch that, because curl had no --fail. On an HTTP 200 it exits 0 regardless of the body.
2. xml2js parsed the HTML successfully — Next.js output is well-formed enough.
3. The root element became html, not urlset, so every transform bailed out silently through its guard, e.g. if (!sitemap.urlset || !sitemap.urlset.url) return sitemap;.
4. The closing-tag "repair" then appended </urlset> to a document that never contained one.

Run #11 did exactly this and reported success:

R2 Info - Uploading content/deriv.com/sitemap.xml (14.13 KB) to sitemap.xml
R2 Success - content/deriv.com/sitemap.xml

Verified by feeding the current staging.deriv.com response through modify_sitemap_new.js locally: the output is byte-identical to what urls.deriv.com serves today, and 14,464 bytes matches the "14.13 KB" in that log.

Why the later runs failed instead

Runs #12 and #13 already had the corrected hostname, but both failed:

Error parsing XML from temp/staging_sitemap.xml: Error: Unexpected close tag
Line: 237  Column: 7  Char: >

sax reports lines 0-indexed and Column as the 1-based position of the >, so this is a 7-character closing tag at zero indentation on line 238 — </head>, </body> or </html>. The real sitemap uses 4-space indentation and has no such tag, so this was again an HTML document, this time one that did not parse. Same missing --fail; the only difference is that the bad HTML happened to be unparseable, so nothing was uploaded and run #11's corrupt file stayed live.

The failing run was the safer outcome. That asymmetry is what this PR removes.

Changes

.github/workflows/generate_new_sitemap.yml

• curl now uses --fail, so an HTTP error can never be saved as content.
• Added --retry 5 --retry-delay 10 --retry-all-errors --max-time, which absorbs the transient origin errors that killed runs ako/ add EOF #12 and ako/ add publish config #13.
• New "Verify downloaded sitemap" step greps for <urlset> and dumps the first 500 bytes on failure, so the log says what actually arrived.

.github/modify_sitemap_new.js

• assertIsSitemap hard-fails when the parsed root element is not <urlset>.
• The xml += "\n</urlset>" append is replaced by a thrown error. That single line is what turned an error page into a "valid" sitemap.
• A MIN_EXPECTED_URLS floor (1,000) catches a silently gutted sitemap.

Verification

Test	Result
Real source sitemap (22 MB, 8,351 URLs)	Succeeds — 15 MB output, 5,813 URLs
Output vs. pre-change output for same input	Byte-identical — no behaviour change
HTML error page as input	Exits 1, no output file written
Generated sitemap re-parsed by xml2js	Valid, root urlset, 5,813 URLs
Domains in output	Only https://deriv.com, zero staging references
generate_new_sitemap.yml	Parses as YAML; all run blocks pass bash -n

Scope

This PR does not change the sitemap's content, filtering rules, or the lastmod removal from #346. For a healthy source the output is byte-identical. It only changes what happens when the source responds with something that is not a sitemap.

It also does not fix the origin. If staging-webflow.deriv.com serves an error page again, the workflow will now fail cleanly rather than publish it. The origin is currently healthy (200, 22,266,855 bytes, application/rss+xml, regenerated at 07:43:49 GMT), so a run on this branch should restore the live sitemap.

Follow-up, deliberately not included

publish-website.yml lines 99-100 still have the identical unguarded curl against the retired staging.deriv.com:

curl -o temp/staging_sitemap.xml https://staging.deriv.com/sitemap.xml || exit 1
curl -o temp/robots.txt https://staging.deriv.com/robots.txt || exit 1

That workflow will corrupt the sitemap in exactly the way run #11 did the next time it runs. Kept out of this PR to hold the blast radius down — happy to do it here or separately.

[github-actions]

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

OpenSSF Scorecard

Package	Version	Score	Details

Scanned Manifest Files