[SPARK-56763][SPARK-56535][INFRA][3.5] Recover branch-3.5 CI

[sarutak]

What changes were proposed in this pull request?

Fix the broken Base image build CI workflow on branch-3.5 by addressing multiple issues in dev/infra/Dockerfile and python/mypy.ini:

1. Stale apt package index: Add apt-get update before each apt-get install that runs in a separate RUN layer from the initial update, preventing 404 errors when packages are superseded on the Ubuntu 20.04 archive.

2. EOL Python get-pip.py URLs: Use version-specific get-pip.py URLs (pip/3.9/get-pip.py, pip/3.8/get-pip.py) since the generic endpoint no longer supports Python 3.8/3.9.

3. PyPy 3.8 build dependency conflicts: Pin beniget==0.4.1 and pyproject-metadata==0.8.1 via pip constraints to resolve incompatibilities in the scipy build chain:

   • beniget>=0.4.2 references gast.MatchStar which doesn't exist in gast<=0.5.3 (required by pythran). See beniget#108.
   • pyproject-metadata>=0.9.0 introduced breaking PEP 639 changes incompatible with the older meson-python in this environment. See meson-python changelog.

4. mypy failures from transitive dependencies: Add follow_imports = skip for pydantic and sqlalchemy in python/mypy.ini. These packages are not used by Spark directly but are pulled in transitively via mlflow (mlflow → sqlalchemy, mlflow → mlflow-tracing → pydantic). Newer versions of these packages ship type annotations incompatible with the Python 3.9 + older mypy version in this CI environment, causing spurious type-check errors on code Spark doesn't own.

5. Add libuv1-dev for R: Install libuv1-dev which is required by the R httpuv/fs package (a transitive dependency of rmarkdown and testthat). Without it, the R package fs fails to compile from source when the Docker layer cache is invalidated. This is the same fix applied to master and other branches in SPARK-56540 / #55414.

6. Pin plotly<6.0: Plotly 6.0 introduced breaking changes in datetime handling that cause PySpark plot-related test failures. This is the same fix applied to master in SPARK-51143 / #49863.

7. Misc: Update FULL_REFRESH_DATE to force a full image rebuild.

Why are the changes needed?

The branch-3.5 CI has been broken for an extended period. The Ubuntu 20.04 (focal) base image is aging, and upstream package repositories have rotated or removed packages that the Dockerfile previously fetched without version pins. Multiple unrelated failures compound to make the Docker image unbuildable.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

• Base image build workflow passes on GitHub Actions.
• docker build dev/infra succeeds locally.

Was this patch authored or co-authored using generative AI tooling?

Kiro CLI / Opus 4.6

[sarutak]

Once Base image build passes on GA, I'll fix Linters, licenses, dependencies and documentation generation workflow.

cc: @dongjoon-hyun @gaogaotiantian @zhengruifeng

[gaogaotiantian]

I don't quite understand

RUN printf 'beniget==0.4.1\npyproject-metadata<0.9.0\n' > /tmp/pypy-constraints.txt && \
    PIP_CONSTRAINT=/tmp/pypy-constraints.txt pypy3 -m pip install numpy scipy coverage matplotlib && \
    SETUPTOOLS_USE_DISTUTILS=stdlib pypy3 -m pip install 'pandas<=2.0.3' && \
    rm /tmp/pypy-constraints.txt

Why pip constraint? Are these package needed by some other packages? Can we just pin the package we need to a specific version? It's an infra docker.

[sarutak]

Why pip constraint? Are these package needed by some other packages? Can we just pin the package we need to a specific version? It's an infra docker.

scipy will be built using meson and beniget==0.4.1 and pyproject-metadata<0.9.0 are required during the build.

[gaogaotiantian]

Usually the reason CI broke is that a new version of something used some new stuff. It's an old branch so it has to be working at certain point of time. Could you show some evidence to support your fix? Like why the specific version 0.4.1 and 0.9.0? What is the dependency chain? What did not work when we didn't pin the version?

[sarutak]

I'll pin pyproject-metadata==0.8.1.

Could you show some evidence to support your fix? Like why the specific version 0.4.1 and 0.9.0? What is the dependency chain? What did not work when we didn't pin the version?

Regarding beniget

scipy uses pythran (>=0.12.0,<0.13.0) during its build process. pythran 0.12.x requires gast<=0.5.3 and beniget~=0.4.0 as dependencies.

• pythran 0.12.2 requirements.txt: https://github.com/serge-sans-paille/pythran/blob/0.12.2/requirements.txt

beniget 0.4.2 and later added code to handle Python 3.10 match statement AST nodes (MatchStar, MatchAs), but these nodes were only implemented in gast 0.5.4. Since pythran constrains gast to <=0.5.3, the combination of beniget 0.4.2 + gast 0.5.3 results in the following error:

AttributeError: module 'gast' has no attribute 'MatchStar'

• beniget issue (bug report): AttributeError: module 'gast' has no attribute 'MatchStar', when gast < 0.5.4 installed serge-sans-paille/beniget#108
• gast 0.5.4 diff where MatchStar was added: serge-sans-paille/gast@0.5.3...0.5.4#diff-44b67094998a7df1e5ce843b5a2faa44950f5896ac9e07ab99d09e2b24c90143R328

Upgrading gast to 0.5.4 or later is not possible because it violates pythran's constraint. Pinning beniget==0.4.1 is the only solution.

Regarding pyproject-metadata

scipy's build system, meson-python, uses pyproject-metadata for metadata processing. pyproject-metadata 0.9.0 introduced breaking changes for PEP 639 support and implicitly requires packaging>=23.2 (this requirement is not correctly recorded in the package metadata due to a bug).

The older version of meson-python used in the Dockerfile environment is not compatible with the API changes in pyproject-metadata 0.9.0, causing the build to fail at the metadata generation stage. Pinning pyproject-metadata==0.8.1 maintains compatibility with the existing meson-python version.

• meson-python changelog (notes that 0.18.0 bumped the minimum pyproject-metadata requirement to 0.9.0, and mentions the packaging version requirement bug in 0.9.1): https://mesonbuild.com/meson-python/changelog.html