[SPARK-58747][PYTHON] Trim unnecessary commands from MANIFEST.in

[nchammas]

What changes were proposed in this pull request?

Trim unnecessary commands from MANIFEST.in. Tweak the various setup.py files to subsume their behavior.

MANIFEST.in recursive-include ... * is recursive, whereas * in setuptools isn't. The equivalent in setuptools is to use **.

Why are the changes needed?

Having duplicate (and unnecessary!) packaging specs across setup.py and MANIFEST.in is confusing.

The explicit license_files argument is not necessary -- setuptools packages these files by default -- but nice to have since it's explicit.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

I built the three sdists off of master and then again from this branch. I diffed their contents and found no differences.

set -euo pipefail

# from inside python/
mkdir -p dist

for pkg in classic client connect; do
  echo "=== $pkg ==="
  rm -f dist/pyspark*.tar.gz

  # --- before (master) ---
  git checkout master
  python packaging/$pkg/setup.py sdist
  mv dist/pyspark*.tar.gz dist/$pkg-before.tar.gz
  tar -tzf dist/$pkg-before.tar.gz | sort > dist/sdist-$pkg-before.txt

  # --- after (this branch) ---
  git checkout pyspark-manifest-cleanup
  python packaging/$pkg/setup.py sdist
  mv dist/pyspark*.tar.gz dist/$pkg-after.tar.gz
  tar -tzf dist/$pkg-after.tar.gz | sort > dist/sdist-$pkg-after.txt

  diff -u dist/sdist-$pkg-before.txt dist/sdist-$pkg-after.txt | tee dist/sdist-$pkg.diff || true
done

I also repeated this test but by calling ./dev/make-distribution.sh --pip instead of python ... sdist, since make-distribution.sh stages license and JAR files before building the sdist. Again, no differences.

set -euo pipefail

# from repo root
mkdir -p python/dist

# --- before (master) ---
git checkout master
./dev/make-distribution.sh --pip
mv python/dist/pyspark-*.tar.gz python/dist/classic-before.tar.gz
mv python/dist/pyspark_client-*.tar.gz python/dist/client-before.tar.gz
mv python/dist/pyspark_connect-*.tar.gz python/dist/connect-before.tar.gz

# --- after (this branch) ---
git checkout pyspark-manifest-cleanup
./dev/make-distribution.sh --pip
mv python/dist/pyspark-*.tar.gz python/dist/classic-after.tar.gz
mv python/dist/pyspark_client-*.tar.gz python/dist/client-after.tar.gz
mv python/dist/pyspark_connect-*.tar.gz python/dist/connect-after.tar.gz

for pkg in classic client connect; do
  echo "=== $pkg ==="
  tar -tzf python/dist/$pkg-before.tar.gz | sort > python/dist/sdist-$pkg-before.txt
  tar -tzf python/dist/$pkg-after.tar.gz | sort > python/dist/sdist-$pkg-after.txt
  diff -u python/dist/sdist-$pkg-before.txt python/dist/sdist-$pkg-after.txt \
    | tee python/dist/sdist-$pkg.diff || true
done

Our CI packaging tests are a bit slim. I've separately proposed #57645 to make them more comprehensive.

Was this patch authored or co-authored using generative AI tooling?

I wrote the test scripts with assistance from GitHub Copilot.

[uros-b]

LGTM