Repository navigation
Conversation
download_file (util/files_handler.py) and Client._download
(util/api_client.py) derived the local file name with
url.split("/")[-1], so a query string or fragment became part of the
file name. On Windows the resulting name (e.g.
"GMT_intermediate_coast_distance_01d.zip?download=1" for the
distance-to-coast dataset) is invalid and Centroids.get_dist_coast
raised OSError: [Errno 22].
Both sites now derive the name from the URL path only, via
PurePosixPath(urlsplit(url).path).name. Client._download keeps its
percent-decoding (unquote) of the file name; files_handler keeps not
decoding it, as before. In files_handler, a URL whose path ends in
"/" (or has no file name at all) still raises the documented
ValueError, as the previous string split did for directory URLs.
Adds unit tests with mocked requests for both sites: plain URL
(regression), query string, fragment, percent-encoded name
(api_client), and file-name-less URLs (files_handler).
Fixes CLIMADA-project#1336
|
A note on the failing Read the Docs check, since the failure is not in this change: in the build log for this PR (https://app.readthedocs.org/projects/climada-python/builds/35502568/), every step through environment creation and raised from So this check should go green on a rebuild once the RTD extension / Sphinx 9 combination is resolved on the docs side; there is nothing in this change set that can address it. Disclosure: this comment was written by Árbol, an AI assistant working under the direction of the account holder, who reviewed it before posting. |
Changes proposed in this PR:
PurePosixPath(urlsplit(url).path).name) indownload_file(climada/util/files_handler.py) andClient._download(climada/util/api_client.py), so a query string or fragment no longer becomes part of the downloaded file name.This PR fixes #1336
Summary
Fixes #1336.
download_file(climada/util/files_handler.py) andClient._download(climada/util/api_client.py) derived the local file name withurl.split("/")[-1], so a query string or fragment became part of the file name — e.g. the distance-to-coast dataset downloaded asGMT_intermediate_coast_distance_01d.zip?download=1, which is an invalid file name on Windows and madeCentroids.get_dist_coastraiseOSError: [Errno 22].Both sites now derive the name from the URL path only:
Fix layer
The two sites do not share a helper today, and their post-processing differs, so each site derives the name locally with the same expression (the shape proposed in #1336):
files_handler.download_filenever percent-decoded the name and still does not; it only stops treating the query/fragment as part of it.Client._downloadkeeps its existingunquote(...)of the derived name, so percent-encoded names decode exactly as before.Behavior preserved / changed, explicitly:
api_client, left encoded infiles_handler, as before).files_handler: a URL whose path ends in/, or has no file name at all, still raises the documentedValueError("cannot download … as a file") — the previous string split rejected directory URLs the same way, and this is preserved even when a query string is present.?queryor#fragmentnow yield that segment — that is the bug fix.One related spot was left alone on purpose:
download_ftpinfiles_handler.pyusesurl.split("/")[-1]only as a tqdm progress-bar description, not as a file name.Tests
New unit tests with mocked
requests.get(no network, no data downloads):climada/util/test/test_files.py— newTestDownloadFileNameclass: plain URL regression, query string (the issue's coast-distance file name), fragment, andValueErrorfor file-name-less URLs (directory + query, trailing slash, host-only).climada/test/test_api_client.py—TestClient.test_download_file_name_from_url: plain URL regression, query string, fragment, and percent-encoded name still decoded.Test-execution note, stated plainly: my local sandbox's package-index access degraded badly while preparing this (repeated proxy connection resets), so I could not finish installing CLIMADA's full dependency set to run the two modules under pytest yet. What I did run, against the real patched code: the new
files_handlerlogic end-to-end with mockedrequests.get(all cases above, 11/11 checks pass, including the issue's exact file name) and theapi_clientderivation expression verbatim. The mocked pytest modules are written to run in CI; if CI flags anything in them I will fix it immediately.Disclosure
Disclosure: this contribution was written by Árbol, an AI assistant working under the direction of the account holder, who set the acceptance criteria and verified the results before submission.
PR Author Checklist
develop)PR Reviewer Checklist