Skip to content

test: deletion strategy integration- testcontainers(MAPCO-10628) - #32

Merged
almog8k merged 16 commits into
masterfrom
test/deletion-strategy-integration
Sep 2, 2026
Merged

almog8k merged 16 commits into
masterfrom
test/deletion-strategy-integration

Conversation

@almog8k

@almog8k almog8k commented May 18, 2026

Copy link
Copy Markdown
Collaborator
Question Answer
Tests added ✔

This pull request adds a comprehensive integration test helper suite for S3 and filesystem-backed tile deletion strategies, introduces new test fakes for tile deletion, and improves test configuration and scripts. The main changes include new helper modules for Minio/S3 and FS test environments, utilities for generating tile ranges and paths, and updates to test scripts and dependencies.

Integration test infrastructure:

  • Added new helper modules for integration tests:
    • minioContainer.ts to spin up or connect to a Minio S3-compatible service for testing.
    • s3TestKit.ts for S3 bucket and object management, including bulk operations and client creation.
    • fsTestKit.ts for creating, writing, listing, and removing files and directories in a temporary filesystem.
    • backendFixtures.ts to orchestrate S3 and FS test environments, including provider wiring and teardown.
    • testPoller.ts to wire up the poller, strategies, and queue mocks for single-task test runs.
    • tileFixtures.ts for generating tile data, paths, and extra test tiles to verify correct deletion.

Test fakes and utilities:

  • Added tilesDeletionFakes.ts for generating fake tile deletion parameters and ranges for testing.
  • Updated tests/helpers/fakes/index.ts to export the new tile deletion fakes.

Test configuration and scripts:

  • Updated package.json:
    • Enhanced the test script to run both unit and integration tests, and added a dedicated test:integration script.
    • Added testcontainers as a dev dependency for running Minio in tests.
  • Updated eslint.config.mjs to ignore the new integration test config file.

Other improvements:

  • Made S3_MAX_DELETE_BATCH a named export for use in test helpers. [1] [2]
  • Added gisDomain: raster to Helm values for improved chart configuration.

These changes lay the groundwork for robust integration testing of storage provider strategies, ensuring correctness across S3 and filesystem backends.

@almog8k almog8k changed the title test: deletion strategy integration- testcontainers test: deletion strategy integration- testcontainers(MAPCO-10628) May 18, 2026
@github-actions

github-actions Bot commented May 18, 2026 •

Copy link
Copy Markdown

🎫 Related Jira Issue: MAPCO-10628

Comment thread helm/values.yaml
Comment thread tests/helpers/fakes/tilesDeletionFakes.ts
Comment thread tests/integration/helpers/fsTestKit.ts
Comment thread tests/integration/helpers/fsTestKit.ts
Comment thread tests/integration/helpers/minioContainer.ts Outdated
Comment thread tests/integration/helpers/testPoller.ts
Comment thread tests/integration/helpers/testPoller.ts
Comment thread tests/integration/helpers/tileFixtures.ts
Comment thread tests/integration/tilesDeletionStrategy.integration.spec.ts
Comment thread tests/integration/tilesDeletionStrategy.integration.spec.ts
Comment thread package.json Outdated
const fsBasePath = await makeTempFsBase();
const fsSubPath = FS_ALLOWED_SUB_PATH;
/** Per-provider delete batch sizes; each falls back to the production-shaped default. */
interface ProviderBatchSizes {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is dead surface — and its own doc admits it. I confirmed in fsStorageProvider.ts:44-54: FS delete does Promise.allSettled(paths.map(unlink)) — it fans every path out at once, no getChunk, no batchSize. So the fs? knob provably cannot affect the tiles-deletion path these tests exercise, and no test ever passes it (the batching test passes only { s3: ... }). A field whose documentation explains why it does nothing is a YAGNI smell — I'd drop fs? from ProviderBatchSizes and just build the FS provider with FS_DELETE_BATCH_SIZE.

@almog8k
almog8k merged commit d25dd5d into master Sep 2, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants