[Nayor]

Summary

Four endpoints feeding the "Outings" and "Routes" map overlays of c2corg/c2c_ui#4574:

• GET /outings/map/heatmap and GET /routes/map/heatmap: density buckets ({x, y, count}, EPSG:3857) of a geohash grid aggregation on the search index. The size of the cells comes from the bbox on the server (a few pixels once on the screen), never from the client: a fine grid cannot be asked over a large area. The old zoom parameter is no longer needed (it is ignored).
• GET /outings/map/tracks and GET /routes/map/tracks: the tracks crossing the bbox, with a title in the preferred language and what tells them apart (activities, dates of an outing, the ratings that are set). With more than 100 tracks the answer is empty with truncated: true, and the UI asks the heatmap of the same box instead.
• They accept the same filters as the documents listing (act, date, r, w, a, u, ...), described in the OpenAPI document like the listing's, so the overlays can match a search made from the listing. They also accept ids=1,2,3 to restrict the answer to given documents (an empty list matches nothing).
• The bbox is cut to the world of the projection.
• No caching: the answers depend on free-form filters and must reflect new documents.
• TRACKS_OVERLAY_ENABLED in the environment of the container: set to false, 0, no or off, the four endpoints answer 503 before touching the database or the search index (read at each request). The UI reads the same variable, and turns the overlays off when it gets a 503.

How the tracks are found and sent

Following the review of the first version:

1. What has a track in the box comes from the database first (its GIST index): exactly, by the track and not by the point of the document, only the documents that have a track, and the box of the screen (no more box four times larger). Only the ids are read. This replaces the 1000-candidates cap of the search index, which counted the documents without a track and missed long tracks whose point was off-screen (false "too many" answers in busy areas).
2. The search index only filters those ids (an ids filter plus the filters of the request, unsorted), and is not asked at all when there are no filters (the Routes overlay of a route page, for instance). Up to 5000 ids are looked at.
3. The geometry is simplified and written by the database: ST_Simplify (about one pixel of the map, from the size of the box), 2D, ST_AsGeoJSON with the decimals the tolerance needs, no conversion in Python. The track in the database is not changed, and the document pages still give it as it is (the UI downloads that one for the GPX). On the development database, 44 tracks of a zone went from about 480 KB (3D, 15 digits) to 190 KB.
4. The titles are read without the other texts of the locales.

I chose this over indexing the envelope of the tracks in the search index: it has the same benefits (exact candidates, has_track) without changing the mapping and reindexing, which is a heavy operation on the current search index.

Not done: snapping the box to a tile grid so that a CDN can cache the answers.

Related: c2corg/c2c_ui#4573 (frontend counterpart and community context).

Test plan

• c2corg_api/tests/views/test_document_map.py and the whole c2corg_api/tests/contract folder pass in docker (330 tests): validation, heatmap counts and filters, the cells following the size of the box, the box cut to the world, tracks found by their track (not their point), only the documents with a track counted, no search without filters, simplification for the size of the box and 2D output, truncation, ids, ratings and dates, the 503 when turned off (with contract snapshots).
• Called from the UI of feat(map): add Tracks and Routes overlays to every map c2c_ui#4574 on the anonymized development database.
• Only these two test locations were run, not the whole suite; validation.py gained a validator.
• A look at the plan of the tracks query on production data (EXPLAIN (ANALYZE, BUFFERS)), as suggested in the review, would still be worth it.

🤖 Generated with Claude Code

[codacy-production]

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

🟢 Metrics 204 complexity · 0 duplication

Metric	Results
Complexity	204
Duplication	0

View in Codacy

_{NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.}

[lbesson]

@Nayor master moved a lot since this branch started: Python 3.15, SQLAlchemy 2.1, geoalchemy2 0.20, stricter ruff. The test classes are also no longer unittest.TestCase, so self.assertEqual and the other unittest assertions break.

To save you the merge, #2400 targets this branch. It merges master in, resolves the conflicts, converts the tests to plain assert and fixes the new lint findings. The full test suite passes on it. Merging it into your branch updates this PR. Feel free to adjust it, or redo the merge your own way if you prefer.

For new code on this branch: plain assert / pytest.raises / subtests in the tests; make lint (ruff check + ruff format) before pushing.

[eddy-geek]

LGTM.
We should merge + deploy this pr on the prod backend, so we can evaluate performance on real data easily.

@lbesson can you merge your merge? 😋

[lbesson]

i can't because #2231 is a draft. And it's probably better if @Nayor does it, since it's his worK?

[Nayor]

Hello,
Yes, it's a draft, still WIP, I haven't tested it yet, especially the UI ^^
If you want to test it feel free ;)

[eddy-geek]

Thanks for this, the two-stage design (ES filters → PostGIS geometry check) is sound. Some AI-suggested improvements to address, roughly by priority.

1. Reduce the tracks payload (main issue)

geom_detail is returned at full resolution, 3D/4D, ~15 significant digits, then converted in Python (geomet + geojson.dumps). I measured on a synthetic table (tracks of 1,500 XYZ points). The same 58 tracks are:

Encoding	Size
Current	3.6 MB
2D, 1 m precision	1.45 MB
ST_Simplify 5 m (~1 px at z15)	0.7 MB
ST_Simplify 19 m (~1 px at z13)	0.12 MB

At the 100-track limit this is 6+ MB per pan, per overlay. Real GPX tracks recorded every second are often longer than 1,500 points. Please do it in SQL and skip the Python conversion:

tol = (xmax - xmin) / 1500  # ~1 px for a typical viewport; or take `zoom` as a param
geojson = func.ST_AsGeoJSON(
    func.ST_Simplify(func.ST_Force2D(DocumentGeometry.geom_detail), tol), 0
)

2. Candidate selection in ES: false "truncated" answers and missed tracks

• The 1000-id cap counts documents without a track. Most outings have none, and the searched box is 4× the viewport. So in dense areas (Chamonix, Écrins, Valais) we'll often answer truncated while only a few dozen tracks are actually visible.
• A track is only found if its indexed point (track midpoint, or main waypoint for routes) is within the 2× box. A 15 km tour viewed at z15 is missed if its midpoint is more than ~2.5 km off-screen.

Suggested fix: index the track envelope in ES (4 float fields with range queries, or a geo_shape envelope) plus a has_track flag. Then filter on envelope-intersects-viewport and has_track. The candidate set becomes exact, _expanded_bbox can go, and the cap only triggers when there are really that many tracks. Also, the default outing sort (date_end desc) isn't needed for candidates; sorting by _doc would be cheaper.
Possible interim fix: when there are no search filters (e.g. the Routes overlay on a route page), skip ES and query PostGIS directly on the GIST index.

3. Heatmap cell size is controlled by the client

The comment says zoom prevents a fine-grained aggregation over a large area, but a client can send zoom=20 with a world bbox. Please derive the geohash precision from the bbox size on the server, or at least cap it relative to the bbox.

4. Smaller points

• Clamp the bbox to the EPSG:3857 world extent (±20037508.34) before reprojecting. Low-zoom extents can exceed it.
• get_locales_for_documents loads full locale rows (descriptions etc.) just for titles. Use load_only(document_id, lang, title, title_prefix).
• Optional: each viewport bbox is unique, so responses can't be cached by a CDN. Snapping the bbox to a tile grid (or z/x/y requests) would allow it.

FYI: indexing is fine

The existing GIST index idx_documents_geometries_geom_detail is used. EXPLAIN ANALYZE with 1000 candidate ids gives a bitmap index scan on it combined with the pkey scan, in ~6 ms. No new index needed. It would still be worth running the same EXPLAIN (ANALYZE, BUFFERS) once on prod data.

[eddy-geek]

Thanks for the rework. The PG-first approach does fix both earlier issues (long tracks with a far midpoint, trackless outings filling the cap). Two concerns that follow from it:

1. The "too many" path is slow

get_track_ids runs ST_Intersects with LIMIT 5001. In a dense or wide view, every candidate track has to be read and tested exactly. On a synthetic table (20k tracks of 1,500 points, warm cache):

Query	Time
Zoom-13 view, 61 tracks	~10 ms
Dense/wide view, hits the 5000 cap	330–430 ms
Same view with && (bbox only, index)	10–18 ms
Same view, LIMIT 101 (no filters case)	~8 ms

That slow path is the common one in busy areas, and all it returns is truncated, followed by a heatmap request.

Suggested fixes:

• Use the bbox operator for the candidate ids, and keep the exact test (if wanted) for the final ≤ tracks_limit() set in get_tracks:

  .filter(DocumentGeometry.geom_detail.op('&&')(_envelope(bbox_3857)))

• Without search filters ES isn't called, so only tracks_limit() + 1 ids are needed:

  limit = TRACKS_CANDIDATES_MAX if params else tracks_limit()
  document_ids = get_track_ids(model, bbox, limit, document_ids=_document_ids(request))

2. Selective filters are truncated because of unfiltered density

The 5000 cap applies before the ES filters. So on a route page (r=<id>), an outing page (r=…) or a user page (u=<id>), a view with more than 5000 tracks of any kind returns truncated even when only 20 match. The previous ES-first version handled this case.

Suggestion: when filters are given, first ask ES for the matching ids with the filters only (no bbox). Selective filters like r, u or w usually match few documents:

def _matching_ids(doc_type, params, cap):
    """The ids matching the filters (anywhere), or None if there are more than `cap`."""
    search = build_query(params, {'limit': cap, 'offset': 0}, doc_type).sort()
    response = search.execute()
    if response.hits.total > cap:
        return None
    return [int(doc.meta.id) for doc in response]

If it returns a list, pass it to get_track_ids(..., document_ids=...). The document_ids path already exists, the PG query is then exact and small, and the ES filter step afterwards isn't needed. If it returns None, fall back to the current PG-first path. Each case gets the order that suits it.

[eddy-geek]

Both points are fixed, and the tests cover each new path.

One remaining thing: with a broad filter (e.g. act=skitouring), _matching_ids fetches 5000 hits only to discard them because hits.total > cap. Checking the count first avoids that:

def _matching_ids(doc_type, params, cap):
    search = build_query(params, {'limit': 0, 'offset': 0}, doc_type).sort()
    if search.execute().hits.total > cap:
        return None
    return [int(doc.meta.id) for doc in search.extra(size=cap).execute()]

Nits:

• [i for i in matching if i in document_ids] checks membership in a list; use a set, or pass ids to ES as a filter.
• Truncation is now counted on track bounding boxes, so a view can occasionally be reported as truncated with ≤ limit real tracks. Fine, but worth a line in the docstring.