Skip to content

Fix simultaneous label remapping in mean_iou - #810

Open
feng1201 wants to merge 1 commit into
huggingface:mainfrom
feng1201:fix/mean-iou-label-remapping
Open

Fix simultaneous label remapping in mean_iou#810
feng1201 wants to merge 1 commit into
huggingface:mainfrom
feng1201:fix/mean-iou-label-remapping

Conversation

@feng1201

@feng1201 feng1201 commented Sep 6, 2026

Copy link
Copy Markdown

label_map currently runs before converting decoded images to NumPy arrays. Consequently, MeanIoU.compute raises TypeError: 'PngImageFile' object does not support item assignment for non-empty mappings. When the helper receives a NumPy array directly, successive assignments instead cascade and modify the caller's labels: {0: 1, 1: 0} turns [[0, 1]] into [[0, 0]] instead of swapping the classes.

Convert inputs to independent arrays first, then match every mapping against an unchanged copy of the original labels. Document the simultaneous mapping behavior. Label reduction and ignored-pixel handling remain in their existing order.

Reproduction from the repository root:

import evaluate
import numpy as np

metric = evaluate.load("./metrics/mean_iou")
result = metric.compute(
    predictions=[np.array([[1, 0]], dtype=np.uint8)],
    references=[np.array([[0, 1]], dtype=np.uint8)],
    num_labels=2, ignore_index=255, label_map={0: 1, 1: 0},
)
assert result["mean_iou"] == 1.0

Validation on Python 3.11.15, Datasets 2.21.0, NumPy 1.26.4 and Pillow 12.3.0:

  • New regression suite on unchanged source: 18 failed, 4 passed.
  • python -m pytest tests/test_mean_iou.py tests/test_metric_common.py -k mean_iou -q --tb=short: 23 passed, 1 remote test skipped by the repository's default settings.
  • make quality: passed across the repository.

The regressions cover public load(...).compute(...) calls with NumPy/PIL inputs, binary swaps, three-class cycles, mapping order, read-only arrays, caller input preservation, and label reduction/ignored pixels. Other framework-dependent metrics and remote tests were not run. Prepared with Codex assistance; the reported commands were executed locally.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant