Submission failed: "scoring_hostname-worker9"

[Zty031220]

Dear Codabench team

How can this problem be solved?

[Didayolo]

Hi @Zty031220,

Can you please share more information?

• What is your username?
• The URL of the competition?
• What logs can you see when you click on the submission in this table? It will open a panel with different log tabs such as Scoring errors, Ingestion output, etc.

[Zty031220]

Hi @Zty031220,

Can you please share more information?

• What is your username?
• The URL of the competition?
• What logs can you see when you click on the submission in this table? It will open a panel with different log tabs such as Scoring errors, Ingestion output, etc.

Username is zty_77
The URL of the competition is https://www.codabench.org/competitions/13553

Only this interface has content.

[Didayolo]

Thank you for the additional information. I'm investigating the problem.

[Didayolo]

Error log:

2026-02-27 10:53:37.573 | INFO     | compute_worker:_get_container_image:509 - Running pull for image: george200150/ntire26-cvlai:latest
2026-02-27 10:54:15.334 | ERROR    | compute_worker:show_progress:137 - There was an error showing the progress bar
2026-02-27 10:54:15.334 | ERROR    | compute_worker:show_progress:138 - 'status'
[Extraction complete  20043066d3d5]                       ━━━━━━━━━ 100% 0:00:00
[Extraction complete  5313dfe63e12]                       ━━━━━━━━━ 100% 0:00:00
[Downloading 984c22805ebc]                                ━━━━━━━━━ 100% 0:00:00
[Download complete, waiting for extraction  4f4fb700ef54] ━━━━━━━━━ 100% 0:00:00
[Download complete, waiting for extraction  bb0cb655ac41] ━━━━━━━━━ 100% 0:00:00
[Download complete, waiting for extraction  eaffb8580325] ━━━━━━━━━ 100% 0:00:00
[Downloading cef9830f5f21]                                ━━━━━━━━━ 100% 0:00:00
[Download complete, waiting for extraction  3eedb78f5619] ━━━━━━━━━ 100% 0:00:00
[Download complete, waiting for extraction  a65c2444bb5f] ━━━━━━━━━ 100% 0:00:00
2026-02-27 10:54:15.340 | INFO     | compute_worker:_update_submission:478 - Updating submission @ https://www.codabench.org/api/submissions/569340/ with data = {'status': 'Running', 'status_details': 'ingestion_hostname-worker7', 'secret': '9e86d8d6-5361-4d4c-9f39-4b5b00629f6f'}
2026-02-27 10:54:15.901 | INFO     | compute_worker:_update_submission:482 - Submission updated successfully!
2026-02-27 10:54:15.901 | INFO     | compute_worker:start:1125 - Running scoring program, and then ingestion program
2026-02-27 10:54:15.901 | WARNING  | compute_worker:_run_program_directory:819 - Program directory missing metadata, assuming it's going to be handled by ingestion
2026-02-27 10:54:16.691 | INFO     | compute_worker:_run_program_directory:831 - Metadata path is /codabench/tmpks92fl02/ingestion_program/metadata
2026-02-27 10:54:16.692 | INFO     | compute_worker:_run_program_directory:835 - Metadata contains:
 {'command': 'python $ingestion_program/ingestion.py $input $output $ingestion_program $submission_program'}
2026-02-27 10:54:16.692 | INFO     | compute_worker:_run_program_directory:975 - Running container with command python /app/program/ingestion.py /app/input_data /app/output /app/program /app/ingested_program
2026-02-27 10:54:16.696 | INFO     | compute_worker:clean_up:1292 - Destroying submission temp dir: /codabench/tmpks92fl02
2026-02-27 10:54:17.018 | ERROR    | celery.app.trace:_log_error:265 - Task compute_worker_run[62d6d899-dbbb-46ca-a560-c1aab67b1390] raised unexpected: ImageNotFound(HTTPError('404 Client Error: Not Found for url: http+docker://localhost/v1.47/containers/create?name=01936368-2b4a-4289-8e07-2bc997163116'))
Traceback (most recent call last):

  File "/usr/lib/python3.9/site-packages/docker/api/client.py", line 275, in _raise_for_status
    response.raise_for_status()
    │        └ <function Response.raise_for_status at 0x7f1edccaf9d0>
    └ <Response [404]>
  File "/usr/lib/python3.9/site-packages/requests/models.py", line 1026, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
          │         │                        └ <Response [404]>
          │         └ '404 Client Error: Not Found for url: http+docker://localhost/v1.47/containers/create?name=01936368-2b4a-4289-8e07-2bc997163116'
          └ <class 'requests.exceptions.HTTPError'>

requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.47/containers/create?name=01936368-2b4a-4289-8e07-2bc997163116


The above exception was the direct cause of the following exception:


Traceback (most recent call last):

> File "/usr/lib/python3.9/site-packages/celery/app/trace.py", line 451, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/usr/lib/python3.9/site-packages/celery/app/trace.py", line 734, in __protected_call__
    return self.run(*args, **kwargs)

  File "//compute_worker.py", line 222, in run_wrapper
    run.start()

  File "//compute_worker.py", line 1137, in start
    loop.run_until_complete(gathered_tasks)

  File "/usr/lib64/python3.9/asyncio/base_events.py", line 647, in run_until_complete
    return future.result()

  File "//compute_worker.py", line 981, in _run_program_directory
    container = client.create_container(

  File "/usr/lib/python3.9/site-packages/docker/api/container.py", line 440, in create_container
    return self.create_container_from_config(config, name, platform)
  File "/usr/lib/python3.9/site-packages/docker/api/container.py", line 457, in create_container_from_config
    return self._result(res, True)
  File "/usr/lib/python3.9/site-packages/docker/api/client.py", line 281, in _result
    self._raise_for_status(response)
  File "/usr/lib/python3.9/site-packages/docker/api/client.py", line 277, in _raise_for_status
    raise create_api_error_from_http_exception(e) from e
  File "/usr/lib/python3.9/site-packages/docker/errors.py", line 39, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation) from e

docker.errors.ImageNotFound: 404 Client Error for http+docker://localhost/v1.47/containers/create?name=01936368-2b4a-4289-8e07-2bc997163116: Not Found ("No such image: george200150/ntire26-cvlai:latest")

[Didayolo]

Hi @Zty031220,

Looks like the docker image is large, and the compute workers are hitting storage limit frequently. We did manual cleanup but that may still occur. I'll continue investigating the problem.

[Didayolo]

Re-trying your last submission after the cleanup, I get the following error logs:

EVALUATION FAILED: ref/ and res/ PNG filenames mismatch and could not be paired.
Missing in res (first 20): ['0000001.png', '0000002.png', '0000003.png', '0000004.png', '0000005.png', '0000006.png', '0000007.png', '0000008.png', '0000009.png', '0000010.png', '0000011.png', '0000012.png', '0000013.png', '0000014.png', '0000015.png', '0000016.png', '0000017.png', '0000018.png', '0000019.png', '0000020.png']
Extra in res (first 20): []
Counts: ref=200 res=0

Traceback (most recent call last):
  File "/app/program/evaluation.py", line 358, in <module>
    main()
  File "/app/program/evaluation.py", line 318, in main
    pairs = build_pairs(ref_pngs, res_pngs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/program/evaluation.py", line 277, in build_pairs
    raise RuntimeError(
RuntimeError: ref/ and res/ PNG filenames mismatch and could not be paired.
Missing in res (first 20): ['0000001.png', '0000002.png', '0000003.png', '0000004.png', '0000005.png', '0000006.png', '0000007.png', '0000008.png', '0000009.png', '0000010.png', '0000011.png', '0000012.png', '0000013.png', '0000014.png', '0000015.png', '0000016.png', '0000017.png', '0000018.png', '0000019.png', '0000020.png']
Extra in res (first 20): []
Counts: ref=200 res=0