Skip to content

Reduced slow tasks and memory consumption in East_Asia#11595

Merged
micheles merged 15 commits into
masterfrom
genconcat
Jul 4, 2026
Merged

Reduced slow tasks and memory consumption in East_Asia#11595
micheles merged 15 commits into
masterfrom
genconcat

Conversation

@micheles

@micheles micheles commented Jul 4, 2026

Copy link
Copy Markdown
Contributor

Part of #11559. Here are the figures for Japan on engine192:

# before
| calc_12, maxmem=299.5 GB     | time_sec | memory_mb | counts    |
|------------------------------+----------+-----------+-----------|
| total ebrisk                 | 487_498  | 895.1     | 1_879     |
| computing risk               | 426_721  | 0.0       | 401_537   |
| aggregating losses           | 25_695   | 0.0       | 401_537   |
| EventBasedRiskCalculator.run | 12_305   | 1_721     | 1         |

# after
| calc_40, maxmem=307.4 GB     | time_sec | memory_mb | counts  |
|------------------------------+----------+-----------+---------|
| total ebrisk                 | 468_774  | 1_105     | 276     |
| computing risk               | 420_695  | 0.0       | 278_725 |
| aggregating losses           | 24_908   | 0.0       | 278_725 |
| EventBasedRiskCalculator.run | 12_412   | 1_827     | 1       |

The improvement is really minor for Japan since the slow tasks are the same:

| operation-duration | counts | mean  | stddev |   min | max    | slowfac |
|--------------------+--------+-------+--------+-------+--------+---------|
| ebrisk             |    159 | 3_058 |    87% | 323.5 | 12_059 |  3.9436 |
| ebrisk             |    157 | 2_976 |    89% | 288.6 | 12_182 |  4.0935 |

However, in small calculations, we can double the speed of the risk part, for instance for slovenia.zip:

| calc_7973, maxmem=9.8 GB     | time_sec | memory_mb | counts |
|------------------------------+----------+-----------+--------|
| computing risk               | 22.4886  | 0.0       | 4_407  |
| computing risk               | 12.8242  | 0.0       | 2_227  |

Taiwan is even better, 3x for the slowest task:

| operation-duration | counts | mean   | stddev | min     | max     | slowfac |
|--------------------+--------+--------+--------+---------+---------+---------|
| ebrisk             | 185    | 416.1  | 45%    | 43.1735 | 1_620   | 3.8938  |
| ebrisk             | 166    | 395.2  | 23%    | 40.1724 | 596.8   | 1.5099  |

Also, there is a substantial memory saving on small machines: on cassidy Japan runs using ~72 GB or RAM, while before it was going OOM.

@micheles micheles added this to the Engine 3.26.0 milestone Jul 4, 2026
@micheles micheles self-assigned this Jul 4, 2026
@micheles micheles merged commit 07ff505 into master Jul 4, 2026
4 checks passed
@micheles micheles deleted the genconcat branch July 4, 2026 06:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant