Skip to content

Reduced max_assets_chunk to reduce the memory occupation#11589

Merged
micheles merged 6 commits into
masterfrom
more_mem
Jul 3, 2026
Merged

Reduced max_assets_chunk to reduce the memory occupation#11589
micheles merged 6 commits into
masterfrom
more_mem

Conversation

@micheles

@micheles micheles commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Plus other minor changes. The effect is minor oin Taiwan:

# master
$ OQ_SAMPLE_ASSETS=.01 oq engine --run job_Taiwan.ini --hc ../SES.hdf5 
| calc_13, maxmem=183.6 GB     | time_sec | memory_mb | counts  |
|------------------------------+----------+-----------+---------|
| total ebrisk                 | 9_554    | 554.8     | 2_001   |
| computing risk               | 4_716    | 0.0       | 118_293 |
| reading sites                | 1_208    | 184.3     | 188     |
| reading ruptures             | 826.4    | 0.0       | 2_047   |
| EventBasedRiskCalculator.run | 537.2    | 691.2     | 1       |
| reading crmodel              | 501.9    | 369.0     | 188     |

| operation-duration | counts | mean    | stddev | min    | max    | slowfac |
|--------------------+--------+---------+--------+--------+--------+---------|
| ebrisk             | 188    | 50.9786 | 94%    | 9.1446 | 448.2  | 8.7914  |

# more_mem
$ OQ_SAMPLE_ASSETS=.01 oq engine --run job_Taiwan.ini --hc ../SES.hdf5 
| calc_14, maxmem=183.6 GB     | time_sec | memory_mb | counts  |
|------------------------------+----------+-----------+---------|
| total ebrisk                 | 9_204    | 554.7     | 2_001   |
| computing risk               | 4_720    | 0.0       | 118_292 |
| reading sites                | 888.1    | 184.2     | 188     |
| reading ruptures             | 723.0    | 0.0       | 2_047   |
| EventBasedRiskCalculator.run | 534.6    | 711.1     | 1       |
| reading crmodel              | 502.6    | 369.0     | 188     |

| operation-duration | counts | mean    | stddev | min    | max    | slowfac |
|--------------------+--------+---------+--------+--------+--------+---------|
| ebrisk             | 188    | 49.1032 | 99%    | 9.2018 | 446.7  | 9.0977  |

The improvement is in reading from the datastore since we are reading a bit less thanks to the introduction of

pairs = [(idx0taxo, slice(s0, s1))
                 for idx0taxo, s0, s1 in monitor.read('start-stop')]

moved outside the loop.

@micheles micheles added this to the Engine 3.26.0 milestone Jul 3, 2026
@micheles micheles self-assigned this Jul 3, 2026
@micheles micheles merged commit ff9b3b4 into master Jul 3, 2026
4 checks passed
@micheles micheles deleted the more_mem branch July 3, 2026 07:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant