No files generated by monai auto 3d seg during infrence #8045
Description
Describe the bug
I had run the auto 3d seg on private dataset and i have no files in results folder of the test set.
To Reproduce
run code
runner = AutoRunner(
work_dir=work_dir,
input={
"modality": "MRI",
"datalist": datalist_path,
"dataroot": work_dir },
)
runner.run()
in datalist path the path to the file below is passed
{"training": [{"fold": 0, "image": "/workspaces/prostate_opi_year_3/data/dataset/post_norm_min_max_bias_on/1188/1188_adc.nii.gz", "label":
...
"/workspaces/prostate_opi_year_3/data/dataset/post_norm_min_max_bias_on/1218/1218_lesion_union_adc.nii.gz"}], "testing": [{"image": "/workspaces/prostate_opi_year_3/data/dataset/post_norm_min_max_bias_on/3322/3322_adc.nii.gz", "label": "/workspaces/prostate_opi_year_3/data/dataset/post_norm_min_max_bias_on/3322/3322_lesion_union_adc.nii.gz"}, {"image": "/workspaces/prostate_opi_year_3/data/dataset/post_norm_min_max_bias_on/066/66_adc.nii.gz", "label":
...
, "label": "/workspaces/prostate_opi_year_3/data/dataset/post_norm_min_max_bias_on/523/523_lesion_union_adc.nii.gz"}], "modality": "MRI"}
Then I get the information that auto3d seg completed succesfully and files from test set will be in ensemble_output - but no files are present there (although folder gets created)
2024/08/22 19:40:47 INFO mlflow.tracking.fluent: Experiment with name 'Auto3DSeg' does not exist. Creating a new experiment.
dints_3 - training ...: 100%|███████████████████████████████████| 40/40 [9:24:19<00:00, 846.48s/round]
dints_3 - validation at original spacing/resolution
2024-08-23 05:16:03,679 - WARNING - dints_3 - training: finished
2024-08-23 05:16:06,920 - INFO - ['python', '/workspaces/prostate_opi_year_3/data/work_dir_b/dints_4/scripts/train.py', 'run', "--config_file='/workspaces/prostate_opi_year_3/data/work_dir_b/dints_4/configs/hyper_parameters.yaml,/workspaces/prostate_opi_year_3/data/work_dir_b/dints_4/configs/hyper_parameters_search.yaml,/workspaces/prostate_opi_year_3/data/work_dir_b/dints_4/configs/network.yaml,/workspaces/prostate_opi_year_3/data/work_dir_b/dints_4/configs/network_search.yaml,/workspaces/prostate_opi_year_3/data/work_dir_b/dints_4/configs/transforms_infer.yaml,/workspaces/prostate_opi_year_3/data/work_dir_b/dints_4/configs/transforms_train.yaml,/workspaces/prostate_opi_year_3/data/work_dir_b/dints_4/configs/transforms_validate.yaml'"]
2024/08/23 05:16:16 INFO mlflow.tracking.fluent: Experiment with name 'Auto3DSeg' does not exist. Creating a new experiment.
dints_4 - training ...: 100%|███████████████████████████████████| 40/40 [9:25:59<00:00, 849.00s/round]
dints_4 - validation at original spacing/resolution
2024-08-23 14:53:05,172 - WARNING - dints_4 - training: finished
2024-08-23 14:53:08,393 - INFO - ['python', '/workspaces/prostate_opi_year_3/data/work_dir_b/segresnet2d_0/scripts/train.py', 'run', "--config_file='/workspaces/prostate_opi_year_3/data/work_dir_b/segresnet2d_0/configs/hyper_parameters.yaml'"]
2024/08/23 14:53:16 INFO mlflow.tracking.fluent: Experiment with name 'Auto3DSeg' does not exist. Creating a new experiment.
segresnet2d_0 - training: 100%|████████████████████████████████| 300/300 [3:29:16<00:00, 41.86s/epoch]
2024-08-23 18:22:35,724 - INFO - ['python', '/workspaces/prostate_opi_year_3/data/work_dir_b/segresnet2d_1/scripts/train.py', 'run', "--config_file='/workspaces/prostate_opi_year_3/data/work_dir_b/segresnet2d_1/configs/hyper_parameters.yaml'"]
2024/08/23 18:22:43 INFO mlflow.tracking.fluent: Experiment with name 'Auto3DSeg' does not exist. Creating a new experiment.
segresnet2d_1 - training: 100%|████████████████████████████████| 300/300 [3:30:26<00:00, 42.09s/epoch]
2024-08-23 21:53:12,197 - INFO - ['python', '/workspaces/prostate_opi_year_3/data/work_dir_b/segresnet2d_2/scripts/train.py', 'run', "--config_file='/workspaces/prostate_opi_year_3/data/work_dir_b/segresnet2d_2/configs/hyper_parameters.yaml'"]
2024/08/23 21:53:19 INFO mlflow.tracking.fluent: Experiment with name 'Auto3DSeg' does not exist. Creating a new experiment.
segresnet2d_2 - training: 100%|████████████████████████████████| 300/300 [3:30:25<00:00, 42.08s/epoch]
2024-08-24 01:23:47,636 - INFO - ['python', '/workspaces/prostate_opi_year_3/data/work_dir_b/segresnet2d_3/scripts/train.py', 'run', "--config_file='/workspaces/prostate_opi_year_3/data/work_dir_b/segresnet2d_3/configs/hyper_parameters.yaml'"]
2024/08/24 01:23:54 INFO mlflow.tracking.fluent: Experiment with name 'Auto3DSeg' does not exist. Creating a new experiment.
segresnet2d_3 - training: 100%|████████████████████████████████| 300/300 [3:31:10<00:00, 42.23s/epoch]
2024-08-24 04:55:07,927 - INFO - ['python', '/workspaces/prostate_opi_year_3/data/work_dir_b/segresnet2d_4/scripts/train.py', 'run', "--config_file='/workspaces/prostate_opi_year_3/data/work_dir_b/segresnet2d_4/configs/hyper_parameters.yaml'"]
2024/08/24 04:55:15 INFO mlflow.tracking.fluent: Experiment with name 'Auto3DSeg' does not exist. Creating a new experiment.
segresnet2d_4 - training: 100%|████████████████████████████████| 300/300 [3:31:01<00:00, 42.21s/epoch]
2024-08-24 08:26:19,707 - INFO - ['python', '/workspaces/prostate_opi_year_3/data/work_dir_b/segresnet_0/scripts/train.py', 'run', "--config_file='/workspaces/prostate_opi_year_3/data/work_dir_b/segresnet_0/configs/hyper_parameters.yaml'"]
`apex.normalization.InstanceNorm3dNVFuser` is not installed properly, use nn.InstanceNorm3d instead.
2024/08/24 08:26:27 INFO mlflow.tracking.fluent: Experiment with name 'Auto3DSeg' does not exist. Creating a new experiment.
segresnet_0 - training: 89%|███████████████████████████▋ | 268/300 [9:30:51<1 segresnet_0 - training: 90%|███████████████████████████▊ | 269/300 [9:32:53<1 segresnet_0 - training: 90%|███████████████████████████▉ | 270/300 [9:34:55<1 segresnet_0 - training: 90%|████████████████████████████ | 271/300 [9:36:58<1 segresnet_0 - training: 91%|████████████████████████████ | 272/300 [9:39:43<1 segresnet_0 - training: 91%|████████████████████████████▏ | 273/300 [9:41:45<1 segresnet_0 - training: 91%|██████████████████████████████▏ | 274/300 [9:43:47 segresnet_0 - training: 92%|██████████████████████████████▎ | 275/300 [9:45:49 segresnet_0 - training: 92%|██████████████████████████████▎ | 276/300 [9:48:34 segresnet_0 - training: 92%|██████████████████████████████▍ | 277/300 [9:50:36 segresnet_0 - training: 93%|██████████████████████████████▌ | 278/300 [9:52:38 segresnet_0 - training: 93%|██████████████████████████████▋ | 279/300 [9:54:41<44segresnet_0 - training: 93%|██████████████████████████████▊ | 280/300 [9:57:26<46segresnet_0 - training: 94%|██████████████████████████████▉ | 281/300 [9:59:29<42segresnet_0 - training: 94%|██████████████████████████████ | 282/300 [10:01:32<39segresnet_0 - training: 94%|██████████████████████████████▏ | 283/300 [10:04:18<40segresnet_0 - training: 95%|██████████████████████████████▎ | 284/300 [10:06:21<36segresnet_0 - training: 95%|██████████████████████████████▍ | 285/300 [10:08:23<32segresnet_0 - training: 95%|██████████████████████████████▌ | 286/300 [10:11:08<33segresnet_0 - training: 96%|██████████████████████████████▌ | 287/300 [10:13:11<29segresnet_0 - training: 96%|██████████████████████████████▋ | 288/300 [10:15:13<26segresnet_0 - training: 96%|██████████████████████████████▊ | 289/300 [10:17:58<26segresnet_0 - training: 97%|██████████████████████████████▉ | 290/300 [10:20:01<22segresnet_0 - training: 97%|███████████████████████████████ | 291/300 [10:22:46<21segresnet_0 - training: 97%|███████████████████████████████▏| 292/300 [10:24:49<18segresnet_0 - training: 98%|███████████████████████████████▎| 293/300 [10:27:34<17segresnet_0 - training: 98%|███████████████████████████████▎| 294/300 [10:29:37<13segresnet_0 - training: 98%|███████████████████████████████▍| 295/300 [10:32:22<12segresnet_0 - training: 99%|███████████████████████████████▌| 296/300 [10:34:25<09segresnet_0 - training: 99%|███████████████████████████████▋| 297/300 [10:37:10<07segresnet_0 - training: 99%|███████████████████████████████▊| 298/300 [10:39:56<05segresnet_0 - training: 100%|███████████████████████████████▉| 299/300 [10:42:41<02segresnet_0 - training: 100%|████████████████████████████████| 300/300 [10:45:26<00segresnet_0 - training: 100%|████████████████████████████████| 300/300 [10:45:26<00:00, 129.09s/epoch]
`apex.normalization.InstanceNorm3dNVFuser` is not installed properly, use nn.InstanceNorm3d instead.
2024-08-24 19:12:42,797 - INFO - ['python', '/workspaces/prostate_opi_year_3/data/work_dir_b/segresnet_1/scripts/train.py', 'run', "--config_file='/workspaces/prostate_opi_year_3/data/work_dir_b/segresnet_1/configs/hyper_parameters.yaml'"]
`apex.normalization.InstanceNorm3dNVFuser` is not installed properly, use nn.InstanceNorm3d instead.
2024/08/24 19:12:50 INFO mlflow.tracking.fluent: Experiment with name 'Auto3DSeg' does not exist. Creating a new experiment.
segresnet_1 - training: 100%|█████████████| 300/300 [10:48:35<00:00, 129.72s/epoch]
`apex.normalization.InstanceNorm3dNVFuser` is not installed properly, use nn.InstanceNorm3d instead.
2024-08-25 06:02:15,986 - INFO - ['python', '/workspaces/prostate_opi_year_3/data/work_dir_b/segresnet_2/scripts/train.py', 'run', "--config_file='/workspaces/prostate_opi_year_3/data/work_dir_b/segresnet_2/configs/hyper_parameters.yaml'"]
`apex.normalization.InstanceNorm3dNVFuser` is not installed properly, use nn.InstanceNorm3d instead.
2024/08/25 06:02:24 INFO mlflow.tracking.fluent: Experiment with name 'Auto3DSeg' does not exist. Creating a new experiment.
segresnet_2 - training: 100%|█████████████| 300/300 [10:51:15<00:00, 130.25s/epoch]
`apex.normalization.InstanceNorm3dNVFuser` is not installed properly, use nn.InstanceNorm3d instead.
2024-08-25 16:54:29,743 - INFO - ['python', '/workspaces/prostate_opi_year_3/data/work_dir_b/segresnet_3/scripts/train.py', 'run', "--config_file='/workspaces/prostate_opi_year_3/data/work_dir_b/segresnet_3/configs/hyper_parameters.yaml'"]
`apex.normalization.InstanceNorm3dNVFuser` is not installed properly, use nn.InstanceNorm3d instead.
2024/08/25 16:54:37 INFO mlflow.tracking.fluent: Experiment with name 'Auto3DSeg' does not exist. Creating a new experiment.
segresnet_3 - training: 100%|████████████████████████████████████████████████████████████████████████████████████| 300/300 [10:51:07<00:00, 130.23s/epoch]
`apex.normalization.InstanceNorm3dNVFuser` is not installed properly, use nn.InstanceNorm3d instead.
2024-08-26 03:46:35,712 - INFO - ['python', '/workspaces/prostate_opi_year_3/data/work_dir_b/segresnet_4/scripts/train.py', 'run', "--config_file='/workspaces/prostate_opi_year_3/data/work_dir_b/segresnet_4/configs/hyper_parameters.yaml'"]
`apex.normalization.InstanceNorm3dNVFuser` is not installed properly, use nn.InstanceNorm3d instead.
2024/08/26 03:46:43 INFO mlflow.tracking.fluent: Experiment with name 'Auto3DSeg' does not exist. Creating a new experiment.
segresnet_4 - training: 100%|████████████████████████████████████████████████████████████████████████████████████| 300/300 [10:50:27<00:00, 130.09s/epoch]
`apex.normalization.InstanceNorm3dNVFuser` is not installed properly, use nn.InstanceNorm3d instead.
2024-08-26 14:38:00,553 - INFO - ['python', '/workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_0/scripts/train.py', 'run', "--config_file='/workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_0/configs/hyper_parameters.yaml,/workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_0/configs/network.yaml,/workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_0/configs/transforms_infer.yaml,/workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_0/configs/transforms_train.yaml,/workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_0/configs/transforms_validate.yaml'"]
monai.networks.nets.swin_unetr SwinUNETR.__init__:img_size: Argument `img_size` has been deprecated since version 1.3. It will be removed in version 1.5. The img_size argument is not required anymore and checks on the input size are run during forward().
2024-08-26 14:38:10,744 - INFO - Downloaded: /workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_0/pretrained_model/swin_unetr.base_5000ep_f48_lr2e-4_pretrained.pt
2024-08-26 14:38:10,744 - INFO - Expected md5 is None, skip md5 check for file /workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_0/pretrained_model/swin_unetr.base_5000ep_f48_lr2e-4_pretrained.pt.
2024/08/26 14:38:11 INFO mlflow.tracking.fluent: Experiment with name 'Auto3DSeg' does not exist. Creating a new experiment.
swinunetr_0 - training ...: 6%|████▌ | 7/123 [1:38:31<27:12:43, 844.51s/round]
2024-08-26 16:23:58,012 - WARNING - swinunetr_0 - training: finished with early stop
2024-08-26 16:23:59,986 - INFO - ['python', '/workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_1/scripts/train.py', 'run', "--config_file='/workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_1/configs/hyper_parameters.yaml,/workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_1/configs/network.yaml,/workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_1/configs/transforms_infer.yaml,/workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_1/configs/transforms_train.yaml,/workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_1/configs/transforms_validate.yaml'"]
monai.networks.nets.swin_unetr SwinUNETR.__init__:img_size: Argument `img_size` has been deprecated since version 1.3. It will be removed in version 1.5. The img_size argument is not required anymore and checks on the input size are run during forward().
2024-08-26 16:24:25,515 - INFO - Downloaded: /workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_1/pretrained_model/swin_unetr.base_5000ep_f48_lr2e-4_pretrained.pt
2024-08-26 16:24:25,516 - INFO - Expected md5 is None, skip md5 check for file /workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_1/pretrained_model/swin_unetr.base_5000ep_f48_lr2e-4_pretrained.pt.
2024/08/26 16:24:25 INFO mlflow.tracking.fluent: Experiment with name 'Auto3DSeg' does not exist. Creating a new experiment.
swinunetr_1 - training ...: 12%|█████████▋ | 15/123 [3:16:50<23:37:13, 787.35s/round]
2024-08-26 19:48:49,743 - WARNING - swinunetr_1 - training: finished with early stop
2024-08-26 19:48:51,829 - INFO - ['python', '/workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_2/scripts/train.py', 'run', "--config_file='/workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_2/configs/hyper_parameters.yaml,/workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_2/configs/network.yaml,/workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_2/configs/transforms_infer.yaml,/workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_2/configs/transforms_train.yaml,/workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_2/configs/transforms_validate.yaml'"]
monai.networks.nets.swin_unetr SwinUNETR.__init__:img_size: Argument `img_size` has been deprecated since version 1.3. It will be removed in version 1.5. The img_size argument is not required anymore and checks on the input size are run during forward().
2024-08-26 19:49:01,723 - INFO - Downloaded: /workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_2/pretrained_model/swin_unetr.base_5000ep_f48_lr2e-4_pretrained.pt
2024-08-26 19:49:01,724 - INFO - Expected md5 is None, skip md5 check for file /workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_2/pretrained_model/swin_unetr.base_5000ep_f48_lr2e-4_pretrained.pt.
2024/08/26 19:49:01 INFO mlflow.tracking.fluent: Experiment with name 'Auto3DSeg' does not exist. Creating a new experiment.
swinunetr_2 - training ...: 11%|████████▉ | 14/123 [3:05:00<24:00:27, 792.91s/round]
2024-08-26 23:01:44,110 - WARNING - swinunetr_2 - training: finished with early stop
2024-08-26 23:01:46,219 - INFO - ['python', '/workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_3/scripts/train.py', 'run', "--config_file='/workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_3/configs/hyper_parameters.yaml,/workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_3/configs/network.yaml,/workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_3/configs/transforms_infer.yaml,/workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_3/configs/transforms_train.yaml,/workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_3/configs/transforms_validate.yaml'"]
monai.networks.nets.swin_unetr SwinUNETR.__init__:img_size: Argument `img_size` has been deprecated since version 1.3. It will be removed in version 1.5. The img_size argument is not required anymore and checks on the input size are run during forward().
2024-08-26 23:01:55,547 - INFO - Downloaded: /workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_3/pretrained_model/swin_unetr.base_5000ep_f48_lr2e-4_pretrained.pt
2024-08-26 23:01:55,547 - INFO - Expected md5 is None, skip md5 check for file /workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_3/pretrained_model/swin_unetr.base_5000ep_f48_lr2e-4_pretrained.pt.
2024/08/26 23:01:55 INFO mlflow.tracking.fluent: Experiment with name 'Auto3DSeg' does not exist. Creating a new experiment.
swinunetr_3 - training ...: 11%|████████▉ | 14/123 [3:04:04<23:53:06, 788.87s/round]
2024-08-27 02:13:10,402 - WARNING - swinunetr_3 - training: finished with early stop
2024-08-27 02:13:12,505 - INFO - ['python', '/workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_4/scripts/train.py', 'run', "--config_file='/workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_4/configs/hyper_parameters.yaml,/workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_4/configs/network.yaml,/workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_4/configs/transforms_infer.yaml,/workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_4/configs/transforms_train.yaml,/workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_4/configs/transforms_validate.yaml'"]
monai.networks.nets.swin_unetr SwinUNETR.__init__:img_size: Argument `img_size` has been deprecated since version 1.3. It will be removed in version 1.5. The img_size argument is not required anymore and checks on the input size are run during forward().
2024-08-27 02:13:21,974 - INFO - Downloaded: /workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_4/pretrained_model/swin_unetr.base_5000ep_f48_lr2e-4_pretrained.pt
2024-08-27 02:13:21,974 - INFO - Expected md5 is None, skip md5 check for file /workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_4/pretrained_model/swin_unetr.base_5000ep_f48_lr2e-4_pretrained.pt.
2024/08/27 02:13:22 INFO mlflow.tracking.fluent: Experiment with name 'Auto3DSeg' does not exist. Creating a new experiment.
swinunetr_4 - training ...: 15%|████████████▏ | 19/123 [4:05:10<22:21:59, 774.23s/round]
2024-08-27 06:25:33,346 - WARNING - swinunetr_4 - training: finished with early stop
2024-08-27 06:25:35,516 - INFO - Ensembling using single GPU!
2024-08-27 06:25:35,516 - INFO - The output_dir is not specified. /workspaces/prostate_opi_year_3/data/work_dir_b/ensemble_output will be used to save ensemble predictions.
2024-08-27 06:25:35,516 - INFO - Directory /workspaces/prostate_opi_year_3/data/work_dir_b/ensemble_output is created to save ensemble predictions
2024-08-27 06:25:35,822 - INFO - Auto3Dseg picked the following networks to ensemble:
2024-08-27 06:25:35,822 - INFO - segresnet_0
2024-08-27 06:25:35,822 - INFO - segresnet_1
2024-08-27 06:25:35,822 - INFO - segresnet_2
2024-08-27 06:25:35,823 - INFO - segresnet_3
2024-08-27 06:25:35,823 - INFO - segresnet_4
2024-08-27 06:25:35,823 - INFO - Auto3Dseg ensemble prediction outputs will be saved in /workspaces/prostate_opi_year_3/data/work_dir_b/ensemble_output.
Ensembling (rank 0)...: 0%| | 0/55 [00:00<?, ?it/s]`apex.normalization.InstanceNorm3dNVFuser` is not installed properly, use nn.InstanceNorm3d instead.
Ensembling (rank 0)...: 100%|█████████████████████████████████████████████████████████████████████████████████████████████| 55/55 [11:00<00:00, 12.01s/it]
Auto3Dseg pipeline is completed successfully.
Expected behavior
files (outputs of infrence of the test set) should be present in /workspaces/prostate_opi_year_3/data/work_dir_b/ensemble_output
Environment
Ensuring you use the relevant python executable, please paste the output of:
python -c "import monai; monai.config.print_debug_info()"
root@jm-l4-1-vm:/workspaces/prostate_opi_year_3# python -c "import monai; monai.config.print_debug_info()"
================================
Printing MONAI config...
================================
MONAI version: 1.4.0rc1+1.g4877767c
Numpy version: 1.24.4
Pytorch version: 2.3.0a0+40ec155e58.nv24.03
MONAI flags: HAS_EXT = False, USE_COMPILED = False, USE_META_DICT = False
MONAI rev id: 4877767cf92649a38ffda0fc590f2b92ba59f019
MONAI __file__: /usr/local/lib/python3.10/dist-packages/monai/__init__.py
Optional dependencies:
Pytorch Ignite version: 0.4.11
ITK version: 5.4.0
Nibabel version: 5.2.1
scikit-image version: 0.23.2
scipy version: 1.12.0
Pillow version: 10.2.0
Tensorboard version: 2.17.0
gdown version: 5.2.0
TorchVision version: 0.18.0a0
tqdm version: 4.66.2
lmdb version: 1.5.1
psutil version: 5.9.4
pandas version: 1.5.3
einops version: 0.7.0
transformers version: 4.40.2
mlflow version: 2.15.1
pynrrd version: 1.0.0
clearml version: 1.16.3
For details about installing the optional dependencies, please visit:
https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies
================================
Printing system config...
================================
System: Linux
Linux version: Ubuntu 22.04.4 LTS
Platform: Linux-6.5.0-1018-gcp-x86_64-with-glibc2.35
Processor: x86_64
Machine: x86_64
Python version: 3.10.12
Process name: python
Command: ['python', '-c', 'import monai; monai.config.print_debug_info()']
Open files: [popenfile(path='/root/.vscode-server/data/logs/20240815T180848/network.log', fd=20, position=0, mode='a', flags=33793), popenfile(path='/root/.vscode-server/data/logs/20240815T180848/ptyhost.log', fd=21, position=9870, mode='a', flags=33793), popenfile(path='/root/.vscode-server/data/logs/20240815T180848/remoteagent.log', fd=22, position=1283022, mode='a', flags=33793)]
Num physical CPUs: 8
Num logical CPUs: 16
Num usable CPUs: 16
CPU usage (%): [2.9, 2.7, 4.8, 2.6, 2.7, 2.4, 2.8, 15.8, 3.4, 3.6, 2.6, 2.6, 2.8, 3.6, 2.9, 84.7]
CPU freq. (MHz): 2000
Load avg. in last 1, 5, 15 mins (%): [0.9, 0.5, 0.2]
Disk usage (%): 14.7
Avg. sensor temp. (Celsius): UNKNOWN for given OS
Total physical memory (GB): 102.2
Available memory (GB): 96.4
Used memory (GB): 4.8
================================
Printing GPU config...
================================
Num GPUs: 2
Has CUDA: True
CUDA version: 12.4
cuDNN enabled: True
NVIDIA_TF32_OVERRIDE: None
TORCH_ALLOW_TF32_CUBLAS_OVERRIDE: 1
cuDNN version: 90000
Current device: 0
Library compiled for CUDA architectures: ['sm_52', 'sm_60', 'sm_61', 'sm_70', 'sm_72', 'sm_75', 'sm_80', 'sm_86', 'sm_87', 'sm_90', 'compute_90']
GPU 0 Name: Tesla V100-SXM2-16GB
GPU 0 Is integrated: False
GPU 0 Is multi GPU board: False
GPU 0 Multi processor count: 80
GPU 0 Total memory (GB): 15.8
GPU 0 CUDA capability (maj.min): 7.0
GPU 1 Name: Tesla V100-SXM2-16GB
GPU 1 Is integrated: False
GPU 1 Is multi GPU board: False
GPU 1 Multi processor count: 80
GPU 1 Total memory (GB): 15.8
GPU 1 CUDA capability (maj.min): 7.0
Additional context
Dockerfile based on monai dockerfile
ARG PYTORCH_IMAGE=nvcr.io/nvidia/pytorch:24.03-py3
FROM ${PYTORCH_IMAGE}
LABEL maintainer="[email protected]"
# TODO: remark for issue [revise the dockerfile](https://github.com/zarr-developers/numcodecs/issues/431)
RUN if [[ $(uname -m) =~ "aarch64" ]]; then \
export CFLAGS="-O3" && \
export DISABLE_NUMCODECS_SSE2=true && \
export DISABLE_NUMCODECS_AVX2=true && \
pip install numcodecs; \
fi
WORKDIR /opt/monai
# install full deps
COPY requirements.txt requirements-min.txt requirements-dev.txt /tmp/
RUN cp /tmp/requirements.txt /tmp/req.bak \
&& awk '!/torch/' /tmp/requirements.txt > /tmp/tmp && mv /tmp/tmp /tmp/requirements.txt \
&& python -m pip install --upgrade --no-cache-dir pip \
&& python -m pip install --no-cache-dir -r /tmp/requirements-dev.txt
# compile ext and remove temp files
# TODO: remark for issue [revise the dockerfile #1276](https://github.com/Project-MONAI/MONAI/issues/1276)
# please specify exact files and folders to be copied -- else, basically always, the Docker build process cannot cache
# this or anything below it and always will build from at most here; one file change leads to no caching from here on...
COPY LICENSE CHANGELOG.md CODE_OF_CONDUCT.md CONTRIBUTING.md README.md versioneer.py setup.py setup.cfg runtests.sh MANIFEST.in ./
COPY tests ./tests
COPY monai ./monai
# RUN BUILD_MONAI=1 FORCE_CUDA=1 python setup.py develop \
# && rm -rf build __pycache__
RUN python -m pip install git+https://github.com/Project-MONAI/MONAI#egg=monai
# NGC Client
WORKDIR /opt/tools
ARG NGC_CLI_URI="https://ngc.nvidia.com/downloads/ngccli_linux.zip"
RUN wget -q ${NGC_CLI_URI} && unzip ngccli_linux.zip && chmod u+x ngc-cli/ngc && \
find ngc-cli/ -type f -exec md5sum {} + | LC_ALL=C sort | md5sum -c ngc-cli.md5 && \
rm -rf ngccli_linux.zip ngc-cli.md5
ENV PATH=${PATH}:/opt/tools:/opt/tools/ngc-cli
RUN apt-get update \
&& DEBIAN_FRONTEND="noninteractive" apt-get install -y libopenslide0 \
&& rm -rf /var/lib/apt/lists/*
# append /opt/tools to runtime path for NGC CLI to be accessible from all file system locations
ENV PATH=${PATH}:/opt/tools
WORKDIR /opt/monai