Any opportunities to delete intermediate files?

[funnell]

Description of feature

Hello,

I've run into a problem where intermediate files have completely filled up our HPC's scratch space while using funcprofiler. I'm wondering if there are any opportunities to delete large unneeded files at the end of process scripts?
I can batch samples for now, but would be nice not to have to!

Thank you very much for your work on this pipeline!

[nickp60]

Hi @funnell, all the intermediate files should end up in work/ in the working directory, which should make deletion pretty straightforward by specifying cleanup = true in your config. It looks like nf-boost might be a good option? Are there tasks you are noticing writing to somewhere outside of work/? There are some more tips here from the nf-core docs.

[nickp60]

I don't love the ethos of deleting the work directory, but @miraep8 wrote a plugin to make the logs end up in the actual output folder, not just in work/ where they can get lost with cleanup.

[funnell]

Hello! I have work set to a folder in our scratch directory so that is what is filling it up. The pipeline ended up unable to complete as I managed to get scratch to 100%, and prompted some concerned emails from HPC admins!

[nickp60]

Gotcha. Do you know which files were contributing most to the storage issue? Sounds like nf-boost would be the most useful here.

[funnell]

Yes nf-boost is an option, although would be sad to lose resume. I'm not sure what was using the most space as I deleted everything in a hurry (other people's jobs were apparently getting blocked). I'm running this on some MSK data so probably the same as what you might see on your end.
I'll be rerunning in smaller batches and once one batch is complete I can look to see what's taking up space and report back.