LHCb Workflow: UploadLogFile command

[aldbr]

Please read #67 first.

• disentangle the module (https://gitlab.cern.ch/lhcb-dirac/LHCbDIRAC/-/blob/devel/src/LHCbDIRAC/Workflow/Modules/UploadLogFile.py?ref_type=heads)

---

Once you have the green light:

• implement it
• unit test it (reuse the existing ones as much as possible, adapting the code to the conventions of the repo https://gitlab.cern.ch/lhcb-dirac/LHCbDIRAC/-/blob/devel/src/LHCbDIRAC/Workflow/Modules/test/Test_Modules_AnalyseXMLSummary.py?ref_type=heads)
• document it correctly (you can ask for clarification here of course, code is not obvious)

[AcquaDiGiorgio]

If we are going to have some kind of persistent data between steps such as we currently have in the map workflow_commons, then we can split this module in 2 parts with some "workflow setup" as the pre-process step and the rest of the module as the post-process step.
If we are not going to have this type of persistency, we can merge everything in the post-process step. This might be much more readable and easier to work with.

IMO, this Module can be seen as:

Pre process step / Setup

Setup JobID

self.JobID =
1) os.environ["JOBID"] if in the environment
2) wms_job_id (from the arguments of the function)

Setup Request

self.request = workflow_commons["Request"] or Request() if None
self.request.RequestName = "job_%d_request.xml" % self.jobID
self.request.JobID = self.jobID
self.request.SourceComponent = "Job_%d" % self.jobID

Setup logpath

self.logdir = os.path.realpath(f"./job/log/{self.production_id}/{self.prod_job_id}")

Setup failover

self.failoverTransfer = FailoverTransfer(...)

Post process step

Get the corresponing files

systemCall(0, shlex.split("ls -al")) to get the contents of the working directory

self._determineRelevantFiles ->
- returns files that matches certain patterns that will be saved in the logSE

Move the files from workingDir to logDir, setup their permissions and zip them

self.__populateLogDirectory ->
- Creates and changes the permissions of self.logdir
- Copies the files that matched from the current working dir to self.logdir

self.__setLogFilePermissions ->
sets the permissions of each of the files on the directory

zips all the files

Store the zip in the SE

if self.jobID == None or simlar -> Don't store anything at the SE, Return S_OK()

put zipfile in the Storage Element ->
If error, upload it to failoverSE instead

Setup parameters

get url of the self.logSE Storage Element (logHttpsURL)

self.setJobParameter ->
parameter Log URL = "a" xml node with href=logHttpsURL

workflow_commons["Request"] = self.request

[aldbr]

• Everything will be in the PostProcess (for now at least)
• Import DIRAC components as needed