LARGe FARM setup using openfast toolbox

[fbelli997]

Dear all,

I would like to share with you the discussion I started on the openFAST repository.

OpenFAST/openfast#3245 (comment)

In this discussion I'm asking some information for the setup of AMR-wind and you suggested me to use this toolbox to having a good setup of all the files for the simulation.
I started using the 2 example as you suggested (Ex1FASTFarm_discretization and Ex3_AMRWindSamplingSetup) and then I moved forward modifying the EX2b_FASTFarm_LES_driven that you can find linked here below:

LARGE_FARM_FASTFarm_LES_driven.py

Actually, I'm receiving this message:

[WARN] HIGH-RES SPATIAL RESOLUTION GIVEN. CONVERTING FATAL ERRORS ON HIGH-RES BOXES CHECKS TO WARNINGS.LOW-RES SPATIAL RESOLUTION GIVEN. CONVERTING FATAL ERRORS ON LOW-RES BOX CHECKS TO WARNINGS.
[WARN] Low resolution x-sampling grid is not cell-centered with AMR-Wind's grid. 
           x-sampling grid:        -3460.0, -3435.0, -3410.0, -3385.0, ... 
           AMR-Wind grid (subset): -3470.0, -3450.0, -3430.0, -3410.0, ...
[WARN] Low resolution y-sampling grid is not cell-centered with AMR-Wind's grid. 
           y-sampling grid:        -4460.0, -4435.0, -4410.0, -4385.0, ... 
           AMR-Wind grid (subset): -4470.0, -4450.0, -4430.0, -4410.0, ...
[WARN] Low resolution z-sampling grid is not cell-centered with AMR-Wind's grid. 
           z-sampling grid:        5.0, 30.0, 55.0, 80.0, ... 
           AMR-Wind grid (subset): 5.0, 15.0, 25.0, 35.0, ...
[WARN] The LES path /full/path/to/LES/case/.../LESboxes does not exist
Checking inputs... Done.
Checking if we can create symlinks... False
Setting rotor parameters... Done.
INFO] Reading and checking template files
[INFO] Template EDfilename              =C:/DEL_NN/template_for_toolbox/IEA-15-240-RWT-UMaineSemi_ElastoDyn0.T
[INFO] Template HDfilename              =C:/DEL_NN/template_for_toolbox/IEA-15-240-RWT-UMaineSemi_HydroDyn.dat
[INFO] Template MDfilename              =C:/DEL_NN/template_for_toolbox/moordyn.T
[INFO] Template SSfilename              =C:/DEL_NN/template_for_toolbox/SeaState.dat
[INFO] Template SrvDfilename            =C:/DEL_NN/template_for_toolbox/servodyn_ws9_DD6_YD05_TI6_seed1.T
[INFO] Template ADfilename              =C:/DEL_NN/template_for_toolbox/IEA-15-240-RWT_AeroDyn.dat
[INFO] Template IWfilename              =C:/DEL_NN/template_for_toolbox/inflow_ws9_TI4_seed4.dat
[INFO] Template ADbladefilename         =C:/DEL_NN/template_for_toolbox/IEA-15-240-RWT_AeroDyn_blade.dat
[INFO] Template EDbladefilename         =C:/DEL_NN/template_for_toolbox/IEA-15-240-RWT_ElastoDyn_blade.dat
[INFO] Template EDtowerfilename         =C:/DEL_NN/template_for_toolbox/IEA-15-240-RWT-UMaineSemi_ElastoDyn_tower.dat
[INFO] Template turbfilename            =C:/DEL_NN/template_for_toolbox/ws9_DD6_YD05_TI6_seed1.T
[INFO] Template libdisconfilepath       =C:/DEL_NN/template_for_toolbox/libdiscon.T1.so
[INFO] Template controllerInputfilename =C:/DEL_NN/template_for_toolbox/I_ws9_DD6_YD05_TI6_seed1.IN
[INFO] Template FFfilename              =C:/DEL_NN/template_for_toolbox/ws9_DD4_YD05_TI6_seed1.fstf
[INFO] Template EDfilename              =C:/DEL_NN/template_for_toolbox/IEA-15-240-RWT-UMaineSemi_ElastoDyn0.T
[INFO] Template SEDfilename             =None
[INFO] Template HDfilename              =C:/DEL_NN/template_for_toolbox/IEA-15-240-RWT-UMaineSemi_HydroDyn.dat
[INFO] Template MDfilename              =C:/DEL_NN/template_for_toolbox/moordyn.T
[INFO] Template SSfilename              =C:/DEL_NN/template_for_toolbox/SeaState.dat
[INFO] Template SrvDfilename            =C:/DEL_NN/template_for_toolbox/servodyn_ws9_DD6_YD05_TI6_seed1.T
[INFO] Template ADfilename              =C:/DEL_NN/template_for_toolbox/IEA-15-240-RWT_AeroDyn.dat
[INFO] Template ADskfilename            =None
[INFO] Template SubDfilename            =None
[INFO] Template BDfilename              =None
[INFO] Template ADbladefilename         =C:/DEL_NN/template_for_toolbox/IEA-15-240-RWT_AeroDyn_blade.dat
[INFO] Template EDbladefilename         =C:/DEL_NN/template_for_toolbox/IEA-15-240-RWT_ElastoDyn_blade.dat
[INFO] Template EDtowerfilename         =C:/DEL_NN/template_for_toolbox/IEA-15-240-RWT-UMaineSemi_ElastoDyn_tower.dat
[INFO] Template turbfilename            =C:/DEL_NN/template_for_toolbox/ws9_DD6_YD05_TI6_seed1.T
[INFO] Template libdisconfilepath       =C:/DEL_NN/template_for_toolbox/libdiscon.T1.so
[INFO] Template controllerInputfilename =C:/DEL_NN/template_for_toolbox/I_ws9_DD6_YD05_TI6_seed1.IN
[INFO] Template coeffTablefilename      =None
[INFO] Template hydroDatapath           =None
[INFO] Template FFfilename              =C:/DEL_NN/template_for_toolbox/ws9_DD4_YD05_TI6_seed1.fstf
Cannot determine table dimension using NTypes
Cannot determine table dimension using NConnects
> c:\users\fbell\openfast_toolbox\openfast_toolbox\io\fast_input_file.py(850)_read()
-> d['descr'] = '' #
(Pdb)

and it got stuck on this message.

am I doing something wrong?
Can you please check if the data I'm including in the python file are correct?
I'm open to any suggestion.

thank you in advance for you help,
kind regards,
Federico

[rthedin]

Hi Frederico,

To answer your question on the OpenFAST repository (although I think you already figured it out): the workflow is not to use your inp to generate your fstf. The workflow is to use the python-based specification of your farm and atmospheric conditions, and the toolbox will generate TurbSim input files or AMR-Wind sampling input file (depending on what was requested), as well as all the FAST.Farm input files needed. Some settings in the fstf are only active depending on the choice of other variables. This is usually indicated as comments and you if such part is not used, then the values on the fstf do not matter. You are correct that if Mod_AmbWind is 1 (meaning VTK inflow), then the lines 20-34 are not used, but rather 15-18.

I will take a look at your input and try to reproduce your issue now.

[rthedin]

I had to modify your input to be able to run since not all files were provided. Unclear where exactly it stopped, as the last information printed is about the FFfilename. I'm not able to reproduce it using https://github.com/OpenFAST/openfast_toolbox/blob/main/data/IEA15MW/FF.fstf as a starting point. It could be stopping at the libdiscon or on other places. Can you provide a self-contained example that is able to reproduce the issue at hand?

[fbelli997]

Dear rthedin,

If I understand correctly, you are asking me to provide you the "template_toolbox_folder" to run my input file, right?
these are the zip folders including all the files I'm currently using in the .py:

template_for_toolbox.zip
LARGE_FARM_SampleFiles.zip

Let me know whether it is enough to run it.
Thank you in advance for your help,

Best regards,
Federico

[rthedin]

Thanks for the files. I was able to reproduce it with a few small changes to your input.

The issue is on the format of MoorDyn, and how the toolbox parses tables. On your input, you have under POINTS the following line:

ID   Type      X         Y       Z       M    V    CdA   CA

For this toolbox, the Type entry means it will know how many lines are to be read, based on the a priori reading of NTypes or NConnects, as correctly indicated in the error message. The header above is typical of MoorDyn v1, which does have the NConnects value:

openfast_toolbox/openfast_toolbox/io/tests/example_files/FASTIn_MD-v1.dat

Lines 10 to 12 in d34b283

	---------------------- POINT PROPERTIES -----------------------------------------------------
	12 NConnects - number of connections including anchors and fairleads
	Node Type X Y Z M V FX FY FZ CdA Ca

However, on MoorDyn v2, the length of the tables are not known ahead of time, and it is handled as it should by the toolbox. Since it does not have Type on the header, it reads as it goes, not expecting to know the length. This is how a MoorDyn v2 considered by the toolbox looks like:

openfast_toolbox/openfast_toolbox/io/tests/example_files/FASTIn_MD-v2.dat

Lines 8 to 9 in d34b283

	---------------------- POINTS --------------------------------
	ID Attachment X Y Z M V CdA CA

Now, the problem is that you somehow mixed and matched inputs from different versions of MoorDyn. You have:

---------------------- POINTS --------------------------------
ID   Type      X         Y       Z       M    V    CdA   CA

which means ID from MoorDyn v2, but Type from MoorDyn v1. As you can now see, the Type/Attachment difference is problematic. The behavior considered by this toolbox is consistent with the documentation for MoorDyn and inputs, as given here.

When it can't read, there is a hook for pdb, which is what you see printed on your error message.

openfast_toolbox/openfast_toolbox/io/fast_input_file.py

Lines 846 to 849 in d34b283

	try:
	d['value'], d['tabColumnNames'], d['tabUnits'] = parseFASTNumTable(self.filename,lines[i:i+nTabLines+nHeaders+nOffset],nTabLines,i, nHeaders, tableType=tab_type, nOffset=nOffset, varNumLines=d['tabDimVar'])
	except:
	import pdb; pdb.set_trace()

No fix is required on the toolbox. You have an issue on your input file. If you replace "Type" with "Attachment" on line 9 of your MoorDyn input file, it will work as expected.

Finally, a couple of notes on your input deck:

1. You requested PotMod 1 on HydroDyn but did not give the hydroDataPath on the templates. I have now added a check for that. If you request PotMod 1, you have to give the hydro data. I changed PotMod to 0 on your inputs since I do not have your hydroData files. (you can get the latest version with the additional checks on FF: Add AMReX support #77)

2. In your fst file, you have CompSeaState, and not the correct CompSeaSt. Similar for SeaStateFile and SeaStFile. The toolbox will crash and tell you that.

[fbelli997]

Dear rthedin,

Thank you very much for your feedback, I'll try to fix my files and run again the toolbox. I'll let you know as soon as possible.

Kind regards,
Federico

[fbelli997]

Dear rthedin,

It looks like I succesfully generated most of the files requested for a FASTfarm simulation, however, i still receive this error:

[ OK ] All OpenFAST files were copied successfully.
Traceback (most recent call last):
  File "c:\Users\fbell\openfast_toolbox\openfast_toolbox\fastfarm\examples\LARGE_FARM_FASTFarm_LES_driven.py", line 288, in <module>
    amr, ffcase = main(test=False)
                  ^^^^^^^^^^^^^^^^
  File "c:\Users\fbell\openfast_toolbox\openfast_toolbox\fastfarm\examples\LARGE_FARM_FASTFarm_LES_driven.py", line 273, in main
    ffcase.FF_setup() # Write FAST.Farm input files
    ^^^^^^^^^^^^^^^^^
  File "c:\users\fbell\openfast_toolbox\openfast_toolbox\fastfarm\FASTFarmCaseCreation.py", line 2666, in FF_setup
    self._FF_setup_LES(**kwargs)
  File "c:\users\fbell\openfast_toolbox\openfast_toolbox\fastfarm\FASTFarmCaseCreation.py", line 2726, in _FF_setup_LES
    self._symlink(src, dst)
  File "c:\users\fbell\openfast_toolbox\openfast_toolbox\fastfarm\FASTFarmCaseCreation.py", line 929, in _symlink
    raise Exception(error)
Exception: Src file not found: /full/path/to/LES/case/.../LESboxes\Low

It seems like he is looking for the precursor vtk files, is that correct?
Actually I don't still have them since I haven't run the LES yet.
However, I successfully generated this file:

FF_boxes.i

including all the informations of the LES box.

Should my next step be running the LES simulation using AMR-wind, right?
Then, once the VTK files are available, should I place them in the LESboxes\Low (and corresponding high-resolution) folders and run LARGE_FARM_FASTFarm_LES_driven.py again to generate the remaining files? Is that correct?

This workflow seems a bit unusual to me, so I just want to make sure I am understanding it correctly.

Thanks in advance.

Kind regards,

Federico

[rthedin]

You did not paste the entire message log given. The first few lines should have [WARN] The LES path <path> does not exist, which gets triggered on these lines:

openfast_toolbox/openfast_toolbox/fastfarm/FASTFarmCaseCreation.py

Lines 750 to 752 in d34b283

	for p in self.inflowPath:
	if not os.path.isdir(p):
	WARN(f'The LES path {p} does not exist')

The exception printed there gets triggered exactly when it can't find the path. The error message is correct and indicates it is unable to find source files. There is a skipchecks option on the constructor if you want to skip certain checks. Keep in mind, though, that since you are on a Windows machine, the toolbox will perform copies instead of symbolic links. Symbolic links are successfully created even though whatever they are pointing to does not exist. That is the reason we can use the skipchecks. With a Windows machine, the skipchecks might not be super useful as you will still run into issues when it tries to copy a file that does not exist. I unfortunately do not have a Windows machine available to test/debug Windows-only issues.

The input FF_boxes.i should be appended to your AMR-Wind input file, which, in turn, should have settings exactly like you setup on your example (prob_lo, prob_hi, etc). So yes, the next step is to run the AMR-Wind.

The output of AMR-Wind will not be in VTK format. This option of VTK is code-agnostic, so you are able to run whatever LES code you prefer, and process the data into the correct VTK format. FAST.Farm requires a very specific file and directory convention for the VTK files, outlined here. So after the execution of AMR-Wind, you will need to read the data, format, save VTK, and re-arrange the directory structure as required by FAST.Farm.

There is an example SLURM script to process the AMR-Wind data in parallel on the repo, here. This script requires routines to read/process AMR-Wind data available in the windtools repository, so you will need to grab that too. After the VTK is saved, you will need to make sure the naming convention is correct following the link above. We have scripts to do it with symbolic links; however, as you're on Windows, you will need to rename them all.

With all of that said, we have a very recent capability of directly using native AMReX data directly as outputted by AMR-Wind, skipping all this intermediate processing. That option is documented here and updated toolbox to handle it is currently PR #77.