Serial/Parallel Computing of Fluent/CFX Solver

The most efficient and convenient way to run Fluent solver for your CFD simulations, which need hours, days or weeks to finish, is to run the solver in batch/parallel mode. First, you shall setup the CFD problem, including mesh, models, boundary conditions etc., on an interactive Fluent interface. Next save the initial problem to Fluent case and data file pair. Then you can choose a suitable batch queue to submit your case for computing.

Batch Serial Fluent/CFX Job

Submit Batch Fluent Job

Small or medium sized jobs, which may need about one hour or slightly longer to complete, are recommended submitting to run in serial-batch mode (on single CPU processor/core). The small case is recommended to be submitted to queue “serial” and large case is recommended to be submitted to queues “parallel8” or “parallel” for parallel computation.

For example, the following is the command used to submit a serial-batch Fluent job to queue serial:

fluent-job -q serial -o myjob.o "fluent 3d -gu -i myjob_script"

where:

-q serial
-o myjob.o
fluent 3d -gu
-i myjob_script
define to submit Fluent job to queue serial
specify the filename to keep the standard output message for the job
run Fluent 3D solver without graphics display (required for batch)
specify the Fluent command script file in text. The file contains the Fluent commands for the execution, you may use Unix text editor,
such as vi, emacs or pico, to edit this file.

Here is an example of the script inside the file myjob_script (for steady-state case):

rcd test.cas
it 1000
wd test_1000.dat
exit
y
# read test.cas and test.dat into Fluent solver
# do 1000 iterations
# store (write data) the solution to test_1000.dat
# exit Fluent solver
# confirm the exit with “y” or “yes“

There have many ways to create/edit the text-based script file. To learn those ways, please check at the article on editing files on Linux from the HPC@NUS Issue No. 7 in 2005 March.

When the job is completed, you should find the final data file “test_1000.dat” at the working directory where you submitted the job. To do the post-processing and visualise the results, launch a Fluent interface interactively from the command line and read the case and data files into it to do the visualisation.
Here is another example of the script for unsteady flow simulation, or time-dependent flow:

/file/rcd myfile.cas
/file/autosave/data-frequency
100
/solve/set/time-step 6
/solve/dual-time-iterate
1000
20
wd mydata_new.dat
exit
y
# read myfile.cas and myfile.dat into Fluent solver
# define auto-save frequency of result data as per
100
 iterations
# set time step as 6 seconds
# start to do iterations of
1000
 iterations in total
20
 sub-iterations within one time step
# store (write data) the final solution to mydata_new.dat
# exit Fluent solver
# confirm the exit with “y” or “yes

When the job is completed, you should find 10 (=1000/100) sets of intermediate data file and the final data file “mydata_new.dat“. You can use those data files to visualise the results.

Submit Batch CFX Job

CFX batch job can be submitted when you have the CFX .def file created correctly. The job submission is as below.

cfx-job -q serial -o myjob.o "cfx5solve -batch -def case1.def -name case1_new"

where:

-q serial
-o myjob.o
cfx5solve -batch -def case1.def -name case1_new
define to submit CFX job to queue serial
specify the filename to keep the standard output message for the job
run CFX solver in batch mode to process the problem defined in cas1.def, and save the results into case1_new.
Batch Parallel Fluent/CFX Job

For long hour Fluent simulations, parallel computing with Fluent solver is encouraged. By running simulations on parallel model, or using multiple CPUs, the computing time will be shortened and hence improve the production. Fluent solver is well and ready parallelised using MPI and is able to run in parallel manner using multiple CPUs on all HPC clusters. The parallel queues supported Fluent solver include parallel8 and parallel.

To use parallel computing for a Fluent simulation, similarly as the serial-batch job process you need to setup your Fluent model first and save the model in Fluent case and data files. Initial debug and/or trial is necessary to ensure the model is properly defined.
At the same time, prepare the Fluent script file that is used for the job submission (same as above). To submit a parallel Fluent job, run command “fluent-job” and follow the 6 steps:

fluent-job 
1. Fluent version [16, 18, default=18]: 18
2. Solver type [2d, 2ddp, 3d, 3ddp]: 3d <Enter> 
3. Fluent Inputput/Script file Name: myjob_script <Enter>
4. Output file Name [e.g. stdout_fluent.o]: <Enter>
----------------------------------------
   Your Fluent batch job is:
----------------------------------------

1. FLUENT VERSION..........> 18
2. SOLVER..................> 3d
3. Input File..............> myjob_script
4. Stderr Output File......> STDOUT_fluent
5. Num of CPU..............> 12
6. Run Queue...............> parallel12
Enter <1,2,3,4,5> to change the above inputs, or 
Enter <y,n> to submit/discard this job. <1,2,3,4,5,y,n,q(quit)>: Y <Enter>

There have two mandatory inputs for the parallel Fluent job: 1) the solver type and 2) the Fluent input script file. The solver type is related to your Fluent model, in either 3d or 2d.

For Steps 3, 4 and 5, you can use the default settings (just press Enter to accept) which was tuned to choose the most suitable parameters and resources for the job for you. You can also input your own parameters for any specific reasons or purpose.

A summary of the batch job will be listed after all the necessary information is provided. Modifications to the batch job are allowed, or enter “Y” to confirm the job submission.
The post-processing is the same as mentioned above.

CFX solver can be submitted to run in parallel mode as below with the .def is ready to process.

cfx-job -q parallel12 -o myjob.o "cfx5solve -batch -def case1.def -initial case1_001.res –par-local –partition 12"
Job Monitor

You can monitor the progress of your job with the following PBS commands:

qstat
to list all your batch jobs
qstat -f <jobid> to list a particular batch jobs in detail
qstat -ans <jobid> to find out the pending reason of job
qcat -j <jobid> to display the standard output of a particular job
qdel <jobid> to terminate a particular job