FVCOM MPI Run with METIS Libraries on HA8000

Compiling METIS Libraries

First, enter the directory of METIS_source and edit the makefile pointing to your C compiler. It is recommended to use a C compiler from the same vender as your intended Fortran compiler. The optimizations -msse3 -O3 for Intel Compiler were activated.

• $ make

Edit makefile for building FVCOM

The following is an example for Intel Compiler:

• CPP = /usr/bin/cpp  # GNU compiler may be required.
• CPPFLAGS = $(DEF_FLAGS) ?DINTEL
• FC = mpif90
• DEBFLAGS =
• OPT = -msse3 -O3 -ipo -no-prec-div  # -msse3 is an option for HA8000
• CLIB =

Do not forget to uncomment “MULTI_PROCESSOR.”

Compiling FVCOM

• $ make

Then, you will obtain an executable file “fvcom.”

Run FVCOM

Make a bash job script file of mpi.sh as follows:

• #!/bin/bash
• #@$-q short          # “parallel” for within 24 hours while “short” for within 1 hour
• #@$-N 4              # the number of nodes 4, 8, or 16 (dependent on your contract)
• #@$-J T16            # uncomment or T2, T4, T8, or T16. T16 or T8 seems the best choice for my case.
• #@$-lT 1:00:00     # job time limitation dependent on submitted job class
• cd ~/FVCOM_K/run  # full path to the directory of executable file 
• mpirun ./fvcom casename

Then, run fvcom using NQS.

• $ qsub mpi.sh
• $ qstat  # you can check the status of the job.
• $ qstat ?b  # you can check the status of all of the jobs.

Check mail and then check mpi.sh.e****** and mpi.sh.o******. In mpi.sh.e****** sometimes you may see errors starting with something

• =>> PBS: job killed: walltime 313 exceeded limit 300

which means the computational wall clock time exceeds the job time limitation.