This page is a memorandum of building and executing FVCOM4.1 in MPI using Intel compilers on the supercomputer Subsystem-A of ITO at Kyushu University. Those in Series are introduced in this page. The building environment for Series should be prepared. The source files are supposed to exist in FVCOM4.1/FVCOM_source and the test case is Estuary in FVCOM4.1/Examples/Estuary.

Preparation for METIS

METIS library requires to be installed for building for MPI. First load Intel compilers:

$ module load intel/2018

Environmental variables for Intel C compiler should be made as follows:

$ export CC=icc
$ export CPP="icc -E"

Move to the directory of FVCOM4.1/METIS_source where the METIS source code exists and extract the source codes:

$ tar xf metis.tgz
$ cd metis

Open makefile and edit the MOPT at the line 12 as follows:

            MOPT     = -O3 -no-prec-div -fp-model fast=2 -xHost

The above option of MOPT is the recommended value by the vendor. This makefile should read make.inc in FVCOM4.1/FVCOM_source that was prepared for Series where FLAGs do not matter. For this purpose, include in this makefile should be corrected as follows:

include ../../FVCOM_source/make.inc

Building METIS

To build METIS, just make install in the source directory.

$ make install

Edit make.inc

The file of make.inc for Series created in (here) requires slight modification for FLAG_4 and RANLIB as follows:

            FLAG_1 =  -DDOUBLE_PRECISION
            FLAG_3 = -DWET_DRY
            FLAG_4 = -DMULTIPROCESSOR
            PARLIB = -lmetis  #-L/usr/local/lib -lmetis
            FLAG_8 =  -DLIMITED_NO
            FLAG_10  = -DGCN
            FLAG_14 = -DRIVER_FLOAT

Then comment out the setting for Intel compiler for Series in make.inc and edit for Intel MPI as follows:

#--------------------------------------------------------------------------
#  Intel/MPI Compiler Definitions (ITO-A@kyushu-u)
#--------------------------------------------------------------------------
         CPP      = icc -E
         COMPILER = -DIFORT  
         CC       = mpiicc
         CXX      = mpiicpc
         CFLAGS   = -O3 -no-prec-div -fp-model fast=2 -xHost
         FC       = mpiifort
         DEBFLGS  = #-check all -traceback
         OPT      = -O3  -no-prec-div -fp-model fast=2 -xHost
#--------------------------------------------------------------------------

This setting is for FLAT MPI and the option is the recommended values by the vendor.

makefile for building FVCOM4.1

In building for Series, mod_esmf_nesting.F was deleted in makefile to avoid link errors. In building for MPI, this modified makefile is valid while the original one is also no problem.

Minor correction of a source file

Similarly in the case of Series introduced here, the comments at the line 131 of wreal.F should be deleted to avoid warning during compilation:

#   endif       !!ice_embedding  yding

Build FVCOM4.1 for MPI

In the directory of FVCOM_source, build FVCOM4.1 for MPI as follows:

$ make

Taking some time and the executable file of fvcom should be created.

Executing test case of Estuary as batch job

MPI run may only work in batch job (not interactively). First move to FVCOM4.1/Examples/Estuary/run, open tst_run.nml and edit the number of rivers to be 0 (originally 3), which is a bug.

&NML_RIVER_TYPE
RIVER_NUMBER = 0,

Copy the executable of fvcom to FVCOM4.1/Examples/Estuary/run. Move to FVCOM4.1/Examples/Estuary/run and create a script named, e.g., mpi.sh containing the following:

#!/bin/bash
#PJM -L "rscunit=ito-a"
#PJM -L "rscgrp=ito-ss-dbg"
#PJM -L "vnode=1"
#PJM -L "vnode-core=36"
#PJM -L "elapse=10:00"
#PJM -j
#PJM -X

module load intel/2018

NUM_NODES=${PJM_VNODES}
NUM_CORES=36
NUM_PROCS=36

export I_MPI_PERHOST=$NUM_CORES
export I_MPI_FABRICS=shm:ofa

export I_MPI_HYDRA_BOOTSTRAP=rsh
export I_MPI_HYDRA_BOOTSTRAP_EXEC=/bin/pjrsh
export I_MPI_HYDRA_HOST_FILE=${PJM_O_NODEINF}

mpiexec.hydra -n $NUM_PROCS ./fvcom --casename=tst

Here this script for FLAT MPI is supposed to use one node with 36 cores/node. The number of cores per node should be specified in NUM_CORES and NUM_PROCS denotes the number of nodes × the number of cores per node. Further information is given in this page (in Japanese).
To submit the batch job, use the command of pjsub as follows:

$ pjsub mpi.sh

To check the status of the job, invoke pjstat command in the terminal. Further information about batch job if given in this page and in this page (in Japanese).