This is a memorandum of how to build and execute FVCOM 4.1 in series (non MPI) using Intel compilers on Supercomputer ITO Subsystem A at Kyushu University. Suppose the source code exists in FVCOM4.1/FVCOM_source, and FVCOM4.1/Examples/Estuary is a selected test case.

Installing nkf

Line feed of some distributed source files is for Windows and need to be fixed using nkf.
For installing nkf, download the source of nkf from this site , copy it to  ~/src, and extract and build it.

$ tar xf nkf-2.1.4.tar.gz
$ cd nkf-2.1.4
$ make
### Create ~/local/bin and add its PATH
$ cp nkf ~/local/bin/

Check line feed of source files

Some line feed in the source files was CRLF (this is for Windows while LF for Linux). Check them using the file command and convert them to LF using nkf.

$ file *.F

If displayed as ASCII text, it is LF. Convert the non LF files to LF.

$ nkf -Lu --overwrite ice_albedo.F
$ nkf -Lu --overwrite ice_coupling.F
$ nkf -Lu --overwrite ice_domain.F
$ nkf -Lu --overwrite ice_fileunits.F
$ nkf -Lu --overwrite ice_grid.F
$ nkf -Lu --overwrite ice_kinds_mod.F
$ nkf -Lu --overwrite ice_model_size.F
$ nkf -Lu --overwrite ice_scaling.F
$ nkf -Lu --overwrite ice_state.F
$ nkf -Lu --overwrite swmod3. F

Preparation for using Intel compilers

The version of Intel compilers can be selected from 2018, 2017 or 2016. Here we use the latest version of 2018. When compiling with Intel compilers as well as execution, it is necessary to set its environment using module command at all times. It may be a good idea to add it at the end of .bashrc.

$ module load intel/2018

Preparing libraries

Before compiling FVCOM, you need to prepare the necessary libraries in ./FVCOM_source/libs. netCDF should be installed separately (see here). In ./FVCOM_source, copy make.inc_example to make.inc and edit it.

$ cp make.inc_example make.inc

First, set TOPDIR as follows:

           TOPDIR = /home/usr1/account/FVCOM4.1/FVCOM_source

Next, set the path of the libraries. Comment out its part of MEDM ENVIRONMENT and set LOCAL INSTALL as follows. Also, activate DTLIBS and DTINCS.

####################### MEDM ENVIRONMENT ####################################
# Use the environmental variables, LIBPATH and INCLUDEPATH, set by MODULE
# to find the packages used by this build.
# colon =:
# empty =
# dashI = $(empty) - I
# dashL = $(empty) - L
# ### UNCOMMENT HERE!
# LIBDIR = -L $(subst $(colon), $(dashL), $(LIBPATH))
# INCDIR = -I $(subst $(colon), $(dashI), $(INCLUDEPATH))

############################################################################

# LOCAL INSTAL
             LIBDIR = -L$(INSTALLDIR)/lib
             INCDIR = -I$(INSTALLDIR)/include
# ------------------------------------------------------------------- -------------------------
# STANDARD LIBRARIES FOR DATA AND TIME IN fVCOM:
#
             DTLIBS = -ljulian
             DTINCS =
#            DTLIBS = -L/${TOPDIR}/libs
#            DTINCS =

Next, set the environment variable for Intel compilers in series. Optimization is to stay at -O3 at this moment.

# ------------------------------------------------------------------- -------------------------
# Intel Compiler Definitions for Series
# ------------------------------------------------------------------- -------------------------
         CPP      = /usr/bin/cpp
         COMPILER = -DIFORT
         FC       = ifort
         CC       = icc
         CXX      = icpc
         CFLAGS = -O3
         DEBFLGS = # -check all
         OPT = -O3

Next, move to ./FVCOM_source/libs, and open and edit makefile:

PACKAGES = proj fproj julian

Delete metis and netcdf. Similarly, comment out the parts associated with metis and netcdf below. metis should be installed when executing in MPI. Then execute make:

$ make

./install These libraries should have been installed in directories below ./install.

Edit make.inc

Return to ./FVCOM_source directory, open make.inc, confirm the Intel compiler environment parameters are defined and the full path to FVCOM_source is set at TOPDIR. In addition, set the environmental parameters for netcdf:

### netcdf
IOLIBS = -L/home/usr1/account/local/netcdf-3.6.3-intel/lib -lnetcdf
IOINCS = -I/home/usr1/account/local/netcdf-3.6.3-intel/include

Next, FLAG should be edited according to the simulation condition. Here, as an example for Examples/Estuary, FLAGs to be activated are shown below:

            FLAG_1 =  -DDOUBLE_PRECISION
            FLAG_3 =  -DWET_DRY
            FLAG_8 =  -DLIMITED_NO
            FLAG_10 = -DGCN
            FLAG_14 = -DRIVER_FLOAT

Correct typo in makefile

The module of mod_esmf_nesting.F is not in use in anywhere, and thus it seems that it is not used. To avoid linking error, delete mod_esmf_nesting.F in makefile.

Recreate makedepends

The file of makedepends needs to be recreated using makedepf90 (unless installed, see here) in the directory of ./FVCOM_source as follows:

$ rm makedepneds
$ touch makedepends
$ make depend

Minor correction of source file

To avoid Warning at compile time, delete the comment below at line 131 in wreal.F.

# endif !! ice_embedding yding

Build FVCOM4.1

Now you can build FVCOM4.1 in ./FVCOM_source:

$ make

Successful if executable file of fvcom is created.

Interactive execution of test case Estuary

Run a basic test case of Estuary interactively. The file of tst_run.nml of Estuary has a bug. Move to ../Examples/Estuary/run and open tst_run.nml with an editor and change the number of rivers to 0 (the original was 3).

& NML_RIVER_TYPE
RIVER_NUMBER = 0,

Copy the executable file of fvcom created in ./FVCOM4.1/FVCOM_source/ to the current directory (./FVCOM4.1/Examples/Estuary/run (or set up the link)) and execute:

$ ./fvcom --casename=tst

Progress of computation is displayed on the screen, and when completed, TADA! should be displayed at the end. Also, tst_0001.nc containing the calculation result should have been created.

Execution as batch job

To execute in series in batch job mode, prepare the following script named series.sh:

#!/bin/bash
#PJM -L "rscunit=ito-a"
#PJM -L "rscgrp=ito-ss-dbg"
#PJM -L "vnode=1"
#PJM -L "vnode-core=36"
#PJM -L "elapse=10:00"
#PJM -j
#PJM -X

module load intel/2018
NUM_CORES=36
export OMP_NUM_THREADS=$NUM_CORES
./fvcom --casename=tst

Please refer to here. rscgrp denotes resource selection, vnode denotes the number of nodes, vnode-core denotes the number of cores per node, which number should be given for NUM_CORES. elapse denotes limitation of time, giving hour, minute and second in the form of HH:MM:SS. In the above script, 10 minutes is set. -j superimposes standard error output on standard output and -X inherits environment variables at job submission to the job execution environment.
Job submission can be made as follows:

$ pjsub series.sh

To check the status of the job, enter the command of pjstat:

$ pjstat

  ACCEPT QUEUED  STGIN  READY RUNING RUNOUT STGOUT   HOLD  ERROR   TOTAL
       0      0      0      0      1      0      0      0      0       1
s      0      0      0      0      1      0      0      0      0       1

JOB_ID     JOB_NAME   MD ST  USER     START_DATE      ELAPSE_LIM NODE_REQUIRE     VNODE  CORE V_MEM
594176     series.sh  NM RUN account  04/29 12:50:29< 0000:10:00 -                1      36   unlimited

As the symbol of RUN appears under the symbol of ST, the job is currently running. This JOB_ID is 594176 and if you unintentionally submitted this job, you can delete it as follows:

$ pjdel 594176

Commands related to batch jobs can be found at here. In addition, information specific to the Intel compiler is here.