Build FVCOM4.1 (Series) on ITO Subsystem-A using Intel Compilers

This is a memorandum of how to build and execute FVCOM 4.1 in series (non MPI) using Intel compilers on Supercomputer ITO Subsystem A at Kyushu University. Suppose the source code exists in [cci]FVCOM4.1/FVCOM_source[/cci], and [cci]FVCOM4.1/Examples/Estuary[/cci] is a selected test case.

Installing nkf

Line feed of some distributed source files is for Windows and need to be fixed using nkf.
For installing nkf, download the source of [cci]nkf[/cci] from this site , copy it to [cci] ~/src [/cci], and extract and build it.
[cc lang='bash']
$ tar xf nkf-2.1.4.tar.gz
$ cd nkf-2.1.4
$ make
### Create ~/local/bin and add its PATH
$ cp nkf ~/local/bin/
[/cc]

Check line feed of source files

Some line feed in the source files was [cci]CRLF[/cci] (this is for Windows while LF for Linux). Check them using the [cci]file[/cci] command and convert them to [cci]LF[/cci] using [cci]nkf[/cci].
[cc lang='bash']
$ file *.F
[/cc]
If displayed as [cci]ASCII text[/cci], it is [cci]LF[/cci]. Convert the non [cci]LF[/cci] files to [cci]LF[/cci].
[cc lang='bash']
$ nkf -Lu --overwrite ice_albedo.F
$ nkf -Lu --overwrite ice_coupling.F
$ nkf -Lu --overwrite ice_domain.F
$ nkf -Lu --overwrite ice_fileunits.F
$ nkf -Lu --overwrite ice_grid.F
$ nkf -Lu --overwrite ice_kinds_mod.F
$ nkf -Lu --overwrite ice_model_size.F
$ nkf -Lu --overwrite ice_scaling.F
$ nkf -Lu --overwrite ice_state.F
$ nkf -Lu --overwrite swmod3. F
[/cc]

Preparation for using Intel compilers

The version of Intel compilers can be selected from 2018, 2017 or 2016. Here we use the latest version of 2018. When compiling with Intel compilers as well as execution, it is necessary to set its environment using [cci]module[/cci] command at all times. It may be a good idea to add it at the end of [cci].bashrc[/cci].
[cc lang='bash']
$ module load intel/2018
[/cc]

Preparing libraries

Before compiling FVCOM, you need to prepare the necessary libraries in [cci]./FVCOM_source/libs[/cci]. netCDF should be installed separately (see here). In [cci]./FVCOM_source[/cci], copy [cci]make.inc_example[/cci] to [cci]make.inc[/cci] and edit it.
[cc lang='bash']
$ cp make.inc_example make.inc
[/cc]
First, set [cci]TOPDIR[/cci] as follows:
[cc]
           TOPDIR = /home/usr1/account/FVCOM4.1/FVCOM_source
[/cc]
Next, set the path of the libraries. Comment out its part of MEDM ENVIRONMENT and set LOCAL INSTALL as follows. Also, activate DTLIBS and DTINCS.
[cc lang='bash']
####################### MEDM ENVIRONMENT ####################################
# Use the environmental variables, LIBPATH and INCLUDEPATH, set by MODULE
# to find the packages used by this build.
# colon =:
# empty =
# dashI = $(empty) - I
# dashL = $(empty) - L
# ### UNCOMMENT HERE!
# LIBDIR = -L $(subst $(colon), $(dashL), $(LIBPATH))
# INCDIR = -I $(subst $(colon), $(dashI), $(INCLUDEPATH))

############################################################################

# LOCAL INSTAL
             LIBDIR = -L$(INSTALLDIR)/lib
             INCDIR = -I$(INSTALLDIR)/include
# ------------------------------------------------------------------- -------------------------
# STANDARD LIBRARIES FOR DATA AND TIME IN fVCOM:
#
DTLIBS = -ljulian
DTINCS =
# DTLIBS = -L/${TOPDIR}/libs
# DTINCS =
[/cc]
Next, set the environment variable for Intel compilers in series. Optimization is to stay at -O3 at this moment.
[cc lang='bash']
# ------------------------------------------------------------------- -------------------------
# Intel Compiler Definitions for Series
# ------------------------------------------------------------------- -------------------------
         CPP = /usr/bin/cpp
         COMPILER = -DIFORT
         FC = ifort
         CC = icc
         CXX = icpc
         CFLAGS = -O3
         DEBFLGS = # -check all
         OPT = -O3
[/cc]
Next, move to [cci]./FVCOM_source/libs[/cci], and open and edit [cci]makefile[/cci]:
[cc]
PACKAGES = proj fproj julian
[/cc]
Delete [cci]metis[/cci] and [cci]netcdf[/cci]. Similarly, comment out the parts associated with [cci]metis[/cci] and [cci]netcdf[/cci] below. [cci]metis[/cci] should be installed when executing in [cci]MPI[/cci]. Then execute [cci]make[/cci]:
[cc lang='bash']
$ make
[/cc]
[cci]./install[/cci] These libraries should have been installed in directories below [cci]./install[/cci].

Edit make.inc

Return to [cci]./FVCOM_source[/cci] directory, open [cci]make.inc[/cci], confirm the Intel compiler environment parameters are defined and the full path to [cci]FVCOM_source[/cci] is set at [cci]TOPDIR[/cci]. In addition, set the environmental parameters for [cci]netcdf[/cci]:
[cc]
### netcdf
IOLIBS = -L/home/usr1/account/local/netcdf-3.6.3-intel/lib -lnetcdf
IOINCS = -I/home/usr1/account/local/netcdf-3.6.3-intel/include
[/cc]
Next, [cci]FLAG[/cci] should be edited according to the simulation condition. Here, as an example for [cci]Examples/Estuary[/cci], [cci]FLAGs[/cci] to be activated are shown below:
[cc]
            FLAG_1 = -DDOUBLE_PRECISION
            FLAG_3 = -DWET_DRY
            FLAG_8 = -DLIMITED_NO
            FLAG_10 = -DGCN
            FLAG_14 = -DRIVER_FLOAT
[/cc]

Correct typo in makefile

The module of [cci]mod_esmf_nesting.F[/cci] is not in [cci]use[/cci] in anywhere, and thus it seems that it is not used. To avoid linking error, delete [cci]mod_esmf_nesting.F[/cci] in [cci]makefile[/cci].

Recreate makedepends

The file of [cci]makedepends[/cci] needs to be recreated using [cci]makedepf90[/cci] (unless installed, see here) in the directory of [cci]./FVCOM_source[/cci] as follows:
[cc lang='bash']
$ rm makedepneds
$ touch makedepends
$ make depend
[/cc]

Minor correction of source file

To avoid [cci]Warning[/cci] at compile time, delete the comment below at line 131 in [cci]wreal.F[/cci].
[cc lang='bash']
# endif !! ice_embedding yding
[/cc]

Build FVCOM4.1

Now you can build [cci]FVCOM4.1[/cci] in [cci]./FVCOM_source[/cci]:
[cc]
$ make
[/cc]
Successful if executable file of [cci]fvcom[/cci] is created.

Interactive execution of test case Estuary

Run a basic test case of [cci]Estuary[/cci] interactively. The file of [cci]tst_run.nml[/cci] of [cci]Estuary[/cci] has a bug. Move to [cci]../Examples/Estuary/run[/cci] and open [cci]tst_run.nml[/cci] with an editor and change the number of rivers to [cci]0[/cci] (the original was 3).
[cc]
& NML_RIVER_TYPE
RIVER_NUMBER = 0,
[/cc]
Copy the executable file of [cci]fvcom[/cci] created in [cci]./FVCOM4.1/FVCOM_source/[/cci] to the current directory ([cci]./FVCOM4.1/Examples/Estuary/run[/cci] (or set up the link)) and execute:
[cc lang='bash']
$ ./fvcom --casename=tst
[/cc]
Progress of computation is displayed on the screen, and when completed, [cci]TADA![/cci] should be displayed at the end. Also, [cci]tst_0001.nc[/cci] containing the calculation result should have been created.

Execution as batch job

To execute in series in batch job mode, prepare the following script named [cci]series.sh[/cci]:
[cc lang='bash']
#!/bin/bash
#PJM -L "rscunit=ito-a"
#PJM -L "rscgrp=ito-ss-dbg"
#PJM -L "vnode=1"
#PJM -L "vnode-core=36"
#PJM -L "elapse=10:00"
#PJM -j
#PJM -X

module load intel/2018
NUM_CORES=36
export OMP_NUM_THREADS=$NUM_CORES
./fvcom --casename=tst
[/cc]
Please refer to here. [cci]rscgrp[/cci] denotes resource selection, [cci]vnode[/cci] denotes the number of nodes, [cci]vnode-core[/cci] denotes the number of cores per node, which number should be given for [cci]NUM_CORES[/cci]. [cci]elapse[/cci] denotes limitation of time, giving hour, minute and second in the form of [cci]HH:MM:SS[/cci]. In the above script, 10 minutes is set. [cci]-j[/cci] superimposes standard error output on standard output and [cci]-X[/cci] inherits environment variables at job submission to the job execution environment.
Job submission can be made as follows:
[cc lang='bash']
$ pjsub series.sh
[/cc]
To check the status of the job, enter the command of [cci]pjstat[/cci]:
[cc lang='bash']
$ pjstat

ACCEPT QUEUED STGIN READY RUNING RUNOUT STGOUT HOLD ERROR TOTAL
0 0 0 0 1 0 0 0 0 1
s 0 0 0 0 1 0 0 0 0 1

JOB_ID JOB_NAME MD ST USER START_DATE ELAPSE_LIM NODE_REQUIRE VNODE CORE V_MEM
594176 series.sh NM RUN account 04/29 12:50:29< 0000:10:00 - 1 36 unlimited [/cc] As the symbol of [cci]RUN[/cci] appears under the symbol of [cci]ST[/cci], the job is currently running. This [cci]JOB_ID[/cci] is [cci]594176[/cci] and if you unintentionally submitted this job, you can delete it as follows: [cc lang='bash'] $ pjdel 594176 [/cc] Commands related to batch jobs can be found at here. In addition, information specific to the Intel compiler is here.