MPI run of FVCOM2.6.1 on FX10@cc.u-tokyo.ac.jp

Edit a makefile for METIS

Edit the makefile in ~/FVCOM2.6.1/METIS_source as follows:

CC = fccpx ### on login node. = fcc on interactive job node
OPTFLAGS = -Kfast

Then make METIS library.

$ cd ~/FVCOM2.6.1/METIS_source
$ make


Edit a makefile for MPI

Edit the makefile in ~/FVCOM2.6.1/FVCOM_source as follows:

# METIS
FLAG_4 = -DMULTIPROCESSOR
PARLIB = -L../METIS_source -lmetis ### set METIS installed directory

# FX10/MPI Compiler Definitions (The University of Tokyo)
CPP = /usr/bin/cpp
CPPFLAGS = $(DEF_FLAGS) -DGFORTRAN
FC = mpifrtpx ### on login node. = mpifrt on interactive job node
DEBFLGS = #-check all
OPT = -Kfast
CLIB = #-static-libcxa -i-static


Creating MPI header file for FX10

You need to delete the original MPI header file and then create a header file for FX10 on a login node following the procedure below:


$ cd ~/FVCOM2.6.1/FVCOM_source
$ rm mpif.h
$ rm makedepends
$ touch makedepends
$ make depend
$ make


MPI run on interactive job node

Interactive job node is available for node=1 (2 hours) or mode=8 (10 minutes). Use batch job node for longer computation.

$ pjsub --interact -L "node=8"
$ cd ~/FVCOM2.6.1/run # move to the execution directory
$ mpiexec ./fvcom tst
$ exit ### back to login node


MPI run on batch job node (1 process/node)

Create a job control script, e.g., mpi_1p.sh as follows:
Notice: You have to delete comments starting with ### below.

#!/bin/sh
#PJM -L "rscgrp=debug" ### set queue
#PJM -L "node=8" ### the number of nodes
#PJM -L "elapse=00:30:00" ### Max elapsed time
#PJM -j ### std error output is merged to stdout
mpiexec ./fvcom tst

Then, submit the job.

$ pjsub mpi_1p.sh
$ pjstat ### check the status of the submitted job.
$ pjstat -A ### Only for Admin: check the status of the jobs of my group.
$ pjstat -H ### show history of job submission.


MPI run on batch job node (16 processes/node)

Create a job control script, e.g., mpi_16p.sh as follows:
Notice: You have to delete comments starting with ### below.

#!/bin/sh
#PJM -L "rscgrp=debug" ### set queue
#PJM -L "node=12" ### the number of nodes
#PJM -L "elapse=00:30:00" ### Max elapsed time
#PJM --mpi "proc=192" ### 192 = 12 nodes * 16 processes
#PJM -j ### std error output is merged to stdout
mpiexec ./fvcom tst

Then, submit the job.

$ pjsub mpi_16p.sh

Efficient selection of the number of proc may depend. You may try to find out the best by trial and error manner for each case.