MPI run of FVCOM2.6.1 on FX10@cc.u-tokyo.ac.jp

Table of Contents

Edit a makefile for METIS

Edit the makefile in ~/FVCOM2.6.1/METIS_source as follows:

CC = fccpx ### on login node. = fcc on interactive job node
OPTFLAGS = -Kfast

Then make METIS library.

$ cd ~/FVCOM2.6.1/METIS_source
$ make

Edit a makefile for MPI

Edit the makefile in ~/FVCOM2.6.1/FVCOM_source as follows:

# METIS
FLAG_4 = -DMULTIPROCESSOR
PARLIB = -L../METIS_source -lmetis ### set METIS installed directory

# NetCDF
FLAG_6 = -DNETCDF_IO
IOLIBS = -L/usr/local/netCDF/4.1.1/lib -lnetcdf ### set netCDF installed directory
IOINCS = -I/usr/local/netCDF/4.1.1/include

# FX10/MPI Compiler Definitions (The University of Tokyo)
CPP = /usr/bin/cpp
CPPFLAGS = $(DEF_FLAGS) -DGFORTRAN
FC = mpifrtpx ### on login node. = mpifrt on interactive job node
DEBFLGS = #-check all
OPT = -Kfast
CLIB = #-static-libcxa -i-static

Creating MPI header file for FX10

You need to delete the original MPI header file and then create a header file for FX10 on a login node following the procedure below:


$ cd ~/FVCOM2.6.1/FVCOM_source
$ rm mpif.h
$ rm makedepends
$ touch makedepends
$ make depend
$ make

MPI run on interactive job node

Interactive job node is available for node=1 (2 hours) or mode=8 (10 minutes). Use batch job node for longer computation.

$ pjsub --interact -L "node=8"
$ cd ~/FVCOM2.6.1/run # move to the execution directory
$ mpiexec ./fvcom tst
$ exit ### back to login node

MPI run on batch job node (1 process/node)

Create a job control script, e.g., mpi_1p.sh as follows:
Notice: You have to delete comments starting with ### below.

#!/bin/sh
#PJM -L "rscgrp=debug" ### set queue
#PJM -L "node=8" ### the number of nodes
#PJM -L "elapse=00:30:00" ### Max elapsed time
#PJM -j ### std error output is merged to stdout
mpiexec ./fvcom tst

Then, submit the job.

$ pjsub mpi_1p.sh
$ pjstat ### check the status of the submitted job.
$ pjstat -A ### Only for Admin: check the status of the jobs of my group.
$ pjstat -H ### show history of job submission.

MPI run on batch job node (16 processes/node)

Create a job control script, e.g., mpi_16p.sh as follows:
Notice: You have to delete comments starting with ### below.

#!/bin/sh
#PJM -L "rscgrp=debug" ### set queue
#PJM -L "node=12" ### the number of nodes
#PJM -L "elapse=00:30:00" ### Max elapsed time
#PJM --mpi "proc=192" ### 192 = 12 nodes * 16 processes
#PJM -j ### std error output is merged to stdout
mpiexec ./fvcom tst

Then, submit the job.

$ pjsub mpi_16p.sh

Efficient selection of the number of proc may depend. You may try to find out the best by trial and error manner for each case.

MPI run of FVCOM2.6.1 on FX10@cc.u-tokyo.ac.jp” に対して1件のコメントがあります。

コメントは受け付けていません。