HOWTO: Run a simple ocean or atmosphere ensemble on HPCx

HOW TO: Run a simple ocean or atmosphere ensemble job on HPCx with the Ensemble Creation Toolkit

Created: 16th June 2006 by Ed Hawkins
Updated: 9th January 2007 by Ed Hawkins - added ID_STYLE in config file
Updated: 19th December 2007 by Ed Hawkins - changed machine name
Updated: 7th January 2008 by Ed Hawkins - version of UMCET changed in code for .profile on HPCx
Updated: 14th January 2008 by Ed Hawkins - list of MODS added

More details and official documentation available

Add an extra line in your .umrc file:
```
hpcxens login.hpcx.ac.uk <username> ssh
```

Add the following lines to your .profile locally (on puma),

# Setup UMCET variables
UMCETDIR=/home/umui/umcet ; export UMCETDIR
UMCETVN=normal ; export UMCETVN
PATH=$PATH:$UMCETDIR/$UMCETVN/bin
MANPATH=$MANPATH:$UMCETDIR/$UMCETVN/man

# Setup UMT variables
UMTDIR=/home/umui/umt ; export UMTDIR
UMTVN=normal ; export UMTVN
PATH=$PATH:$UMTDIR/$UMTVN/bin
MANPATH=$MANPATH:$UMTDIR/$UMTVN/man

and the following to your .profile on HPCx,

# Setup UMCET variables
UMCETDIR=$UMDIR/umcet ; export UMCETDIR
UMCETVN=dev ; export UMCETVN
PATH=$PATH:$UMCETDIR/$UMCETVN/bin
MANPATH=$MANPATH:$UMCETDIR/$UMCETVN/man

and logout, and re-login to both puma and hpcx.

You first need a reconfiguration job in your UMUI, with the following MODS:

Compile option overrides for the model:
$UMCET_OVERRIDES/gcom_Rdg_0.1b9
Modifications for the model:
$UMCET_SOURCE/startup.mod
Script inserts and modifications:
$UMCET_SCRIPT/gen_env.mod
$UMCET_SCRIPT/gsm2u406_pum.11.umcet
$UMCET_SCRIPT/script_fix_ibm_umcet.mod
$UMCET_SCRIPT/resubmit.mod
Environment variables:
UMCET_SOURCE   $UMCETDIR/$UMCETVN/um/um$VN/source_mods
UMCET_SCRIPT   $UMCETDIR/$UMCETVN/um/um$VN/script_mods
UMCET_OVERRIDES    $UMCETDIR/$UMCETVN/um/um$VN/overrides/platform/hpcx

Run this reconfiguration job in the usual way - this will create the start dumps, usually in $DATAW:

/hpcx/devt/n02/n02-ncas/<username>/um/<exptname>/

Take your executing job in UMUI (with the same mods as above) and insert names of your start dumps just created by the reconfiguration job - usually in directory $DATAW with names such as
```
<exptname><job id>.astart <exptname><job id>.ostart
```
- in the appropriate sections of the UMUI. SAVE and PROCESS this executing job. Job-stepping and automatic archiving work in the usual way.
Using your favourite text editor, create an ensemble configuration file in your $HOME/umui_jobs/ directory, called
```
<jobname>.config
```
This file defines the makeup of the ensemble. A simple example is:
```
 EXPTID <ensemblename>
CONTROL false 
ID_STYLE unpadded 
SPLIT factor1:8 
MULTI_ODUMP factorx
```
and this will create an ensemble of 8 runs, starting from 8 different ocean start dumps, without a control run. Change
```
MULTI_ODUMP to MULTI_ADUMP
```
to make it use 8 different atmosphere start dumps. Change
```
CONTROL false 
to 
CONTROL true
```
if you want a control run as well. For more complicated ensembles see the more detailed documentation.
Use the ect utility to create your ensemble submission files. In your $HOME/umui_jobs directory, use:
```
ect -j <job> -c <configfile>
```
Remember that the number of processors allowed on the parn queue at HPCx are multiples of 16. The minimum ensemble size would be 4 members, each using 4 processors.
Create your new start dumps. For ocean dumps this should be done from those made during the reconfiguration step. Put your new start dumps in
```
/hpcx/devt/n02/n02-ncas/<username>/um/<exptname>/
```
with names such as
```
<exptname><job id>.astart.X or <exptname><job id>.ostart.X
```
depending on whether you have an atmosphere or ocean ensemble, and where X is for: 1-n = ensemble member number. Note: control run has no .X
Submit the job from your local machine using:
```
umsub -h hpcxens -e <ensemblename> <job>
```
Note: The <ensemblename> must match that given in the configuration file. Do not use the normal umsubmit script - it won't work!
Run the SUBMIT script given by umsub on HPCx. Note: this is not the usual script, but one that is located in the Ens_Control subdirectory of the job.
When the NRUN part is complete, change NRUN to CRUN in the SUBMIT script, and resubmit.

Ed Hawkins