Quick Start

HPCC is currently in test mode. Please contact Adam.Wang@williams.edu if you need access. Jobs can be submitted to the job scheduler (Lava) at shell command line but it is strongly recommended to create a shell script and submit it to the queue as a batch job. If unix is new to you, here is a unix cheat sheet to get you started.

Job Submission (standard) using R as an example:

  1. create R script called “myRcode.R”.
  2. create a shell script “myRjob.sh” which has all the specs of the job e.g. output, queue name, job slots requested etc.
  3. enter ‘bsub’ to submit myRjob.sh to the queue “long” as a batch job.

myRjob.sh looks like the following:

———————————————-
#!/bin/bash
# submit via ‘bsub < myRjob.sh’
#BSUB -q long    # queue name
#BSUB -o out      # output filename
#BSUB -e err      # error filename
#BSUB -J myR # job name
#BSUB -n 64 # job slots (64 cores) requested

R CMD BATCH myRcode.R
———————————————-

Enter the following to submit the job:

bsub < myRjob.sh

When the job is submitted correctly, the scheduler will generate a job id which looks something similar to the following.

Job <394> is submitted to default queue <long>.

Enter the following to view the status of the job in details

bjobs 394 -l

Enter the following to view the history of the job in details

bhist 394 -l

Enter the following to view the history of all your jobs submitted

bhist -a

Job Submission (standard) using Mathematica as an example:

  1. create a mathematica in file script called “math.infile”.
  2. create a shell script “math.sh” which has all the specs of the job e.g. output, queue name, job slots requested etc.
  3. enter ‘bsub’ to submit math.sh to the queue “long” as a batch job.

math.sh looks like the following:

———————————————-
#!/bin/bash
# submit via ‘bsub < math.sh’

#BSUB -q long        #queue name
#BSUB -e math.err    #error filename
#BSUB -o math.out    #output filename
#BSUB -n 12        #job slots (12 cores) requested
#BSUB -J MyMath        #job name

math < math.infile
———————————————-

Enter the following to submit the job:

bsub < math.sh

When the job is submitted correctly, the scheduler will generate a job id which looks something similar to the following.

Job <399> is submitted to default queue <long>.

Enter the following to view the status of the job in details

bjobs 399 -l

Enter the following to view the history of the job in details

bhist 399 -l

Enter the following to view the history of all your jobs submitted

bhist -a

Job Submission (standard) using Matlab as an example:

  1. create a matlab m file called “mymatlab.m”.
  2. create a shell script file “mymatlab.sh” which has all the specs of the job e.g. output, queue name, job slots requested etc.
  3. enter ‘bsub’ to submit mymatlab.sh to the queue “long” as a batch job.

simulation.sh looks like the following:

———————————————-
#!/bin/bash
# submit via ‘bsub < mymatlab.sh’

#BSUB -q long        #queue name
#BSUB -e mymatlab.err    #error filename
#BSUB -o mymatlab.out    #output filename
#BSUB -n 12        #job slots (12 cores) requested
#BSUB -J mymatlab        #job name

matlab -nodisplay -r “mymatlab, exit”
———————————————-

Enter the following to submit the job:

bsub < mymatlab.sh

When the job is submitted correctly, the scheduler will generate a job id which looks something similar to the following.

Job <399> is submitted to default queue <long>.

Enter the following to view the status of the job in details

bjobs 399 -l

Enter the following to view the history of the job in details

bhist 399 -l

Enter the following to view the history of all your jobs submitted

bhist -a

Job Submission (standard) using Stata as an example

  1. create do file called “MyStataDo.do”.
  2. create a shell script “myStataJob.sh” which has all the specs of the job e.g. output, queue name, job slots requested etc.
  3. enter ‘bsub’ to submit myStataJob.sh to the queue “statamp” as a batch job.

myStataJob.sh looks like the following:

———————————————-
#!/bin/bash
# submit via ‘bsub < auto.sh’
# bsub stata-mp -b do auto.do

rm -f auto.err auto.out auto.log

#BSUB -q statamp  #queue name
#BSUB -e auto.err #output err file
#BSUB -n 8          #job slots (8 cores) requested
#BSUB -J stata    #job name
#BSUB -R “span[hosts=1]”    # run this job on a single node

# And run my stata program; output will be on auto.log
set procs_use 8

stata-mp -b do auto.do
———————————————-

Enter the following to submit the job: bsub < myStataJob.sh. When the job is submitted correctly, the scheduler will generate a job id which looks something similar to the following.

Job <398> is submitted to default queue <statamp>.

Enter the following to view the status of the job in details

bjobs 398 -l

Enter the following to view the history of the job in details

bhist 398 -l

Enter the following to view the history of all your jobs submitted

bhist -a

Job Submission (parallel with openMPI):

MPI is a bit complicated but that’s what you need to take the full advantage of the cluster. To run Parallel with MPI, the process is similar but you will need to write MPI program and then compile it. HPCC@williams implementation of Parallel is openMPI. So it’s recommended to use mpicxx (/usr/lib64/openmpi/bin/) that comes with openMPI to compile your MPI programs. Here are the steps (“quad_mpi.cpp” is from Chris Paciorek at UC Berkley):

  1. create an MPI program called “quad_mpi.cpp”.
  2. compile it with mpicxx to get “quad_mpi” (/usr/lib64/openmpi/bin/mpicxx quad_mpi.cpp).
  3. create a shell script “run.quad_mpi” which has all the specs of the job e.g. output, queue name, etc.
  4. enter ‘bsub’ to submit run.quad_mpi to the queue “long” as a batch job.

run.quad_mpi looks like the following:

———————————————-
#!/bin/bash
# submit via ‘bsub < run.quad_mpi’
#BSUB -q long        # queue name
#BSUB -e err        # error filename
#BSUB -o quad_mpi.out    # output filename
#BSUB -J myMPIrun    # job name
#BSUB -n 4        # job slots (4 cores) requested
export PATH=/usr/lib64/openmpi/bin:$PATH export LD_LIBRARY_PATH=/usr/lib64/openmpi/lib/:$LD_LIBRARY_PATH

echo “make sure we have the right mpirun” mpirun quad_mpi
———————————————-

Enter the following to submit the job:

bsub < run.quad_mpi

When the job is submitted correctly, the Lava scheduler will generate a job id which looks something similar to

Job <394> is submitted to default queue <long>.

Enter the following to view the status of the job in details

bjobs 394 -l

Check Host Status

Enter the following
bhosts
It will display the current status of the available hosts, something similar to the following:

HOST_NAME STATUS JL/U MAX NJOBS RUN SSUSP USUSP RSV
compute-0-0 ok 64 4 4 0 0 0
compute-0-1 ok 64 4 4 0 0 0
compute-0-2 ok 64 4 4 0 0 0
HPCC ok 16 0 0 0 0 0

Check Queue Status

Enter the following
bqueues
It will display the current status of the available queues, something similar to the following which indicates that the ‘long’ queue is open and active and 12 jobs are running :

QUEUE_NAME PRIO STATUS MAX JL/U JL/P JL/H NJOBS PEND RUN SUSP
test 50 Open:Active 16 16 0 0 0 0
short 40 Open:Active 192 64 0 0 0 0
statamp 35 Open:Active 64 64 0 0 0 0
long 30 Open:Active 192 64 12 0 12 0

Other Common Commands:

  • bkill: kill, suspend or resume a job
  • bstop: suspend jobs
  • bresume: resume jobs
  • bmod: modifie job submission options of a job
  • bpeek: displays the stdout and stderr output of an unfinished job
  • bhosts: display hosts and their static and dynamic resources
  • Click here for more ……