How to acquire an account
Please send an e-mail to hpc-admin[at]ceng[dot]metu[dot]edu[dot]tr stating who you are and why you need an account. We will send further information and instructions to you
How to run parallel programs using an MPI library
Choose an MPI Library
First, you should choose an MPI library. It can be done using mpi-selector-menu command:
$ mpi-selector-menu Current system default: <none> Current user default: <none> "u" and "s" modifiers can be added to numeric and "U" commands to specify "user" or "system-wide". 1. mvapich2_gcc-1.2p1 2. mvapich2_intel-1.2p1 3. mvapich_gcc-1.1.0 4. mvapich_intel-1.1.0 5. openmpi_gcc-1.2.8 6. openmpi_intel-1.2.8 U. Unset default Q. Quit Selection (1-6[us], U[us], Q):
For example, to select mvapich_gcc-1.1.0 type “3u”. Then simply type “q” to quit.
Alternatively, you can use mpi-selector command:
$ mpi-selector --set mvapich_gcc-1.1.0
Please remember that you should start a new shell to begin using the selected MPI library. You can simply logout and login back to start a new shell.
Compile your program
Then, you should compile your program using mpicc (for C) or mpicxx (for C++):
$ mpicc myprog.c -o myprog
Prepare a PBS script
Afterwards, prepare a PBS script for submitting your job to the queue manager:
## select queue for submitting job #PBS -q cenga ## set the name of the job #PBS -N myjob ## request resources #PBS -l nodes=2:ppn=2 ## switch to the working directory (by default PBS launches processes from your home directory) cd $PBS_O_WORKDIR ## run the program on 4 processors mpirun -np 4 myprog
For different MPI libraries, some modifications in “mpirun” line will be necessary.
MVAPICH: Number of processors on which the program will run should be specified using -np option:
mpirun -np 4 myprog
OpenMPI: A hostfile should be provided using -hostfile option. The hostfile provided by PBS can be used for this purpose:
mpirun -hostfile $PBS_NODEFILE myprog
MVAPICH2: Both number of processors and a hostfile should be specified. Furthermore, mpirun_rsh should be used instead of mpirun:
mpirun_rsh -np 4 -hostfile $PBS_NODEFILE myprog
Submit job to queue manager
Finally, you should submit your PBS script to the queue manager:
$ qsub mypbsscript
How to send jobs from kefal.ulakbim.gov.tr to nar.ceng.metu.edu.tr
You can run jobs on METU CENG HPC Cluster with your TR-grid user account from kefal.ulakbim.gov.tr. To do that you should do the following:
1. Create a directory under /trgrid_scratch on kefal.ulakbim.gov.tr with your user name
$ mkdir /trgrid_scratch/user-name
2. Change the permissions of this directory, so that other users can not reach it
$ chmod 700 /trgrid_scratch/user-name
3. Copy the files related to the job you are going to submit to “Nar” cluster to that directory
$ cp -r ~/related-directory /trgrid_scratch/user-name/
4. Prepare the PBS Script for your job. Meanwhile, remember the following:
Specify the name of the queue server with concatenating ”@kavun-ib” to the line where you write the name of the queue you are going to submit your job
<code>#PBS -q trgridb@kavun-ib</code>
After specifying other PBS parameters, add the following lines to your script
<code>. /usr/share/Modules/init/sh module load trgrid/scratch</code>
For the remaining configurations required for your application, use the “module load” command.
(You can get information about loadable libraries compilers and applications from Available Modules) Ex:
<code>module load espresso/intel/IB</code>
Use TRGRID_SCRATCH environment variable for specifying file paths. Ex:
cd $TRGRID_SCRATCH/user-name/related-directory mpiexec -comm ib $TRGRID_SCRATCH/user-name/related-directory/executable-name > output.txt 2> error.txt
5. Send your job to the queue manager, using your script
$ qsub your-script
Furthermore, you can check the status of your job or cancel it. Ex:
$ qstat 42099.kavun-ib $ qstat @kavun-ib $ qdel 42099.kavun-ib
How to use Microsoft Windows HPC Server 2008
Microsoft Windows HPC Server 2008 is installed on 3 nodes of our system one being the head node. These nodes are dual-bootable and normally booted with Scientific Linux. To use this product, you should do the following:
Ordered List ItemSend an e-mail to email@example.com indicating that you want to use MSW HPC Server. We will reboot the nodes and create an account for you.
Install MSW HPC Server's Client Side to your computer.
Check Microsoft Windows HPC Server 2008 for examples and usage guidelines.
How to use Hadoop On Demand (HOD)
In order to use HOD, firstly the required environment variables should be set. This can be accomplished simply by the following command:
module load application/hod
This line might be appended to ~/.profile file for making it available on every login.
Afterwards, Hadoop jobs might be run in a typical HOD session by using the following sequence of commands:
hod allocate ... hadoop ... hadoop ... ... hod deallocate ...
Furthermore, HOD might also be used for directly running Hadoop scripts. For example, a sample script with the following content
sleep 60 hadoop fs -copyFromLocal hadoop-input.txt / hadoop jar /home1/software/application/hadoop/hadoop-0.20.1-examples.jar wordcount /hadoop-input.txt /wc-output &> wc-out hadoop fs -copyToLocal /wc-output .
might be run using the following command:
hod script -d ~/hod-clusters/test -n 5 -s hadoop-script
For more information about using HOD, please refer to http://hadoop.apache.org/common/docs/r0.20.1/hod_user_guide.html