UNICORE at XSEDE

UNICORE (Uniform Interface of Computing Resources) offers a user-friendly grid system which includes client and server software that provides seamless access to remote and distributed computing capabilities. UNICORE is now offered on all XSEDE systems and eliminates the need to login to the Linux environment by providing two client interfaces: UNICORE command-line client (UCC) and the UNICORE Rich Client (URC) graphical user interface. For detailed information about the UNICORE client interfaces, please refer to the UNICORE site. There are also a nice set of Client Video Tutorials on how to use the UNICORE client programs.

UNICORE URC Installation

To install the URC client go to the XSEDE UNICORE URC download page and locate the download file for your system. This URC client has downloads for Windows, Mac, and Linux. The current version that is available and has been tested to work with XSEDE resources is URC version 6.6.0. This document will describe the Mac-based URC installation and use. Click on the Rich Client download link for Mac and install it on your system with your favorite file unzipping program.

For any questions, email help@xsede.org. If you experience a problem, please include a full description of the executed commands, output, and relevant log file. For access to XSEDE UNICORE services please include your XSEDE User Portal login and allocation information.

Login Credential Requirements

To use the UNICORE clients you must obtain and use a X.509 MyProxy credential that XSEDE recognizes. One way to get a credential is to login to an XSEDE resource and obtain an XSEDE or XSEDE X.509 MyProxy user credential. This credential can be created on any XSEDE machine where you have an account. and will need to be copied to your local computer to the UNICORE Client 6.6.0 certificates (certs) directory on your local machine. First, login to an XSEDE resource via SSH. Run the command:

$ module load globus
$ module load unicore

This will load the UNICORE module. Then to get your XSEDE MyProxy credential and convert it to the correct format use the myproxy-logon-unicore command:

$ myproxy-logon-unicore

The system will now prompt you first for your XSEDE MyProxy credential passphrase and then for a password to protect the credential in a new PKCS12 format credential file. We recommend that you use your XSEDE User Portal (XUP) password for both.

$ module load globus
$ module load unicore
$ myproxy-logon-unicore -u janesmith
/nics/d/home/janesmith/.ucc directory does not exist.
/nics/d/home/janesmith/.ucc directory created.
No accessible truststore in /etc/grid-security.
Downloading a copy from the XSEDE repository...
Download successful.
Retrieving MyProxy credential...
Enter MyProxy pass phrase:
Done.
/nics/d/home/janesmith/.ucc/preferences file does not exist.
/nics/d/home/janesmith/.ucc/preferences file created.
Exporting keystore to pkcs12 format...
This requires a pkcs12 keystore password.
Enter Export Password:
Verifying - Enter Export Password: Done.

This gets you a PKCS12 formatted X.509 myproxy certificate that is valid for 12 hours and saves this credential into a file named "~/.ucc/janesmith-myproxy.p12".

Confirm that the file has been created by issuing the "ls" command on your ~/.ucc directory.

$ ls ~/.ucc
default-myproxy.p12  janesmith-myproxy.p12  preferences  ucc.log  xsede-certs.jks 

The credential time limit can be changed by issuing the following command:

$ myproxy-logon-unicore -t 240

This will change the credential time limit to 240 hours (or 10 days).

Now use scp (secure copy) or whatever tool is available on your system to copy this myproxy certificate and your xsedecerts.jks file from an XSEDE resource to your computer. Open a new terminal window on your local machine and type the following command:

$ scp janesmith@darter.nics.utk.edu:~/.ucc/janesmith-myproxy.p12 ~/janesmith-myproxy.p12
$ scp janesmith@darter.nics.utk.edu:~/.ucc/xsedecerts.jks ~/xsedecerts.jks

The files should then copy after authentication.

On your local machine, find the directory where the UNICORE-Client-6.6.0 directory is located. Once you find it, copy the janesmith-myproxy.p12 and xsedecerts.jks files into the certs directory.

Start the UNICORE Rich Client. Click on the browse button to select a keystore.

Click on the browse button and select the xsede-certs.jks file that you just secure copied. It should be in the certs directory now. Navigate to the certs directory and select "open". Now type the truststore password into the UNICORE Rich Client password prompt box. The password is "xsede-certs.jks" (without quotes). Click "OK". Now you will have started the UNICORE Rich Client (URC).

Click on the workbench. Once you are in the workbench, click on the Keystore tab.

Right click in the keystore pane and select "import keystore". This will open a new dialog box. Select ".p12" in the lower right and then find the UNICORE-Client-6.6.0/certs directory that has your pkcs12 format myproxy certificate in it. Select your .p12 file and click on "Open". It will prompt you for your myproxy certificate password. Use your XSEDE password.

Now, right click on the Grid button in Grid Browser pane in the upper left and select "Add Registry". Type the following into the URL in the "Add a new Registry" dialog box and click OK.

https://unicore-registry.nics.xsede.org:8080/REGISTRY/services/Registry?res=default_registry

Now double click on the Registry and it should show a list of several XSEDE resources. That completes the installation! Note: If the machine does not show up under the Registry, contact help@xsede.org.

Using the URC

Example: "hello world" on Darter

To run a "hello world" job on Darter using the URC here are the steps.

  • First, launch the URC and get to the workbench.
  • Right click on "XSEDE-Darter" and select "create job".
  • Select "Script v2.2" and click the "Finish" button. This will create a script area in the right pane.
  • Type in the following "Hello world" script into the right pane.

     

      echo "Hello world!"
      hostname
      date
      qstat -u $USER

     

  • Then click on the green "run" button.

  • Before submitting the job, you will need to change to the resources that you will use.
    • Click on the resources tab at the bottom of the window.
    • Change the number of cores to a multiple of 16 (since we are on Darter.
    • If you want to change any other settings, check the limits by double clicking on the ellipsis next to the value. If everything is okay, the job will submit and show up as a new tab in the Script pane to the right. When the job is finished it will say finished in the "script submitted" tab. Get the job output from Darter by clicking on the "output" button just to the right of the run job button. A dialog box will pop up. Just select "OK" to request all output files. The output files when retrieved will show up in the bottom left pane labeled "Output...".

The previous hello world job ran on a Darter service node and not on a compute node. Here is how to submit a job that runs on Darter's compute nodes.

Example: "hello world" on Darter compute nodes

Right click on "XSEDE-Darter" and select create job. Select "Script v2.2" and click "Finish". Type the following into the script pane in the upper right pane.

cd /lustre/medusa/$USER
echo "Create the C program"
cat << EOF > testjob.c

#include <stdio.h>
#include <unistd.h>

int main()
{
char host[64];
gethostname(host,
sizeof(host));
printf("hello world ran on compute node %s\n", host);
}

EOF

echo "check on this job"
qstat -u $USER
echo "show the C program"
cat /lustre/medusa/$USER/testjob.c
echo "Compile the C program"
cc -o /lustre/medusa/$USER/testjob testjob.c
echo "Run the program on a compute node"
aprun -n 1 /lustre/medusa/$USER/testjob
echo "Create the C program"
cat testjob.c 

Once you have typed in the code, click on the run button. This program will:

  • Change directory to your scratch directory which is readable from the compute nodes (/lustre/medusa/$USER)
  • It will write into the output file "create the C program"
  • Then using a "here document" we will type in a C program and save it to /lustre/medusa/$USER/testjob.c
  • It will then write into the output file "check on this job"
  • The next command is a qstat command which gives status on the batch queue system and will show you your jobs (qstat -u $USER)
  • Next it will print out the testjob.c program to the output file
  • Afterwards, it will print "Compile the C program" to the output file
  • The "cc" command compiles the C program and makes an executable that can be run on the compute nodes.
  • Next, to launch a job on the compute nodes, you must use the "aprun" command. This says run the following application on 1 node (12 processors) and the application name is "/lustre/medusa/$USER/testjob".
  • Once the job completes then hit the get output button and click on the "OK" button to retrieve the output files. If everything worked okay, you will get the stdout (standard output) file with contents that look something like this:
Create the C program
check on this job
nid00016: Darter (UT/NICS Cray XC30)
 Req'd Req'd Elap
Job ID Username Queue Jobname SessID NDS Tasks Memory Time S
Time
-------------------- -------- -------- ---------------- ------ ----- ------ ------ ----- - -----
3062227.nid00016 janesmith small Script 30935 1 12 -- 01:00 R --
show the C program
#include <stdio.h>
#include <unistd.h>
int main()
{
char host[64];
gethostname(host, sizeof(host));
printf("hello world ran on compute node %s\n", host);
}
Compile the C program
Run the program on a compute node
hello world ran on compute node nid12231
Application 5168209 exit codes: 41
Application 5168209 resources: utime ~0s, stime ~0s

UCC commands

The UNICORE Commandline Client, also known as the UCC, is a full-featured client for the UNICORE Grid middleware. UCC has client commands for all the UNICORE basic services, including the UNICORE workflow system.

Make sure to load both the globus and unicore modules to ensure that the ucc command is in your path.

$ module load globus
$ module load unicore

The following are some commands:

  • To get information about the resources on the Grid you are connecting to, do the following...

    $ ucc list-sites

     

  • With this next command, the UCC will connect you to all machines that are registered to the registry address you are connecting to

    $ ucc connect

     

  • You can access 7 target system(s).

     $ ucc list-sites
      GATech-Keeneland https://unicore-gateway.nics.utk.edu:8080/GATech-Keeneland/services/TargetSystemService?res=318dfdc8-fd3f-4e90-a468-127a7ab47d84
      NICS-Darter https://unicore-gateway.nics.utk.edu:8080/NICS-Darter/services/TargetSystemService?res=ee474671-e742-4755-829c-05e11845d90c
      NICS-Nautilus https://unicore-gateway.nics.utk.edu:8080/NICS-Nautilus/services/TargetSystemService?res=0635f7bf-8210-4d1b-b35b-8ada880421e1
      SDSC-Trestles https://oldglobe.sdsc.edu:8080/SDSC-Trestles/services/TargetSystemService?res=2422eb39-18f9-4f02-89e8-e5e7d41c6984
    
      SDSC-Gordon https://oldglobe.sdsc.edu:8080/SDSC-Gordon/services/TargetSystemService?res=4e2b9416-ea26-423e-aeee-2579f6d38859
    
      Blacklight https://giu1.psc.xsede.org:8080/PSC/services/TargetSystemService?res=c2ce31e6-c905-4807-9769-ec2dba578e17

     

Example: using UCC, run an MPI job

To run an MPI job using the UCC, do the following.

  • Retrieve this script $ wget http://software.xsede.org/production/UNICORE/examples/mpihw.jsdl

  • Edit the JobProject line of the "mpihw.jsdl" file with your XSEDE Project ID then replace "/lustre/scratch/tsamuel" with "/lustre/medusa/$USER" or another location on the system you are submitting the job to as an output location. The "-s" option is used to specify the site and the "-a" option is to run the job asynchronously.

    $ ucc run -j mpihw.jsdl -s SDSC-Trestles
          QUEUED
          RUNNING
          SUCCESSFUL exit code: 0
          mpihwout 100% 413.0 473.6B/s
          /nics/d/home/janesmith/./5022d794-631b-4e24-a1eb-2f60b8d76ef1/5022d794-631b-4e24-a1eb-2f60b8d76ef1.properties

     

    Long-running jobs should be run asynchronously. In other words, just submit the job, stage in any files and start the job, so that the results will be finished later. When submitting a job in asynchronous mode, a job descriptor file is written that contains the job's address, and any information about export files.

$ ucc run -j mpihw.jsdl -s SDSC-Trestles -a
/nics/d/home/janesmith/./cca58565-d1d7-4e6a-9566-b05d0e70585c.job
/nics/d/home/janesmith/./cca58565-d1d7-4e6a-9566-b05d0e70585c.properties

Users can also apply the following command to do the same as what was done above.

$ ucc bes-submit-job -j mpihw.jsdl -s bes.1
/lustre/medusa/janesmith/mpitest/./cd5c43ae-d470-4de6-9669-20153513bdb0.properties
/lustre/medusa/janesmith/mpitest/./cd5c43ae-d470-4de6-9669-20153513bdb0.job 
$ ucc bes-job-status cd5c43ae-d470-4de6-9669-20153513bdb0.job 
Running

Eventually, your job should run.

$ ucc bes-job-status cd5c43ae-d470-4de6-9669-20153513bdb0.job 
Finished

Now look at the output file.

$ more mpihwout
cd /lustre/medusa/unicore/FILESPACE/cd5c43ae-d470-4de6-9669-20153513bdb0 ; aprun -n 12 ./a.out
Hello from rank 11 of 12
Hello from rank 4 of 12
Hello from rank 8 of 12
Hello from rank 1 of 12
Hello from rank 7 of 12
Hello from rank 6 of 12
Hello from rank 10 of 12
Hello from rank 5 of 12
Hello from rank 0 of 12
Hello from rank 2 of 12
Hello from rank 9 of 12
Hello from rank 3 of 12
Application 7250890 resources: utime ~1s, stime ~0s
cat ~/.ucc/preferences | grep bes
#BES entries are not required but are a nice shortcut to use when running jobs via bes-submit-job:
#ucc bes-submit-job -s bes.#
bes.1=https://unicore-gateway.nics.utk.edu:8080/NICS-
     Darter/services/BESFactory?res=default_bes_factory

As shown above, by default, there is only one resource defined: Darter. Users
can submit to a different resource other by editing the ~/.ucc/preferences
file and define "bes.2" the following way:

bes.2=https://oldglobe.sdsc.edu:8080/SDSC-Trestles/services/BESFactory?res=default_bes_factory

Workflow: automating output retrieval

We will execute the same MPI job as above, except that we will use gridftp to transfer the output file. As in the previous examples, please substitute the existing gridftp url in the example for a gridftp url you can access in the target field.

$ wget http://software.xsede.org/production/UNICORE/examples/mpihw-gridftp.jsdl
$ ucc bes-submit-job -j mpihw-gridftp.jsdl -s bes.1
/lustre/medusa/janesmith/mpigridftp/./8e38be3a-4dc7-48f7-8d88-fa3536b8eee7.properties
/lustre/medusa/janesmith/mpigridftp/./8e38be3a-4dc7-48f7-8d88-fa3536b8eee7.job
$ ucc bes-job-status 8e38be3a-4dc7-48f7-8d88-fa3536b8eee7.job 
Running

Workflow Example

The true strength in UNICORE lies in its ability to make creating custom workflows an easy task. For most researchers with allocations on different XSEDE machines, the ability to compute on one machine, transfer data to another machine, post-process on that machine, and then transfer the new data to a different machine, is a reality that must be addressed. Doing such a workflow by hand over-and-over can take quite a bit of time, but with UNICORE, a workflow such as this can be done with very little time and preparation. The following video tutorial demonstrates how.

Running a Serial Job

To run a serial job, do the following. The system binary file "date" will be used to show how easy it is to do.

  • Retrieve the file:

    $ wget http://www.nics.tennessee.edu/files/scripts/date.jsdl

     

  • Open "date.jsdl" in your favorite editor and edit the jsdl to specify an output location that is accessible to you.

  • Edit the JobProject line with your XSEDE Project ID then replace "/lustre/scratch/tsamuel" with "/lustre/medusa/**username**" or another location on the system you are submitting the job to as an output location.

  • Now submit the job with this command:

      $ ucc bes-submit-job -s bes.1 -j date.jsdl -v
      [ucc bes-submit-job] UCC 1.5.1, http://www.unicore.eu
      [ucc bes-submit-job] Reading properties file /d/home/janesmith/.ucc/preferences>
      [ucc bes-submit-job] Current directory is /medusa/janesmith/test>
      [ucc bes-submit-job] Output goes to <.>
      [ucc bes-submit-job] Keystore = /nics/d/home/janesmith/.ucc/default-myproxy.p12 [ucc bes-submit-job] No keystore password given.
      Please enter your keystore password: ********
      [ucc bes-submit-job] Using alias
    
      [ucc bes-submit-job] Keystore type = pkcs12
      [ucc bes-submit-job] Truststore password given.
      [ucc bes-submit-job] Truststore (assumed JKS) = /nics/d/home/janesmith/.ucc/xsede-certs.jks
      [ucc bes-submit-job] Using user certificate
    
      [ucc bes-submit-job] Short-lived user certificate, remaining validity time: 10.99h (expires Sat Oct 19 01:22:29 GMT 2013)
      [ucc bes-submit-job] Current directory is /medusa/janesmith/test>
      [ucc bes-submit-job] Output goes to <.>
      [ucc bes-submit-job] Contacting BES via Url https://unicore-gateway.nics.utk.edu:8080/NICS-Darter/services/BESFactory?res=defaultbesfactory ..
      [ucc bes-submit-job] Current directory is /medusa/janesmith/test>
      [ucc bes-submit-job] Output goes to <.>
      [ucc bes-submit-job] Loading JSDL file. /lustre/medusa/janesmith/test/./b4739fab-eb59-43ee-8a25-15b5a7e2e107.properties
      [ucc bes-submit-job] Job submitted, job url=https://unicore-gateway.nics.utk.edu:8080/NICS-Darter/services/BESActivity?res=b4739fab-eb59-43ee-8a25-15b5a7e2e107
      [ucc bes-submit-job] Wrote job descriptor to
      /lustre/medusa/janesmith/test/./b4739fab-eb59-43ee-8a25-15b5a7e2e107.job
      /lustre/medusa/janesmith/./b4739fab-eb59-43ee-8a25-15b5a7e2e107.job

     

  • To monitor the job's progress issue the following command:

    $ ucc bes-job-status b4739fab-eb59-43ee-8a25-15b5a7e2e107.job

     

    Your job should complete shortly.

     

    $ ucc bes-job-status b4739fab-eb59-43ee-8a25-15b5a7e2e107.job
      Finished

     

  • You should now have some new added files in your directory.

      $ ls
      b4739fab-eb59-43ee-8a25-15b5a7e2e107.job
      b4739fab-eb59-43ee-8a25-15b5a7e2e107.properties
      date.jsdl
      date.txt
      ucc.log

     

Frequently Asked Questions (FAQ)

The UNICORE team provides a FAQ located at http://sourceforge.net/apps/mediawiki/unicore/index.php?title=FAQ. See sections 3 and 4 for user FAQ information.

Last update: October 14, 2014

No comments yet. Be the first.