NOTICE: As of September 30, 2013 Nautilus has been decommissioned and removed from production.
Nautilus is an SGI Altix UV 1000 and is the centerpiece of NICS Remote Data Analysis and Visualization Center (RDAV). The primary purpose of Nautilus is to enable data analysis and visualization of data from simulations, sensors, or experiments. Nautilus is intended for serial and parallel visualization and analysis applications that take advantage of large memories and multiple computing cores. Nautilus allows for both utilization of a large number of processors for distributed processing and the execution of legacy serial analysis algorithms for very large data processing by large numbers of users simultaneously. One can reach Nautilus via login.nautilus.nics.utk.edu.
Nautilus is a Shared Memory Processing (SMP) machine consisting of 1024 cores (Intel Nehalem EX processors) with 4 terabytes of global shared memory in a single system image. There is a 960 TB Lustre file system, access to the NICS HPSS storage system, a CPU speed of 2.0 GHz, and a peak performance of 8.2 Teraflops. Currently, there are 4 NVIDIA Tesla (codename Fermi) GPUs with 3GB of GDDR5 memory per GPU. These GPUs support CUDA C/C++/Fortran, OpenCL, and DirectCompute Toolkits.
The Network File System (NFS) server contains user's home directories, project directories, and software directories. Project data can be stored in HPSS and NFS directories. The NFS space is shared among NICS resources having a 2 GB quota limit. Scratch disk space is provided by a Lustre high performance parallel file system. There is a 30 day age limit on files stored in one's Lustre space. Modifying file access times (using "touch" or any other method) for the purpose of circumventing purge policies may result in the loss of access to the scratch file systems. Under special circumstances, users may request a purge exemption by submitting a request in a timely manner that includes detailed justification to firstname.lastname@example.org. Please include file system (e.g. Nautilus: /lustre/medusa ), PI of the project, user requesting exemption, TG-Account, time requested (two weeks etc.), and detailed justification.
There are common environment variables one can use to easily access these directories. These include
$SCRATCHDIR (points to
$HOME, that is
You may access Nautilus via the Unix
ssh utility. In order to
ssh to Nautilus you must use a one time password (OTP) token. New users will receive a NICS Token Activation form via email. The user must return this notarized form and then the OTP token will be mailed to the user via snail mail. Only then can you login via
ssh to Nautilus:
Currently, gridftp and gsissh are disabled on Nautilus. In the future there will be a separate grid node to handle these requests. There are ftp, sftp, scp, bbcp, and hsi available on Nautilus.
The first time one logs into Nautilus via
ssh, it will ask to set a Personal Identification Number (PIN) and then it will ask you to repeat this number for verification. This will be your private 4 to 6 digit PIN. The next step will prompt you to enter in your PIN followed by the numbers on your OTP token. The numbers on your token will change every 30 seconds and you can view the time left on that passcode by the bar on the left hand side of the viewing window. For further information on this, please see, http://www.nics.tennessee.edu/user-support/activating-otp
Please note the following:
- PIN changes: If you have forgotten your PIN number, please email email@example.com.
- Access policies: Do not share your OTP and PIN with anyone. This will lead to immediate deactivation of your account. Citizenship has to be provided to use Nautilus.
- User Responsibilities: When using resources at NICS, you agree to the following user responsibilities:
- You have the responsibility to protect your account from unauthorized use. Never share your login information. If you believe your account has been compromised, immediately notify us as 866-907-2383.
- You have responsibility for the security of your programs and data.
- You may not copy and/or distribute proprietary software without the software owner's permission. Possession or use of illegally copied software is prohibited; all software must be appropriately acquired and used according to the specific licensing.
- NICS resources may only be used by authorized users and is limited to the purpose prescribed in the project award. Use of these resources for processing proprietary information, source code or executable code must be disclosed in the award process and is prohibited unless authorized by the project award. Use of NICS resources for export controlled information; source code or executable code is prohibited.
- To ensure protection of data and resources, user activity may be monitored, intercepted, recorded, copied, audited, inspected, and disclosed to authorities. By using this system, the user consents to such at the discretion of authorized site personnel.
- Activities in violation of any laws may be reported to the proper authorities for investigation and prosecution. Abusive activity may be reported to your home institution for review and action.
- NICS file systems are generally very reliable, however, data may still be lost or corrupted. Users are responsible for backing up critical data.
- Violations of NICS policy can result in loss of access to NICS resources and possible prosecution.
If you have questions you may contact NICS User Support during normal working hours, 9:00 am - 6:00 pm ET at 1-865-241-1504. Consultants are also available to respond to your questions by phone 24/7 at 866-907-2383 or email firstname.lastname@example.org.
The default account setup each user has is their home directory, Lustre scratch space, and project name (given by XSEDE).
The command-line interpreter (a.k.a. shell) is the traditional Unix/Linux operating system. The default shell environment on Nautilus is bash. There are other shells available, such as sh, csh, tcsh, and zsh. Users may change their default shell in the NICS User Portal https://portal.nics.tennessee.edu/. To log into the portal, you need to use your RSA SecurID OTP token.
One can find the pre-set environment variables by the
env command. In addition, there are pre-set modules loaded when one logs in. This includes the default Intel compiler, Moab, and Torque - one can view these by the
module list. Common environment variables include: $HOME (points to /nics/[a-d]/home/$USER) and $SCRATCHDIR (points to /lustre/medusa/$USER).
The modules software package allows you to dynamically modify your user environment by using
modulefiles. Modules are very useful to compile and running on Nautilus. For a full list of Nautilus installed software, please see http://www.nics.tennessee.edu/computing-resources/nautilus/software
Modules can make your computing experience easy. The default compiler is Intel. The compiler programming environment modules are denoted by
PE-. It is recommended to switch to the compiling environment you want and then load the compiler version and other modules. Here is a short list and description of commonly used module commands. Note, if no version number is preceding the package name, it will use the default package.
Shows all loaded modules.
module swap <package A> <package B>
This will swap package B for package A. Useful to change PE modules to switch compilers and versions of other modules.
module avail <package>
If no package is given, it will list all available modules. This command is useful to see which versions of particular software are installed. Try:
module avail namd
module show <package>
This gives you the information concerning the installed software. One will see the
setenv commands that will modify your environment if you decide to load that module. This is useful for two major reasons. First, you can make sure what executable you like to run- there might be a small difference of the executable name on Nautilus versus another machine. Here, you can perform a
ls on the outputted bin directory. Secondly, some environment variables could be introduced. For instance, the FFTW module will provide an environment variable that points to the library and include directories- simply include this variable into your makefile versus including the full path.
NICS maintains high speed network connections with the XSEDE network and Internet2. NICS currently maintains the following options for file transfer:
- globus-url-copy (GridFTP)
The standard transfer utilities, scp, sftp, ftp and rsync can be used to transfer files to and from NICS systems. These utilities are usually already installed on Linux/Unix machines, and there are many command and graphical clients available. Due to familiarity and ease, these may be the best choice for transferring scripts and small files, however, these options can be slow in comparison, and may be ill suited for transferring large amounts of data. More information on these utilities can be found on the transfer utilities page.
When available, the preferred method for file transfer is GridFTP. All XSEDE resources should have GridFTP installed, and it may be installed at other non-XSEDE locations as well, check first for local availability. Before using GridFTP, check out the Globus Online File Transfer User Guide . Although GSI-SSH is not currently available on Nautilus, GridFTP is available for file transfers. The URL for the Nautilus GridFTP server is
If GridFTP is not available, the multistreaming transfer utility BaBar copy (BBCP) may be used. The BBCP utility is capable of breaking up your transfer into multiple simultaneously transferring streams, thereby transferring data faster than single-streaming utilities such as SCP and SFTP. Use of BBCP requires the installation of BBCP on both host systems involved in the transfer. Examples may be found on the BBCP page http://www.nics.tennessee.edu/user-support/general-support/data-transfer/bbcp.
Nautilus has C/C++ and fortran compilers from PGI, GNU, and Intel. Only one compiler is available at a time to avoid ambiguity. The module commands may be used to determine and switch one's compiling environment. Modules that define the programming environment begin with
PE-. One can see the available ones by issuing the command,
module avail PE
In addition, there are modules for the compilers themselves, pgi, intel and gcc. These can be used to change which version of a compiler you are using. If you are changing to a different compiler entirely, you should not need to change these, PE- should do that automatically.
To compile OpenMP code with PGI, you need to use the flag
-mp, whereas with GNU, you would use
-fopenmp. For details on the flags accepted by each compiler, see the man pages for each executable name.
|Fortran||pgf90 or pgf77||gfortran||ifort|
Once the proper module is loaded, the desired compilers are available to be called directly (eg, gfortran test.f).
As you can see using
module show PE-env,some libraries are automatically loaded with the programming environment. This will also provide any predefined environment variables defined when the module is loaded. Many other libraries are available via modules if you wish to, these may need to be added to the link line with the -l flag. Of course, you can install your own libraries if they are not provided, use the -L dir flag so the compiler can find them.
It should be noted that some libraries link against different library versions depending on the programming environment at the time they are loaded. It is best to change the programming environment first, and then load any additional modules. However, if the programming environment is changed after additional modules are loaded then the
module update <package> command can be used to reload these modules.
One can find a list of compiler options via investigating the man-page associated with the selected compiler. First, one would load the programming environment of their choice. There are no parallel compiler executables on Nautilus, in order to compile MPI codes, provide the
-lmpi flag. Please see examples later in this document.
Before making an application run faster, it is a good rule of thumb to make it run fast on one processor. Accessing memory efficiently, cache blocking, loop reordering, loop unrolling, vectorization, logic reordering, and using libraries all can assist you in creating a more efficient binary. Some of this can be done via modifying source code, if you do not know or feel comfortable in this can do this, certain compile time flags could help. In optimizing code, a deeper understanding of the code's performance is required. Another useful point when compiling is the use of output from the compiler of what is occurring (for PGI it is -Minfo).
When optimizing parallel code, it is best to begin with a small number of nodes and then scale up until a particular limiting number is reached. Then one can modify the number and size of MPI messages that are passed to attempt to surpass this limit. Some codes do not scale well, it depends on the type of problem and how it is coded. Some codes have SMP and multi-core makefiles, these are best suited for Nautilus.
icc -lmpi helloworld.c -o helloworld.x
SGI's Message Passaging Toolkit (MPT) is loaded by default. The -lmpi flag invokes the MPI library (no matter your compiler). Most codes will have predefined
mpif90 for their parallel builds. You must replace these with the serial copiler name, for instance the Intel builds,
CC -lmpi, and/or
ifort -lmpi should used instead of the typical parallel compiler name. For parallel C++ code, one needs to add
cc -fopenmp openmp_hello.c -o openmp_hello.x
With Fortran programs, you will also need to specify the "-fpp" Fortran precompiler option. OpenMP statements are actually comments, unless the compiler knows to look for them, hence the
-mp flag is needed (for PGI). For Intel and GNU compilers it is
Hybrid codes will require the use of the OpenMP flags and OMP_NUM_THREADS to be set in the PBS script. A detailed example can be found here, http://www.nics.tennessee.edu/tutorials/HybridHello
CUDA is installed and available for users. Simply find the version you would like to use and add it to your path:
module avail cuda module load cuda/4.0RC2
(to load version 4.0RC2)
The CUDA modules have libraries for CUDA and OpenCL programming. Specify either the "-lcuda" or "-lopencl" flags for CUDA or OpenCL, respectively.
There are a few libraries preinstalled on Nautilus. These include Intel's MKL, ATLAS, ACML, GLIB, HDF4 and 5, Python, TCL, NUPY, SCIPY, SILO, and LAPACK. For a list of these installed libraries and other software, please visit, http://www.nics.tennessee.edu/computing-resources/nautilus/software.
Nautilus has TotalView for a debugging tool available. When using TotalView, you will need to have X11 forwarding enable at log on (include the
-X flag in one's
ssh command). The package names for TotalView it is
totalview. If the process ID number is known, these can be attached to running processes. Typically, if it is a small job, the best route is to submit an interactive job, launch the debugger there and execute your program through that. If the program requires a large number of processes, usually creating a core dump file (place in PBS script
ulimit -c unlimited) and then use a debugger on the core dump file.
A common code tuning issue on Nautilus is the effectiveness of the code's threading. Since Nautilus is a large SMP machine, utilizing both threads for core binding processes and enough space for global variables for all threads to access can be a balancing act. One has to systematically work through these options to performance tune their application to this system. A good starting point is to look at makefiles for multi-core and SMP builds.
When you log in, your
ssh session has limited resources available, particularly a memory limit of only 4 GB. Therefore, you should not run your computing, analysis or visualization jobs directly from your
ssh login session but rather submit them to the batch system (Portable Batch System, or PBS) which manages the resources on Nautilus and performs job scheduling (via Moab). There are four queues available on Nautilus:
The two production queues on Nautilus are "
analysis" and "
analysis queues have a wall time of 24 hours. The
hpss queue can be used to transfer files or directories to hpss using a batch file. The
dmover queue is for moving data to or from any location (aside from HPSS) in a batch job. Jobs running in this queue are not allocated compute nodes. One may only submit jobs to the
hpss queue if you logged into a node with your RSA SecurID OTP token. Using job dependencies, you can schedule an hpss job to stage data before and/or after a normal production job.
The batch and interactive sessions are available. Batch jobs can be submitted to the queue via the qsub command. Batch mode is the typical method to submit production simulations. If one is not certain on how to construct a proper job execution command, it is beneficial to use the interactive queue.
Interactive submission For interactive jobs, PBS options are passed through qsub on the command line.
qsub -I -A XXXYYY -l ncpus=12,walltime=1:00:00
-I Start an interactive session
-A Charge to the
Putting it together
Request 12 compute cores for one hour
After running this command, you will have to wait until enough compute nodes are available, just as in any other batch job. However, once the job starts, the std input and std output of this terminal will be linked directly to the head node of our allocated resource. Issuing the exit command will end the interactive job. From here commands may be executed directly instead of through a batch script.
All non-interactive jobs must be submitted on Nautilus by a job script via the qsub command. All job scripts start with a series of #PBS directives that describe requirements of the job to the scheduler. The rest is a shell script, which sets up and runs the executable: the aprun command is used to launch one or more parallel executables on the compute nodes.
The following example shows a typical job script that includes the minimal requirements to submit a parallel job that executes ./a.out on 96 cores, charged to the fictitious account UT-NTNL0121 with a wall clock limit of one hour and 35 minutes:
#PBS -N hpl #PBS -A UT-NTNL0121 #PBS -S /bin/bash #PBS -j oe #PBS -l ncpus=96 #PBS -l mem=384GB #PBS -l walltime=01:35:00 cd /lustre/medusa/$USER mpirun -np 96 ./a.out
There is directly 4 GB attached to each processor. When requesting memory or processors, for whichever is highest you should specify 4GB of memory per processor on Nautilus. There is a known bug when requesting too few processors and a lot of memory.
Note: When using X11 forwarding for graphic displays (e.g. the
-X), you should use the PBS directive "
-v" (e.g. in a batch script "
#PBS -v DISPLAY"). This will port the
$DISPLAY environment variable to your compute processors.
For both batch and interactive jobs that utilize GPUs one can request this resource by "-l gpus=N". Currently, there is a limitation in that Moab interprets this value as GPUs-per-core, which may take some time to resolve. In the meanwhile, the following combinations are known to work:
To determine which GPUs one has allocated, the environment variable
$PBS_GPUS can be used. For example, here is the result for a job that requests four GPUs and four CPUs (-l ncpus=2,gpus=2):
login1$ cat $PBS_GPUS /dev/nvidia1 /dev/nvidia0 /dev/nvidia3 /dev/nvidia2
One can use the same PBS script as above with the added statement before the mpirun line,
export OMP_NUM_THREADS=96. In addition, one can place the generated threads via the
dplace command. Using the above example, the mpirun line becomes:
mpirun -np 96 dplace ./a.out
The parent process is launched from the first processor and the following children threads are launched from sequential cpus from the nodelist.
Jobs are submitted using the
To check the status of one's queued jobs, the
qstat command is available.
qstat -u <username>
To remove your job from the queue, use the
To see all running jobs on Nautilus, you can pass the
-r flag (
qstat -r). An important column in these command's output is the job state column, marked by
S. The job state (a.k.a. status) can be H (Held), Q (Queued), R (Running), W (Waiting), and C (recently Completed).
Moab has some useful job status commands: showq, checkjob, and showstart.
This command gives a different view of jobs in the queue. The utility will show jobs in the following states:
- Active : These jobs are currently running.
- Eligible : These jobs are currently queued awaiting resources. A user is allowed five jobs in the eligible state.
- Blocked : These jobs are currently queued but are not eligible to run. Common reasons for jobs in this state would be jobs on hold, or the owning user currently has five jobs in the eligible state.
This can be used to view details of a job in the queue. For example, if job 736 is currently in a blocked state, the following can be used to view the reason:
The return may contain a line similar to the following: BlockMsg: job 736 violates idle HARD MAXJOB limit of 2 for user (Req: 1 InUse: 2) This line indicates the job is in the blocked state because the owning user has reached the limit of five jobs currently in the eligible state. The Moab utility showstart gives an estimate of when the job will start.
showstart 100315 job 100315 requires 16384 procs for 00:40:00 Estimated Rsv based start in 15:26:41 on Fri Sep 26 23:41:12 Estimated Rsv based completion in 16:06:41 on Sat Sep 27 00:21:12
The start time may change dramatically as new jobs with higher priority are submitted, so you need to periodically rerun the command.
This utility provides the current backfill. This can help you create a job that can be backfilled immediately. As such, it is primarily useful for short jobs.
Benchmarking and profiling one's calculation is important on any resource. Understanding the necessary resources required for the calculation can be sought by plotting size vs. wall time or completion time, the
ja command. TotalView can also aid in profiling memory usage and bottlenecks in one's code.
All Nautilus policies concerning User responsibilities, directory spaces, HPSS storage, Grid Services, accounting & allocation status, job scheduling, and file system purges can be found here http://www.nics.tennessee.edu/policies
Last updated: June 25, 2012