XSEDE Resource Allocation Committee (XRAC) Announcements
XSEDE is now accepting Research Allocation Requests for the allocation period, July 1, 2017 to June 30, 2018. The submission period is from March 15, 2017 thru April 15, 2017. Please review the new XSEDE systems and important policy changes before submitting your allocation request through the XSEDE User Portal
The Research allocation submission period is open: March 15, 2017 - April 15, 2017.
Starting this submission period both the Pittsburgh Supercomputing Center (PSC) and the San Diego Supercomputer Center (SDSC) will be allocating their GPU compute resources separately from their standard compute nodes. See SDSC's Comet GPU and PSC's Bridges GPU resource details below. TACC's Stampede 2 system also debuts its availability and will begin production in Summer, 2017.
See the XSEDE Resources Overview for a complete list of XSEDE compute, visualization and storage resources, and more details on the new systems.
TACC's newest resource, Stampede 2, will enter full production in the Fall 2017 as the 18 petaflop national resource that builds on the successes of the original Stampede system it replaces. The first phase of the Stampede 2 rollout features the second generation of processors based on Intel's Many Integrated Core (MIC) architecture. These 4,200 Knights Landing (KNL) nodes represent a radical break with the first generation Knights Corner (KNC) MIC coprocessor. Unlike the legacy KNC, a Stampede KNL is not a coprocessor: each 68-core KNL is a stand-alone, self-booting processor that is the sole processor in its node. Phase 2 will add approximately 50% of additional compute power to the system as a whole by introducing new nodes equipped with a future Intel processor. When fully deployed, Stampede 2 will deliver twice the performance of the original Stampede system. Please note that Stampede 2 is allocated in service units (SU)s where an SU is defined as 1 wall-clock node hour (not core hour).
SDSC's Comet GPU has 36 general purpose GPU nodes, with 2 Tesla K80 GPU graphics cards per node, each with 2 GK210 GPUs (144 GPUs in total). Each GPU node also features 2 Intel Haswell processors of the same design and performance as the standard compute nodes (described separately). The GPU nodes are integrated into the Comet resource and available through the SLURM scheduler for either dedicated or shared node jobs (i.e., a user can run on 1 or more GPUs/node and will be charged accordingly). Like the Comet standard compute nodes, the GPU nodes feature a local SSD which can be specified as a scratch resource during job execution - in many cases using SSD's can alleviate I/O bottlenecks associated with using the shared Lustre parallel file system.
Comet's GPUs are a specialized resource that performs well for certain classes of algorithms and applications. There is a large and growing base of community codes that have been optimized for GPUs including those in molecular dynamics, and machine learning. GPU-enabled applications on Comet include: Amber, Gromacs, BEAST, OpenMM, TensorFlow, and NAMD.
PSC introduces Bridges GPU, a newly allocatable resource within Bridges that features 32 NVIDIA Tesla K80 GPUs and 64 NVIDIA Tesla P100 GPUs. Bridges GPU complements Bridges' Regular, Bridges Large, and its Pylon storage system to accelerate deep learning and a wide variety of application workloads. The 16 GPU nodes, each with 2 NVIDIA Tesla K80 GPU cards, 2 Intel Xeon CPUs (14 cores each), and 128GB of RAM and 32 GPU nodes, each with 2 NVIDIA Tesla P100 GPU cards, 2 Intel Xeon CPUs (16 cores each), and 128GB of RAM.
The PSC's Bridges is a uniquely capable resource for empowering new research communities and bringing together HPC and Big Data. Bridges integrates a uniquely flexible, user-focused, data-centric software environment with very large shared memory, a high-performance interconnect, and rich file systems to empower new research communities, bring desktop convenience to HPC and drive complex workflows.
Bridges supports new communities through extensive interactivity, gateways, persistent databases and web servers, high productivity programming languages, and virtualization. The software environment is extremely robust, supporting enabling capabilities such as Python, R, and MATLAB on large-memory nodes, genome sequence assembly on nodes with up to 12TB of RAM, machine learning and especially deep learning, Spark and Hadoop, complex workflows, and web architectures to support gateways.
|Jetstream (IU/TACC)||5,000,000 vCPU hours|
|SuperMIC (LSU)||6,500,000 node hours|
|OSG||2,000,000 CPU hours|
|Bridges Regular Memory (PSC)||38,000,000 core hours|
|Bridges Large Memory (PSC)||700,000 core hours|
|Bridges GPU (PSC)||TBD|
|Pylon (PSC)||2,000,000 TB|
|Comet GPU (SDSC)||TBD|
|Data Oasis (SDSC)||300,000 TB|
|XStream (Stanford) |
Cray CS-Storm GPU Supercomputer
|500,000 GPU hours|
|Maverick (TACC)||4,000,000 core hours|
|Stampede2 - Phase 1 (TACC) |
TACC Dell/Intel Knight's Landing System
|10,000,000 node hours|
|Wrangler (TACC) |
Data Analytics System
|180,000 node hours|
|Wrangler Storage (TACC)||500,000 TB|
|Ranch (TACC) |
Long-term tape Archival Storage
These retiring resources, listed below, are not available for new or renewal research requests.
- TACC's Stampede 1 System (July 2017)
- SDSC's Gordon Compute Cluster (March 2017)
Continuing this submission period, access to XSEDE storage resources along with compute resources will need to be requested and justified, both in the XSEDE Resource Allocation System (XRAS) and in the body of the proposal's main document. The following XSEDE sites will be offering allocatable storage facilities, these are:
Storage needs have always been part of allocation requests, however, XSEDE will be enforcing the storage awards in unison with the storage sites. Please vist XSEDE's Storage page for more info.
In the past code performance and scaling was to be a section addressed in all research requests main document, this section seems to have been overlooked by many PIs in the recent quarterly research submission periods which has led to severe reductions or even complete rejection of both new and renewal requests. Continuing this quarterly submission period it will be mandatory to upload a scaling and code performance document detailing your code efficiency. Please see section 7.2 Review Criteria under Allocations Policies.
It is mandatory to discuss and detail the disclosure of access to other cyberinfrastructure resources (e.g., NSF Blue Waters, DOE INCITE resources, local campus, ...) in the proposal's main document. Please see section 7.3 Access to Other CI Resources in the Allocations Policies. The failure to disclose access to these resources could lead to severe reductions or even complete rejection of both new and renewal requests. If there is no access to other cyberinfrastructure resources this should be made clear as well.
The XRAC review panel requests that PIs include the following in their requests:
The description of the computational methods must include explicit specification of the integration
time step value, if relevant (e.g. Molecular Dynamics Simulations). If these details are not provided
a 1 femtosecond (1fs) will be assumed with this information being used accordingly to evaluate the
All funding used to support the Research Plan of an XRAC Research Request must be reported in the Supporting Grants form in the XRAS submission. Reviewers use this information to assess whether the PI has enough support to accomplish the Research Plan, analyze data, prepare publications, etc.
Publications that have resulted from the use of XSEDE resources should be entered into your XSEDE portal profile which you will be able to attach to your Research submission.
Also note that it is expected that the scaling and code performance information is from the resource(s) being requested in the research request.
Reference: XSEDE Allocations Policies document
Storage allocation requests for Archival Storage in conjunction with compute and visualization resources and/or Stand Alone Storage need to be requested explicitly both in your proposal (research proposals) and also in the resource section of XRAS.
Furthermore, the PI must describe the peer-reviewed science goal that the resource award will facilitate. These goals must match or be sub-goals of those described in the listed funding award for that year.
After the Panel Discussion of the XRAC meeting, the total Recommended Allocation is determined and compared to the total Available Allocation across all resources. Transfers of allocations may be made for projects that are more suitable for execution on other resources; transfers may also be made for projects that can take advantage of other resources, hence balancing the load. When the total Recommended considerably exceeds Available Allocations a reconciliation process adjusts all Recommended Allocations to remove oversubscription. This adjustment process reduces large allocations more than small ones and gives preference to NSF-funded projects or project portions. Under the direction of NSF, additional adjustments may be made to achieve a balanced portfolio of awards to diverse communities, geographic areas, and scientific domains.
Conflict of Interest (COI) policy will be strictly enforced for large proposals. For small requests, the PI/reviewer may participate in the respective meeting, but leave the room during the discussion of their proposal.
XRAC proposals for allocations request resources representing a significant investment of the National Science Foundation. The XRAC review process therefore strives to be as rigorous as for equivalent NSF proposals.
The actual availability of resources is not considered in the review. Only the merit of the proposal is. Necessary reductions due to insufficient resources will be made after the merit review, under NSF guidelines, as described in Section 6.4.1.
10% max advance on all research requests, as described in Section 3.5.4.
Please visit the Submitting a Successful Research Allocation Request page for more information on writing a successful research proposal as well as examples of successful research allocation requests.
If you would like to discuss your plans for submitting a research request please send email to the XSEDE Help Desk at email@example.com. Your questions will be forwarded to the appropriate XSEDE Staff for their assistance.
Beginning April 1, 2015, XSEDE will offer Trial Allocations. Trial Allocations are designed to give potential users rapid, but limited access to XSEDE resources. Within one business day, users will be able to log on and evaluate an XSEDE resource prior to requesting a larger startup or research allocation. Trial allocation sizes will be limited, but sufficient, for initial resource evaluation. For compute resources it will give new users the ability to compile and run software, and evaluate results, within the XSEDE resource software and hardware environment.
Trial Allocations are currently offered only on SDSC's Comet resource. To apply for a trial allocation, create an XSEDE User Portal (XUP) account. Once you have your XUP account, please submit a ticket via the XSEDE help-desk.
|Resource||Trial Allocation limit||Request a Trial Account|
|Comet||1000 SUs, 6 months||Please submit an XSEDE help-desk ticket|
Last update: March 16, 2017