What is XSEDE?
The Extreme Science and Engineering Discovery Environment(XSEDE) is a powerful collection of virtual resources and a way for scientists and researchers alike to interactively share resources and expertise. XSEDE provides an SSO(Single Sign On) Hub for access to several national computing clusters. You must have an XSEDE account in order to use XSEDE's resources. See the Xsede at Notre Dame Page for more information.
Using XSEDE's resources
The easiest way to access XSEDE's resources is through its SSO(Single Sign On) login hub. Once you sign onto this hub, you can access the clusters that XSEDE provides (assuming your XSEDE account has the proper clearance) without additional credentials. Using the SSO is easy, to connect to it use any SSH client you like (Linux and Mac users can use their terminal and Windows users can download MobaXterm ).
To use the SSO and other features, you will need to enroll in DUO authentication. Please be sure you already have an XSEDE account before attempting any of the following tips.
- First sign into XSEDE's user portal located here.
- Find the tab towards the top labled "Profile" and click on it.
- On the right hand side of the page, there will be the DUO logo and a link to enroll in DUO. Follow the steps on screen and setup a DUO token to be used with XSEDE.
- Once you finish enrolling your DUO token, you can now have secure access to XSEDE's SSO login node.
ssh into Login Hub
NOTE: You must enroll in DUO authentication to use the SSO / Login Hub !
Once you have access to an SSH client, you can login to XSEDE.
- The first step is to type:
ssh Your_XSEDE_User_Name@login.xsede.orgYour ssh client may give you a warning about connecting to this server, if so, type yes and press Enter.
- You will then be prompted for your XSEDE password, this is the same one you used when you created your XSEDE account. Note: You will not see anything as you type in your password, this is a security feature.
- XSEDE will then present you with some options for DUO. Pick the desired action. See Below
- Once you are logged on, XSEDE will display the MOTD(message of the day) with some system information and tips for logging into its resources. If you want to see this message again, type:
- Once you login to the SSO Hub, you will receive an X.509 certificate which will give you 12 hours before your session 'expires' and you will have to logout of the SSO hub and log back in, even if you are using one of the XSEDE's resources (i.e. Stampede, Comet, etc). You can see your remaining time left with the following command:
GSISSH into Clusters
To access super-computing clusters through XSEDE, your account must have the 'clearance' to login to a specific cluster. Go to the XSEDE website and login, there you will see what clusters you have access to under My XSEDE Resources. If you have access to a cluster, you can login to it through the XSEDE SSO without being required to enter extra credentials. The protocol used to gain access to these clusters is gsissh, which is gsi-enabled ssh. It operates in a similar manner as ssh--for us, the user, we won't notice a difference. Scroll below to find the specific cluster you're looking for, if it is not found within this page you can visit XSEDE's user Guides and click on the cluster you're trying to access for more information.
To get a quick overview of all possible systems the XSEDE SSO can connect to, type: xsede-gsissh-hosts. Note that you must have access to a cluster before logging into it.
[UserName@ssohub ~]$ xsede-gsissh-hosts bridges comet mason maverick osg rmacc-summit stampede stampede2 supermic wrangler-iu wrangler-tacc xstream
NOTE:Stampede has reached its 4 year life cycle and the cluster is retiring. For continued use of TACC resources, Stampede-2 must be used.
To assist in the transfer from Stampede to Stampede-2, please see the Transition Guide.
Stampede-2 is the new flagship supercomputer at Texas Advanced Computing Center (TACC). It begins full production in Fall of 2017. Currently, both Stampede and Stampede-2 will coexist, however in the future Stampede will be retiring leaving Stampede-2 as the supercomputer for production use. Stampede-2 has completed its two infrastructure implementation phases. Phase 1 included 4,200 KNL (Knights Landing) compute nodes consisting of:
- Intel Xeon Phi 7250 with 68 cores on a single socket
- 4 hardware threads per core totalling 272 threads on a single node
- 96 GB of DDR4 RAM in addition to 16GB high speed MCDRAM. See Programming Notes for more info.
Phase 2 included 1,736 SKX nodes consisting of:
- Intel Xeon Platinum 8160 "Skylake" processors with 48 cores on each node with a clock-rate of 2.1 GHZ.
- 192 GB of RAM per node
- 132 GB in /tmp on a SSD
Both Phase 1 and Phase 2 include a 100GB/sec Intel Omni-Path (OPA) network. Large memory nodes are expected to arrive in 2018. There are currently no plans for GPU systems in Stampede-2.
Logging into Stampede-2
To access Stampede-2 through XSEDE's SSO, simply enter
gsissh stampede2. Note that you must have an allocation with Stampede to login to the supercomputer. If you could access Stampede, you should be able to access Stampede-2. To find more information on Stampede-2, see TACC's User Guide.
For help with submitting jobs to Stampede-2, check out the Stampede-2 job submission page.
Comet is a dedicated XSEDE cluster with 1,984 total compute nodes which can reach ~2.0 petaflops designed by Dell. The compute nodes contain Intel Xeon E5-2680v3's, 128 GB of RAM, and 320 GB of local scratch memory. There are GPU nodes that have 4 NVIDIA GPU's per each GPU node. There are also large memory nodes which contain 1.5 TB of RAM with 4 Intel Haswell processors each. The cluster uses CentOS as the OS and SLURM (just like Stampede) as the batch environment. Comet provides Intel, PGI, and GNU compilers.
Logging into Comet through SSO
- Comet can be accessed through XSEDE's SSO. Once logged into the SSO, you can access the Comet cluster through the following command:
- If you have clearance to be on Comet, you will now be on a front-end for Comet.
- If you would like to see information for submitting jobs into Comet, visit our Comet Job submission page.
- For more information on the Comet cluster, visit XSEDE's Comet page.