Available Storage

Target Functionality and User Requirements

  • High Speed Computational Scratch Space
  • Distributed Workspace
  • Incremental Data Backup (1day, 1week, 1month, etc...)
  • Data Archive

Storage Allocations

All storage is allocated to users and attributed to their account sponsor (faculty advisor). To see the storage used by a particular faculty advisor simply type their ND ID into the following storage reporting tool:

http://www.crc.nd.edu/info/storage.html

Current Components

Panasas ActiveStor 16 Scratch Space (/scratch365) Overview

63 x Storage  Blades - each of them is 12TB 
14 x Director Blades
680TB  of available storage mounted via PanFS (/scratch365) using Panasas DirectFlow protocol

To view your quota on the /scratch365 filesystem use the command:

pan_df -H /scratch365/netid   # Note that the -H option gives the output in Gigabytes

/scratch365 filesystem limits include:

Maximum number of files per each directory in your volume - 350000
Maximum number of files per your volume                   - 10000000
Maximum number of users/volumes for our configuration     - 600

Note: /scratch365 space is temporary storage and files are automatically deleted after 365 days. There are no backups of these files and we recommend archiving important data to AFS or an external medium.

Typically new users are given quotas of 500 GB for the /scratch365/netid.

CRC AFS Cell

The CRC has setup an AFS cell for greater performance, flexibility, and reliability.

  • Sun "Thumpers"
3x Sun Fire X4540 servers "Thumpers", 48TB raw storage each with 48 x 1 TB SATA disks.
RAID 10 Configuration with ZFS filesystem, each with 34 TB available space
  • HP Fileservers
4x HP DL180 G6 servers, each with 34.544 TB available
RAID 6 Configuration with XFS filesystem
  • Dell Fileservers
13x Dell R510 servers, each with 34.525 TB available
RAID 6 Configuration with XFS filesystem

AFS filesystem limits include:

 Maximum number of files per directory - Typically 16,000
 Maximum number of files per volume    - In the millions

See script: /opt/crc/usr/local/bin/cell_usage for real time CRC AFS cell statistics

CRC Backup/Persistent Storage

The CRC has developed a backup infrastructure for automatic AFS backup, high speed caching, and long term archiving in order to provide storage backup and archive services to the ND Research Community.

Comparison

CRC Storage Comparisons


Storage Resources

The following table gives an overview of the available file system resources on the CRC Linux Clusters.

Purpose Segment of the Linux Cluster File system type and full name How the user should access the files Space available Approx. aggregated bandwidth Backup Lifetime, deletion strategy, and remarks
Globally accessible Home and Project Directories

User's Home Directories

All

AFS - crc.nd.edu

- /afs/crc.nd.edu/user/first/netid

Directly using Open Source OpenAFS client

$HOME

100 GB - 2 TB volume

- Approximately 70-85 MB/s per node

- Up to 100 MB/sec per node with a fast cache disk.

- Approximately 200 MB/s aggregated using multiple nodes

- AFS iozone benchmark

- In general daily backup by CRC

- Currently 3 months retention limit

- Online backup volumes can be easily accessed within a day (YESTRDAY)

- Expiration of affiliation with Notre Dame

- AFS quotas apply

- Best performance by running jobs on multiple nodes in separate directories due to callbacks of AFS

- Refer to CRC storage policy for Tier 1, 2, 3 allocation policy

Group Directories

All

AFS - crc.nd.edu

- /afs/crc.nd.edu/group

Directly using Open Source OpenAFS client

100 GB - 2 TB volume

- NOTE: Multiple volumes may be used for larger storage needs

- Approximately 70-85 MB/s

- AFS iozone benchmark

- In general daily backup by CRC

- Currently one year retention limit

- Online backup volumes can be easily accessed within a day (YESTRDAY)

- Expiration of affiliation with Notre Dame

- AFS quotas apply

- Refer to CRC storage policy for Tier 1, 2, 3 allocation policy

Temporary File Systems

Panasas High Performance Parallel scratch /scratch365 file system

Panasas ActiveStor 16 Installed June 2015

All

/scratch365/netid

Directly using proprietary Panasas panfs client

500 GB - 1 TB volume

- Approximately 70 - 90 MB/sec per node with 1 Gb network

- Approximately 250 - 500 GB/sec with 10 Gb network

- Approximately 10 GB/sec aggregated using multiple nodes (50 - 100)

- /scratch365 iozone benchmark

None

Sliding window file deletion - 365 days - No guarantee for data integrity

- Refer to CRC storage policy for Tier 1, 2, 3 allocation policy

Focus is on performance not reliability

Performance using small files somewhat reduced

Local file Systems

node-local temporary /scratch filesystem - /tmp

All

local disks /scratch is a link to /tmp

Directly - shared with other users on node

R815 - 500 GB

HP DL160 - d6copt 100 GB

IBM nx360M4 - 400 GB

Lenovo nx360M5 - 1 TB

daccssfe - 5 TB

R815 - H700 RAID controller 250 - 300 MB/sec

HP DL160 -d6copt 50 - 60 MB/sec

IBM nx360M4 90 - 100 MB/sec

Lenovo nx360M5: 130 - 160 MB/sec

daccssfe: 800 - 1,000 MB/sec

None

Batch Nodes: files deleted using tmpwatch if older than 10 days - Users encouraged to remove files at end of run. /tmp cleared at reboot.

Login Nodes: files deleted using tmpwatch if older than 10 days - /tmp retained at reboot.

Users may find contention with inconsiderate users