NDCMS NDPC

¡Please edit this Wiki to reflect the current state of the ND PCs!

General Info

  • the new machines (as of early 2016) are ndpc5, ndpc6
  • CMSSW is delivered by CVMFS (no manual installation needed)
  • storage is in the following directories:
 * /store/ndpc5disk1
 * /store/ndpc5disk2
 * /store/ndpc6disk1
 * /store/ndpc6disk2
  • storage directories will be mounted as needed. if they are not present, try accessing them!

Adding Users

  • log in as root (via `su` after logging in as regular user)
  • add user (on both machines):
   addusercern <user>
  • create user directories (only on one machine, the {} will get expanded by the shell):
   mkdir /store/ndpc{5,6}disk{1,2}/<user>
   chown <user>:zh /store/ndpc{5,6}disk{1,2}/<user>

Installation Log

Following the instructions at CERN doc:

  • register ndpc with the CERN network services
  • wait for pc to register with network services
  • hit enter at the PXE screen
  • select "Scientific Linux CERN 6, 64 bit" at the boot menu
  • select us/us for language keyboard
  • select simple/basic storage configuration
  • Next click, Discard any data.
  • Select hostname: ndpc5.cern.ch/ndpc6.cern.ch
  • hit enter a couple times more
  • root password: "vegetable soup"
  • select "use all space" as partitioning scheme
  • select the "ATA LITEONIT" to install on (the SSD)
  • proceed through formatting/disk setup
  • use default software setup
  • reboot
  • let the setup take care to hook it up with CERN

Post-installation setup (from the instructions) to be done if needed.

Registering the main user:

/usr/sbin/cern-config-users <username>

Other users:

addusercern <username>

Adding CVMFS

You need to root access to the machine to apply this recipe.

Add CVMFS repository:

 wget -O /etc/yum.repos.d/cernvm.repo http://cvmrepo.web.cern.ch/cvmrepo/yum/cernvm.repo

And import the GPG key of the repository:

 wget -O /etc/pki/rpm-gpg/RPM-GPG-KEY-CernVM http://cvmrepo.web.cern.ch/cvmrepo/yum/RPM-GPG-KEY-CernVM

Install the required RPMs. Make sure that your OS repositories are enabled, since it is quite likely that ascp matze@lxplus:/etc/cvmfs/default.local /etc/cvmfs/default.localdditional system RPMs, e.g. fuse and autofs, need to be installed in order to resolve the dependencies.

 yum install cvmfs cvmfs-init-scripts

Make sure FUSE is setup right by editing /etc/fuse.conf to contain

 user_allow_other

Configure /etc/cvmfs/default.local:

 CVMFS_HTTP_PROXY="http://ca-proxy.cern.ch:3128;http://ca-proxy1.cern.ch:3128%7Chttp://ca-proxy2.cern.ch:3128%7Chttp://ca-proxy3.cern.ch:3128%7Chttp://ca-proxy4.cern.ch:3128%7Chttp://ca-proxy5.cern.ch:3128"
 CVMFS_REPOSITORIES='cms-ib.cern.ch,cms.cern.ch,'
 CVMFS_QUOTA_LIMIT=20000

2016/05/26: Increased quota

Make sure that /etc/cvmfs/config.d/cms.cern.ch.conf contains

 export CMS_LOCAL_SITE=T2_CH_CERN

Restart autofs:

 service autofs restart

Run some cvmfs commands:

 cvmfs_config setup
 cvmfs_config chksetup

See if anything shows up under /cvmfs.

Storage configuration

Install gparted for some comfort:

 yum install gparted

Execute gparted

  • Look for the large unformatted disks in the dropdown on the top right
  • Then click on "new" to create a new partition
  • Don't worry about "Free space preceeding", just leave the default
  • File system should be "ext4"
  • Label could be something "ndpc5disk1" etc
  • Click "Add"
  • Repeat for all other new hard drives
  • Click "Apply"
  • Confirm
  • Get a coffee, snack, etc.

Prepare mount points:

 mkdir -p /mnt/ndpc5disk1
 mkdir -p /mnt/ndpc5disk2

Look for the labels:

 blkid

Mount the created partitions by appending the following to /etc/fstab

 LABEL=ndpc5disk1 /mnt/ndpc5disk1 ext4 defaults 1 2
 LABEL=ndpc5disk2 /mnt/ndpc5disk2 ext4 defaults 1 2

Then execute:

 mount /mnt/ndpc5disk1
 mount /mnt/ndpc5disk2

NFS setup

Edit /etc/exports to look like the following:

 /mnt/ndpc5disk1 ndpc5.cern.ch(rw,no_root_squash) ndpc6.cern.ch(rw,no_root_squash)
 /mnt/ndpc5disk2 ndpc5.cern.ch(rw,no_root_squash) ndpc6.cern.ch(rw,no_root_squash)

Making sure that the disks can only be mounted on ND machines. Start nfs

 service nfs start

Make sure it gets started at boot

 chkconfig nfs on

Edit the firewall by adding the following to /etc/sysconfig/iptables after *filter and before :INPUT...

 -A INPUT -m state --state NEW -m tcp -p tcp --dport 2049 -j ACCEPT
 -A INPUT -m state --state NEW -m tcp -p tcp --dport 111 -j ACCEPT
 -A INPUT -m state --state NEW -m tcp -p tcp --dport 32803 -j ACCEPT
 -A INPUT -m state --state NEW -m tcp -p tcp --dport 892 -j ACCEPT
 -A INPUT -m state --state NEW -m tcp -p tcp --dport 875 -j ACCEPT
 -A INPUT -m state --state NEW -m tcp -p tcp --dport 662 -j ACCEPT

Restart the firewall:

 service iptables restart

Now the NFS service should be accessible from the other machine.

Create a directory to mount the disks in:

 mkdir -p /store

Edit /etc/auto.master by appending:

 /store /etc/auto.store

Create /etc/auto.store with:

 ndpc5disk1      -rw ndpc5.cern.ch:/mnt/ndpc5disk1
 ndpc5disk2      -rw ndpc5.cern.ch:/mnt/ndpc5disk2
 ndpc6disk1      -rw ndpc6.cern.ch:/mnt/ndpc6disk1
 ndpc6disk2      -rw ndpc6.cern.ch:/mnt/ndpc6disk2

Restart the autofs service

 service autofs reload

Test:

 ls /store
 ls /store/ndpc5disk1
 ls /store/ndpc6disk1
 ls /store

These directories are created dynamically when needed.

Additional Software

Added voms stuff 2016-02-15:

 yum-config-manager --enable igtfca
 yum install voms-clients fetch-crl ca-policy-egi-core myproxy sl
 scp -r matze@lxplus.cern.ch:/etc/fetch-crl.conf /etc
 scp -r matze@lxplus.cern.ch:/etc/grid-security/vomsdir/cms /etc/grid-security/vomsdir/cms
 scp -r matze@lxplus.cern.ch:/etc/vomses /etc/vomses

Additional Configuration

Cache size increases

Added 2016-05-26:

  • Increased CVMFS cache, see modified instructions above
  • Increased AFS cache from 0.5GB to 10GB as below:

Editing /usr/vice/etc/cacheinfo to read:

 /afs:/usr/vice/cache:10240000

restart AFS with:

 service afs restart

TRIM support for SSDs

Added 2016-06-02

Edit /etc/lvm/lvm.conf and set the following option to 1:

 issue_discards = 1

Trimming can be run automatically after every file-system level delete by adding ,discard to /etc/fstab or manually. Since the latter is supposed to stress the system less, let's do that.

Add the following to /etc/rc.local:

 for fs in $(lsblk -o MOUNTPOINT,DISC-MAX,FSTYPE | grep -E '^/.* [1-9]+.* ' | awk '{print $1}'); do
     fstrim "$fs"
 done

And add this to /etc/crontab so that trimming runs once a day at 4:00 in the morning:

 0 4 * * * root /etc/rc.local

Reboot!

Disabling outdated software repo

Done 2016-06-13

 yum-config-manager --disable igtfca

This will potentially break updates to some of the VOMS software installed from this repository (see above).

Updating voms-client to work again

Done 2016-06-23

Fix the carnage from the last step:

 rpm --import http://emisoft.web.cern.ch/emisoft/dist/EMI/3/RPM-GPG-KEY-emi
 wget http://emisoft.web.cern.ch/emisoft/dist/EMI/3/sl6/x86_64/base/emi-release-3.0.0-2.el6.noarch.rpm
 yum localinstall emi-release-3.0.0-2.el6.noarch.rpm
 yum install voms-clients3

Installing Cactus

Done 2016-09-16

Install cactus. Used for upgrade hardware development https://svnweb.cern.ch/trac/cactus

Steps to install:

 yum groupremove uhal
 wget https://svnweb.cern.ch/trac/cactus/export/34680/tags/ipbus_sw/uhal_2_4_0/scripts/release/cactus.slc6.x86_64.repo
 cp cactus.slc6.x86_64.repo /etc/yum.repos.d/cactus.repo
 yum clean all
 yum groupinstall uhal
 export LD_LIBRARY_PATH=/opt/cactus/lib:$LD_LIBRARY_PATH
 export PATH=/opt/cactus/bin:$PATH

Blocking remote root login

Done 2017-10-06

In /etc/ssh/sshd_config, change:

 #PermitRootLogin yes

to

 PermitRootLogin no

and execute:

 /etc/init.d/sshd restart

ImageMagick installed

Done 2017-11-13

 yum install ImageMagick

Usage:

 display xyz.[png,jpg,etc.]

gedit plugins installed

Done 2018-1-25 (Requested by Tony L.)

 sudo yum install gedit-plugins