Setting up a ceph file system

This guide provides brief instructions for setting up a ceph file system. I used RHEL6 with 2.6.32 kernel to support a btrfs file system for ceph (btrfs is ceph's preferred file system).

Note: ceph and btrfs are still experimental--they are not intended for production use with valuable data!

System Overview

ceph consists of (at least) one metadata server, one monitoring server and one storage server. For redundancy and increased performance, multiple storage servers and metadata servers can be dynamically added to the system as the storage needs grow. The system re-balances (or replicates) the metadata and storage load automatically to take advantage of new nodes or usage pattern changes.


1) Obtain ceph-0.20.2 from http://ceph.newdream.net/2010/download/ceph-0.20.2.tar.gz or /afs/crc.nd.edu/user/n/nregola/Public/ceph.

2) Obtain the libedit-devel library for your distribution (it is available at the above-referenced AFS directory for RHEL6 since RHEL6 only includes libedit, not libedit-devel in the BETA release)

3) install libedit-devel and your distribution's equivalent of btrfs-progs (utilities to format and mount a btrfs file system)

4) gunzip and untar the ceph-0.20.2.tar.gz file.

5) run ./autogen.sh in the ceph-0.20.2 directory

6) run ./configure

7) run make (this takes about 30 minutes on an Opteron 248, so go take a break while you wait for this to compile)

8) run make install (under a minute)

9) setup the necessary directories for ceph so it doesn't produce warnings

mkdir /etc/ceph
 mkdir /var/log/ceph

10) create SSH keys on one of the servers to SSH to allow automatic SSH to the other nodes when you create the filesystem ssh-keygen -d and then copy the public key to the other servers with scp /root/.ssh/id_dsa.pub slave:/root/.ssh/authorized_keys and then ssh slave chmod 600 /root/.ssh/authorized_keys

11) place your ceph.conf file in /etc/ceph/

12) use your SSH keys to copy the ceph.conf file to ALL NODES that will run ceph server components (the clients DO NOT NEED the ceph.conf file or any of the ceph server software that we compiled)

13) create the journal directory if it is a different btrfs file system than the data storage location. For example, my journal location for the osd nodes is /btrfs_journal, while the data storage location is /btrfs. /btrfs is automatically mounted by the ceph init.d script, but /btrfs_journal is not. You should ensure that the journal location is mounted at system startup by /etc/fstab until the code is updated to handle automatic mounts of the journal location. My /ets/fstab line for the journal location looks like /dev/sda3 /btrfs_journal btrfs rw,noatime 0 2

--Portions of this guide were based on the ceph wiki at http://ceph.newdream.net/wiki.

--updated --Nregola 12:29, 15 June 2010 (EDT)