Mon Failed to Start

Some common problems when adding a monitor to an existing cluster, for example if config is not found :

 $ service ceph start mon.ceph-03
 /etc/init.d/ceph: mon.ceph-03 not found (/etc/ceph/ceph.conf defines osd.2 , /var/lib/ceph defines osd.2)

If you do not want to specify a section mon.ceph-03 in ceph.conf, you need to have a file sysvinit in /var/lib/ceph/mon/ceph-ceph-03/

$   ls -l /var/lib/ceph/mon/ceph-ceph-03/
total 8
-rw-r--r-- 1 root root   77 août  29 16:56 keyring
drwxr-xr-x 2 root root 4096 août  29 17:03 store.db

Just create the file, then it should start :

$ touch /var/lib/ceph/mon/ceph-ceph-03/sysvinit
$ service ceph start mon.ceph-03
=== mon.ceph-03 === 
Starting Ceph mon.ceph-03 on ceph-03...
failed: 'ulimit -n 32768;  /usr/bin/ceph-mon -i ceph-03 --pid-file /var/run/ceph/mon.ceph-03.pid -c /etc/ceph/ceph.conf '
Starting ceph-create-keys on ceph-03...

Next error on starting monitor, if you have a look to log you can see :

$ tail -f ceph-mon.ceph-03.log
mon.ceph-03 does not exist in monmap, will attempt to join an existing cluster
no public_addr or public_network specified, and mon.ceph-03 not present in monmap or ceph.conf

You shoud verify, that you do not have ceph-create-keys process that hang, if so you can kill it :

$ ps aux | grep create-keys
root      1317  0.1  1.4  36616  7168 pts/0    S    17:13   0:00 /usr/bin/python /usr/sbin/ceph-create-keys -i ceph-03

$ kill 1317

Verify that you have defined this mon on the current monmap

$  ceph mon dump
dumped monmap epoch 6
epoch 6
fsid e0506c4d-e86a-40a8-8306-4856f9ccb989
last_changed 2013-08-29 16:58:06.145127
created 0.000000
0: 10.2.4.10:6789/0 mon.ceph-01
1: 10.2.4.11:6789/0 mon.ceph-02
2: 10.2.4.12:6789/0 mon.ceph-03

You need to retrieve the current monmap and add if for this node :

$ ceph mon getmap -o /tmp/monmap
2013-08-29 17:36:36.204257 7f641a54d700  0 -- :/1005682 >> 10.2.4.12:6789/0 pipe(0x2283400 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x2283660).fault
got latest monmap

$ ceph-mon -i ceph-03 --inject-monmap /tmp/monmap

Try again :

$ service ceph start mon.ceph-03
=== mon.ceph-03 === 
Starting Ceph mon.ceph-03 on ceph-03...
Starting ceph-create-keys on ceph-03...

It’s seems working fine. You can verify the state of monitor quorum :

$ ceph mon stat
e6: 3 mons at {ceph-01=10.2.4.10:6789/0,ceph-02=10.2.4.11:6789/0,ceph-03=10.2.4.12:6789/0}, election epoch 1466, quorum 0,1,2 ceph-01,ceph-02,ceph-03

For more information, have a look to the documentation : http://ceph.com/docs/master/rados/operations/add-or-rm-mons/