Feature Set Mismatch Error on Ceph Kernel Client
Depending on the kernel version you may be missing some features required by the cluster (or vice versa, required by client but not available on the cluster). Hence the errors “feature set mismatch”.
Some examples of errors that can be encountered :
mon0 192.168.0.1:6789 feature set mismatch, my XXXXXX < server's XXXXXX, missing 2040000
—> Upgrade kernel client up to 3.9 or set tunables to legacy : ceph osd crush tunables legacy
mon0 192.168.0.1:6789 feature set mismatch, my XXXXXX < server's XXXXXX, missing 40000000
—> Upgrade kernel client up to 3.9 or unset hashpspool : ceph osd pool set rbd hashpspool false
mon0 192.168.0.1:6789 feature set mismatch, my XXXXXX < server's XXXXXX, missing 800000000
—> Remove cache pool and reload monitors or upgrade kernel client up to 3.14
mon0 192.168.0.1:6789 feature set mismatch, my XXXXXX < server's XXXXXX, missing 1000000000
—> Upgrade kernel client up to 3.14
mon0 192.168.0.1:6789 feature set mismatch, my 4a042a42 < server's 2004a042a42, missing 20000000000
—> Upgrade kernel client up to 3.15 or disable tunable 3 features
CEPH_FEATURE Table and Kernel Version ¶
You can find the feature missing in that table :
For exemple, missing 2040000 means that CEPH_FEATURE_CRUSH_TUNABLES (40000) and CEPH_FEATURE_CRUSH_TUNABLES2 (2000000) is missing on kernel client.
‘R’:required, ’S’:support, ‘-X-’ feature is new since this version
| Feature | BIT | OCT | 3.8 | 3.9 | 3.10 | 3.14 | 3.15 | 3.18 |
|---|---|---|---|---|---|---|---|---|
| CEPH_FEATURE_UID | 0 | 1 | ||||||
| CEPH_FEATURE_NOSRCADDR | 1 | 2 | R | R | R | R | R | R |
| CEPH_FEATURE_MONCLOCKCHECK | 2 | 4 | ||||||
| CEPH_FEATURE_FLOCK | 3 | 8 | ||||||
| CEPH_FEATURE_SUBSCRIBE2 | 4 | 10 | ||||||
| CEPH_FEATURE_MONNAMES | 5 | 20 | ||||||
| CEPH_FEATURE_RECONNECT_SEQ | 6 | 40 | -R- | R | R | R | ||
| CEPH_FEATURE_DIRLAYOUTHASH | 7 | 80 | ||||||
| CEPH_FEATURE_OBJECTLOCATOR | 8 | 100 | ||||||
| CEPH_FEATURE_PGID64 | 9 | 200 | R | R | R | R | R | |
| CEPH_FEATURE_INCSUBOSDMAP | 10 | 400 | ||||||
| CEPH_FEATURE_PGPOOL3 | 11 | 800 | R | R | R | R | R | |
| CEPH_FEATURE_OSDREPLYMUX | 12 | 1000 | ||||||
| CEPH_FEATURE_OSDENC | 13 | 2000 | R | R | R | R | R | |
| CEPH_FEATURE_OMAP | 14 | 4000 | ||||||
| CEPH_FEATURE_MONENC | 15 | 8000 | ||||||
| CEPH_FEATURE_QUERY_T | 16 | 10000 | ||||||
| CEPH_FEATURE_INDEP_PG_MAP | 17 | 20000 | ||||||
| CEPH_FEATURE_CRUSH_TUNABLES | 18 | 40000 | S | S | S | S | S | S |
| CEPH_FEATURE_CHUNKY_SCRUB | 19 | 80000 | ||||||
| CEPH_FEATURE_MON_NULLROUTE | 20 | 100000 | ||||||
| CEPH_FEATURE_MON_GV | 21 | 200000 | ||||||
| CEPH_FEATURE_BACKFILL_RESERVATION | 22 | 400000 | ||||||
| CEPH_FEATURE_MSG_AUTH | 23 | 800000 | -S- | |||||
| CEPH_FEATURE_RECOVERY_RESERVATION | 24 | 1000000 | ||||||
| CEPH_FEATURE_CRUSH_TUNABLES2 | 25 | 2000000 | S | S | S | S | S | |
| CEPH_FEATURE_CREATEPOOLID | 26 | 4000000 | ||||||
| CEPH_FEATURE_REPLY_CREATE_INODE | 27 | 8000000 | S | S | S | S | S | |
| CEPH_FEATURE_OSD_HBMSGS | 28 | 10000000 | ||||||
| CEPH_FEATURE_MDSENC | 29 | 20000000 | ||||||
| CEPH_FEATURE_OSDHASHPSPOOL | 30 | 40000000 | S | S | S | S | S | |
| CEPH_FEATURE_MON_SINGLE_PAXOS | 31 | 80000000 | ||||||
| CEPH_FEATURE_OSD_SNAPMAPPER | 32 | 100000000 | ||||||
| CEPH_FEATURE_MON_SCRUB | 33 | 200000000 | ||||||
| CEPH_FEATURE_OSD_PACKED_RECOVERY | 34 | 400000000 | ||||||
| CEPH_FEATURE_OSD_CACHEPOOL | 35 | 800000000 | -S- | S | S | |||
| CEPH_FEATURE_CRUSH_V2 | 36 | 1000000000 | -S- | S | S | |||
| CEPH_FEATURE_EXPORT_PEER | 37 | 2000000000 | -S- | S | S | |||
| CEPH_FEATURE_OSD_ERASURE_CODES | 38 | 4000000000 | ||||||
| CEPH_FEATURE_OSD_TMAP2OMAP | 38* | 4000000000 | ||||||
| CEPH_FEATURE_OSDMAP_ENC | 39 | 8000000000 | -S- | S | ||||
| CEPH_FEATURE_MDS_INLINE_DATA | 40 | 10000000000 | ||||||
| CEPH_FEATURE_CRUSH_TUNABLES3 | 41 | 20000000000 | -S- | S | ||||
| CEPH_FEATURE_OSD_PRIMARY_AFFINITY | 41* | 20000000000 | -S- | S |
(Note that CEPH_FEATURE_OSD_ERASURE_CODES is no more required for client since firefly)