First glimpse at CoreOS
CoreOS is an emergent project that aims to address one of the most pressing questions in the server’s world. We at eNovance, therefore released eDeploy: a tool that performs bare metal deployment and manages upgrades with ease. Deploying and upgrading servers are currently two major concerns in the IT world since none (or at least very few people know how to do). I couldn’t resist trying CoreOS.
I. What is CoreOS?
I.1. Genesis ¶
The project is based on Google’s ChromeOS and is basically a fork. By bringing a new way of thinking, CoreOS relies on several things:
- Minimal OS: kernel + systemd
- Docker compatible: Built for containers
- Service discovery: Clustered by default with etcd a highly-available key value store for shared configuration
CoreOS provides a uniform basis for all applications since they sit inside containers powered by the excellent Docker. This provides a general and perfect isolation between all the applications. Then it becomes fairly easy to move applications from one CoreOS server to another. Additionally CoreOS root filesystem is read-only mounted which is great since this ensures a perfect consistency from one server to another.
I.2. Upgrades and security ¶
Upgrading CoreOS is a bit different than the usual distros. The update system is based on ChromeOS which does automatic upgrades. CoreOS uses two root partitions: one is active (root A), the other is passive (root B). Initially the system is booted into the root A partition. During this sequence, CoreOS starts talking to the update service to find out if updates are available. If so, they are downloaded and installed on root B. While installing these updates on root B, the update process is being rate limited via cgroups.
If you to learn more updates, read the blog post from CoreOS.
I.3. Service discover ETCD ¶
ETCD is a distributed configuration service The ultimate goal is to share and configure Docker containers all at once and atomically across all the CoreOS servers. Although nothing has been defined yet in this area.
ETCD is also responsible for service discovery. Services running inside Docker containers register themselves in ETCD so they can be used by the cluster for later use.
More about ETCD on the CoreOS blog.
II. Deep dive into CoreOS
II.1. The easy way: Vagrant ¶
Quite recently, with the help of Vagrant’s author, CoreOS team released a vagrant box. As always, this provides a simple way to try a new project.
1 2 3 4 5 6 7 8 9 10 | |
II.2. The minimal OS ¶
Note from the CoreOS website:
Linux kernel + systemd. That’s about it. CoreOS has just enough bits to run containers, but does not ship a package manager itself. In fact, the root partition is completely read-only, to guarantee consistency and make updates reliable.
Ok so what do we have here? I briefly looked into id and found in this lightweight (240MB for root) Gentoo 3.10.4+ VM. See Alex Polvi’s explanations on that:
We’re based on the ChromeOS SDKs… which use emerge to build the binaries required to assemble the distro. You can think of emerge/gentoo as the toolchain used to build all the binaries. We also pull base system packages from upstream portage, then compile them all together in out image.
Additional details:
NAME=CoreOS
ID=coreos
VERSION=32.0.0
VERSION_ID=32.0.0
BUILD_ID=32.0.0
PRETTY_NAME="CoreOS 32.0.0 (Official Build) dev-channel amd64-generic test"
ANSI_COLOR="1;32"
HOME_URL="http://www.coreos.com/"
II.2.1. Services running ¶
See below all the running services (excluding Kernel daemons):
USER PID STAT COMMAND
root 1 Ss /sbin/init
root 180 Ss /usr/lib/systemd/systemd-journald
root 201 Ss /usr/lib/systemd/systemd-udevd
root 256 Ss /sbin/dhcpcd -q --nobackground
201 259 Ss /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
root 265 Ssl /usr/sbin/update_engine -foreground -logtostderr -no_connection_manager
root 266 Ss /bin/bash /usr/sbin/update_engine_reboot_manager
root 278 S \_ dbus-monitor --system type=signal,interface='org.chromium.UpdateEngineInterface',member='StatusUpdate'
root 279 S \_ /bin/bash /usr/sbin/update_engine_reboot_manager
root 270 Ss /usr/lib/systemd/systemd-logind
root 273 Ss+ /sbin/agetty --noclear tty1 38400 linux
root 288 Ssl /usr/bin/docker -d -D
etcd 291 Ssl /usr/bin/etcd -v -d /var/lib/etcd
Interesting businesses are:
etcdprocesses running on port7001and4001dockerrunning on port4243- update_engine
II.2.2. Partitioning ¶
Number Start End Size File system Name Flags
1 32.8kB 134MB 134MB fat16 EFI-SYSTEM boot
2 134MB 201MB 67.1MB BOOT-B
3 201MB 1275MB 1074MB ext2 ROOT-A
4 1275MB 2349MB 1074MB ROOT-B
5 2349MB 2349MB 512B ROOT-C
6 2349MB 2483MB 134MB ext4 OEM
7 2483MB 2483MB 512B coreos-reserved
8 2483MB 2483MB 512B coreos-reserved
9 2483MB 5704MB 3221MB ext4 STATE
II.2.3. Mount points ¶
As mentioned earlier, CoreOS’s root partitions are mounted in READ ONLY. Yes yes, you can’t write to the OS.
/dev/sda3 on / type ext4 (ro,relatime)
Note: the fstab is empty since systemd mounts the partition during boot time. You can find every mount definitions in /usr/lib/systemd/system:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 | |
Note that the following mounts are binded from /media/state/overlays/:
/home/opt/srv/var/lock/var/run/var
II.2.4. Possible interesting stuff ¶
Vrak:
- /usr/sbin/write_gpt.sh
- /usr/lib/coreos/
- /usr/bin/coreos-c10n
II.3. Docker ready ¶
II.3.1. What is Docker? ¶
Description stolen from the Docker website.
Docker is an open-source engine that automates the deployment of any application as a lightweight, portable, self-sufficient container that will run virtually anywhere.
Docker containers can encapsulate any payload, and will run consistently on and between virtually any server. The same container that a developer builds and tests on a laptop will run at scale, in production*, on VMs, bare-metal servers, OpenStack clusters, public instances, or combinations of the above.
Common use cases for Docker include:
- Automating the packaging and deployment of applications
- Creation of lightweight, private PAAS environments
- Automated testing and continuous integration/deployment
- Deploying and scaling web apps, databases and backend services
Docker is built on top of LXC which offers namespaces capabilities such as PID, Network, IPS, mnt and UTS namespaces. It also uses control groups to manage resource accounting and limiting. All those things are provided by the Kernel itself.
Docker is growing really fast and it’s even about to go into OpenStack Havana as a new hypervisor.
II.3.2. Play with Docker! ¶
We are going to demonstrate the power of Docker with a simple memcached example.
By default, CoreOS doesn’t ship any images with Docker, so we have to pull some:
1 2 3 4 5 6 7 8 9 | |
Note that since Docker 0.6 you need root privileges to run the docker command. You revert your configuration with the “-H” flag, if you really want a normal to use the docker command.
By the way, if you get stuck with:
[debug] server.go:457 Updating checksums
[debug] server.go:463 Retrieving the tag list
[debug] server.go:466 Get https://cdn-registry-1.docker.io/v1/repositories/library/base/tags: x509: certificate has expired or is not yet valid
This means that the time of your machine is before the SSL cert was issued or after it expired. Thus you need to manually set the date:
1 2 | |
Start a container:
1 2 3 4 5 6 7 8 9 10 11 12 | |
The container has been stopped but still in the local registry:
1 2 3 4 5 6 | |
As you can see now the image has been added to my local registry:
1 2 3 4 5 6 7 | |
Let’s run it!
1 2 3 4 5 | |
Verify that the container properly runs and that it binds to the 49153 local port:
1 2 3 4 5 | |
You might wonder (as I did) how to interact inside a running container. Let’s say the container runs and I want to allocate more memory to memcached. Docker wasn’t meant to perform live modifications, instead it prefers versioning all the containers. Assuming the container is running in production, you might want to test what you’re about to change in a safe environment, basically a new container.
II.4. Service discovery: ETCD ¶
Description stolen from the GitHub project.
A highly-available key value store for shared configuration and service discovery. etcd is inspired by zookeeper and doozer, with a focus on:
- Simple: curl’able user facing API (HTTP+JSON)
- Secure: optional SSL client cert authentication
- Fast: benchmarked 1000s of writes/s per instance
- Reliable: Properly distributed using Raft
Etcd is written in Go and uses the raft consensus algorithm to manage a highly-available replicated log.
ETCD recently went to v0.1.0, see the official announcement.
I’m not going to copy/paste the really nice instructions from the GitHub project documentation. So I highly suggest you to have a look at the usage here. Also take a look at the client tool.
CoreOS is still Alpha but I found really surprising how far they alreay are with their OS. The community reactivity was excellent too. Let’s hope that this’ll continue like this!