Lately I’ve been delving more-and-more into how Openstack handles block storage. As part of that effort, I went through the process of setting up Cinder, Openstack’s block-storage API, wiring it up to Nova, and then using it to dynamically attach a volume to a running Nova instance.
This blog entry is just a quick summary of what I learned in the process with special attention paid to a few gotchas I ran into along the way.
What is Cinder?
Cinder at its heart is an API which exposes commands to create and destroy volumes within an Openstack installation. Originally this code lived in the Nova compute service as
nova-volume, but was extracted to become its own component, complete with a separate API, language bindings and its own command-line tool (
All this is relatively straightforward, but I think two points are worth mentioning:
First, Cinder is really just an API that happens to ship with a backend you can use out of the box. This means you could use Cinder itself for storage (using its LVM and ISCSI drivers) or you could write a Cinder compliant storage-service to suit your own needs. (If you’re already familiar with Openstack, you’ll recognize this same plugin pattern used across most projects.)
Second, Cinder does not understand compute, but compute understands Cinder. This means areas where volumes and compute intersect, like in the attaching or detaching of volumes to an instance, are the responsibility of Nova: meaning, to use Cinder with Nova, you will have to use both
cinderclient to create the volume, and then
novaclient to attach it.
With this in mind, let’s install and actually use Cinder, specifically:
Install and configure Cinder using its LVM/ISCI backend
Provision a 1 GB volume using
Create a Nova instance and attach the volume using
Log into the instance and setup the volume to be usable space.
Installation and Configuration
For this step, we’re going to be installing Cinder from the perspective of a developer, meaning we’re going to fetch source code and run
cinder directly out of the git repository.
Clone the repo from the Openstack repo on Github:
$ git clone git://github.com/openstack/cinder.git
Copy the base configuration into
$ cp -R etc/cinder /etc/ $ mv /etc/cinder/cinder.conf.sample /etc/cinder.conf
Configure Cinder to use MySQL by adding this line to
Create database and load schema:
$ mysqladmin -uroot create cinder $ ./bin/cinder-manage db sync
Configure LVM to manage the underlying block-storage. In my case, I’m using a second partition but you could also use a loopback device as well:
$ apt-get install lvm2 $ vgcreate cinder-volumes /dev/xvda2 $ pvcreate /dev/xvda2
Install tgt which will expose the block-storage over ISCSI:
$ apt-get install tgt
Configure tgt by editing /etc/tgt/targets.conf to add this line, modifying it to match the
volumesdirectory in your
tgtdaemon. This may display some errors on startup. Those can be ignored for now:
Start the cinder API and associated services:
Cinder now should be up and running but you still need a way to query and manage it. To do this, we’ll need to install
Clone python-cinderclient from the Openstack GitHub repo:
$ git clone git://github.com/openstack/python-cinderclient.git
Install it in developer-mode, this will symlink installed Python package back to your git repo:
$ python setup.py develop
Like other Openstack command-line tools,
cinderclientuses environment variables for configuration, so you should create a config file and then source it, like:
$ cat cinder.env export OS_AUTH_URL=http://127.0.0.1:8776/v1/ export OS_USERNAME=<YOUR_USERNAME> export OS_PASSWORD=<YOUR_PASSWORD> export OS_TENANT_NAME=openstack $ . cinder.env
Now you can test that it works by running:
$ cinder list # <- should be emty since don't have any volumes yet
Configuring Nova to talk to Cinder
In order to attach volumes created with Cinder, Nova needs to know where the Cinder endpoint resides. To keep it simple, I’m not using Keystone (Openstack’s Identity Service), and instead hard-coding the endpoint with the
Add the following to your
Restart nova-api and nova-compute
Provision a 1 GB Volume
With Cinder, Cinderclient and Nova configured, we can now create, attach and use volumes.
Create a 1GB volume
$ cinder create --display_name cindervol 1
Check to see that it exists:
$ cinder list +--------------------------------------+-----------+-----------------+------+-------------+----------+-------------+ | ID | Status | Display Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+-----------------+------+-------------+----------+-------------+ | da6ae608-4673-4b24-acd4-75e527b5969a | available | cindervol | 1 | None | false | | +--------------------------------------+-----------+-----------------+------+-------------+----------+-------------+
Attach Volume to Instance
Create the instance using whatever image and flavor combo you want:
$ nova create --image <YOUR IMAGE> --flavor 1 myinstance
Attach the volume to the running instance.
You’ll need to specify a device identifier within your instance that isn’t being used. In my case, I know that
/dev/xvdcis being used for swap but that
/dev/xvdbis free, so I’ll use that:
$ nova volume-attach myinstance da6ae608-4673-4b24-acd4-75e527b5969a /dev/xvdb
Use the Attached Volume
Now that the volume has been attached, you need to perform some setup within the instance so that you can actually use it.
First, login to your instance
$ ssh myinstance
Next, ensure that the device for new volume is present:
$ ls /dev/xvdb /dev/xvdb
Assuming it’s present, you can now format it:
$ mkfs /dev/xvdb
Mount the newly formatted disk
$ mkdir /cindervol $ mount /dev/xvdb /cindervol
Finally, verify that the newly mounted volume has the correct size.
$ df -h Filesystem Size Used Avail Use% Mounted on /dev/xvda1 12G 738M 11G 7% / tmpfs 28M 0 28M 0% /lib/init/rw udev 15M 44K 15M 1% /dev tmpfs 28M 0 28M 0% /dev/shm /dev/xvdb 1008M 1.3M 956M 1% /cindervol
We can see from the output, the Cinder-created volume has the correct size so the process has worked end-to-end.
The minimal Cinder setup we just created is great for learning the code and see how things fit together but, as you become more familiar with it, you’re probably going to want to expand to a more complex setup to take advantage of advanced features, for example Keystone integration. Your two best bets here are diving into the code itself (daunting at first, but print/raise statements are your friend) and checking out the online documentation.Posted on 14 January 2013.