KVMing for fun and profit : how to make your toy become a serious business

Here I’ll try to present you some tips on how one could easily make a playground become a paid fun. And no, I won’t be teaching you how to become rich, although I would like if you would teach me how that could be accomplished as I’m trying to come up with a way to do just that for a couple of years now and failed miserably.

Recently, I’ve been setting up KVM virtual machines in order to do a lot of testing for a project I’m involved with. Letting boring details aside, one could say that a very long, boring and error prone setup was finished after a lot of work and it worked like a charm in the end, as a KVM machine.

As always, laziness is a given and when you are into it you empower yourself to come up with some nice hacks and goes that extra mile to find out how one could not to have to do boring and repetitive things again and again, in the best spirit of “let’s have some work now and save me ten times more work later”.

As I’m becoming more and more into that spirit of laziness, there I went looking for a way which would let me transform my beloved and nice working KVM machine into a fully featured real physical server. After some tips from microblogosphere friends (thanks fike), I had some ideas on how that could be accomplished.

Turned out my ideas weren’t really all that right so there I went again looking for a way to do what I wanted. After some research I found out that the KVM guys already gave some thought about it and even had a ready solution to my problem : it’s called qemu-nbd.

Yes, by now everyone should know that KVM builds on top of the excelent (just not so snappy) QEMU. However, as I’m a Debian user, here I should say that in Debian, most notably Debian sid/unstable, the qemu-nbd binary was renamed to kvm-nbd, just as the KVM binary is called only kvm and not qemu-system-x86_64 or whatever else is called these days upstream.

NBD (short for Network Block Device) is a nice thing. I hadn’t tried it before (I certainly used and am still using in some projects of mine the not-upstreamed-yet-but-so-nice-that-I-couldnt-resist DRBD, which seems to share some ideas which NBD). In short (read NBD link for more info on that), NBD allows the Linux kernel to use a remote server as a block device. Also, its upstream is a fellow Debian developer (hello Wouter). How nice’s it ?

As NBD is an upstreamed kernel feature, you don’t need to go to the external-module-route-hell. You only need the userland tools, which are nicely packaged and integrated into Debian as well. They’re only one aptitude away, so you could use this command to bring it to your KVM host :

aptitude install nbd-client

After that, you will want to export your image file (the file which represents the disk of you virtual machine) as a NBD block device, so you will be able to mount it under your KVM host and do whatever you want to do with it. Here’s how one would do it :

kmv-nbd –conect=/dev/nbd0 myimagefile.img

I tested it using both qcow2 and raw disk images and both worked like a charm so I don’t think you will have any problems here. Next, you need to create a directory structure under your KVM host which temporarily will be used to mount the partitions inside the virtual block device of your KVM machine. You could do this like this :

mkdir /newserver

mkdir /newserver/usr

mkdir /newserver/var

mkdir /newserver/home

In the example above, I’ve created these directories because I want my new server to have separated root, /usr, /var and /home partitions and mount points. Feel free to adapt to any kind of partitioning layout you please.

Next, you will mount yout virtual machine filesystem under a temporary location, so you will be able to copy all of its content later to your final desired destination, i.e. the real partitions inside the real disk you are going to use in your future real server (not a virtual machine anymore).

I did it using :

mkdir -p /mnt/temp

mount /dev/nbd0p1 /mnt/temp

Notice that the /dev/nb0p1 above is my root partition inside my KVM machine, only that it’s represented now in my KVM host in NBD’s device node notation here. I’ve only one partition inside this particular virtual machine (aside from the swap partition).

Next, you want to mount the partitions on the real disk you want to be used in your new real server under the directories created earlier so you will be able to chroot to then later and do any tweaks you may want to, like fixing /etc/fstab, /boot/grub/device.map, /boot/grub/menu.lst and all these places we know makes references to block devices which may differ from the situation you had in your virtual machine running under the KVM.

Here’s what I used :

mount /dev/sdb1 /newserver

mount /dev/sdb3 /newserver/usr

mount /dev/sdb5 /newserver/var

mount /dev/sdb6 /newserver/home

The device nodes above are from your real disk’s partitions, not from any virtualized device block access method like NBD. Also, take care not to mount your primary disk’s partitions (the ones you are using under in your KVM host) as you could destroy them easily. As you can see above, I’m using /dev/sdbX as /dev/sda is my primary disk, the one under which my KVM host is running. Again, adapt to your scenario.

Next, you just copy everything from your virtual machine filesystem (which is now accessible from within your NBD block device) to your real disk. I did it using :

cp -a /mnt/temp/* /newserver/

After that, you can disconnect your KVM virtual machine image file from the exported NDB device block using :

kvm-nbd -d /dev/ndb0

And then chroot to your real disk layout yout just copied to /newserver in order to fix anything which would make reference to block devices while running as a virtual machine under KVM but which now will be a completely different thing as a real server, like your /etc/fstab, /boot/grub/device.map, /boot/grub/menu.lst and so on.

In order to chroot to your new soon-to-be-real filesystem you could use :

chroot /newserver

Then go nuts and start fixing everything you may think needs to be fixed so this filesystem could be used in a real server to boot it up. Im my case, I needed to fix /etc/fstab in order to mount my additional partitions (as under KVM I was using only a root and a swap partition and now, under the real server, I’ll be using separated root, swap, /usr, /var and /home).

Before I went changing more things, I “fixed” my /boot/grub/device.map and my /boot/grub/menu.lst files to point to /dev/sdb and /dev/sdb[1],  respectively, so I could run grub-install from inside the chroot in order to install GRUB into the real new disk’s MBR (master boot record), using :

grub-install /dev/sdb

Then, finally, I “fixed/boot/grub/device.map and /boot/grub/menu.lst again so they would point to /dev/sda and /dev/sda[1], as /dev/sda will be the device node for this new disk when it wil be running under the new real server and not as a disk inside the KVM host machine.

Next, I exited the chroot using :


And umounted the new real disk’s partitions, using (now already out of the chroot) :

umount /newserver/home

umount /newserver/var

umount /newserver/usr

umount /newserver

Now you can take this new disk out of your KVM host machine, put it inside your new real server, physically install it there, turn on your new real server and everything installed while the filesystem in it was being used under the KVM virtual machine will be there, running nicely.

And you’re done. That’s it. Also, it’s important to point out that it’s much easier being done than being said (or written, in this case) so, if it looks scary because of the size of this post, don’t let it disincourage you as this whole thing is a pretty straightforward procedure, much easier than it seems.

And as I know that I will receive lost of complaints from people which will tell me how inefficient this whole thing is and how I could have accomplished the exact same thing in a much easier and faster way (perhaps even with a already existing tool which would automate almost entirely the whole proccess), I would say to these people to please exercise their right to share ideas using the comments. Just be nice on me, please.

After all, it was fun and I learned new things while doing it all, as well as I tried to share what I learned from this experience with my peers. That’s what matters. Peace, love and geekness to all of you 🙂


8 comentários sobre “KVMing for fun and profit : how to make your toy become a serious business

  1. aptitude install kpartx

    kpartx -a myimagefile.img

    You now have devices /dev/mapper/loop0p{1,2,…} for the partitions of your image file. A lot easier than going through nbd.

  2. @Anonymous

    Damm. I knew there should be an easier way. Worst yet, I actually used kpartx back then when I was in need of accessing data from a Xen domain hosted under one LVM’s LV.

    Well, living and learning, as they say. Anyway, thanks “Anonymous” for the tip. Surely, it will be useful in the near future.

  3. Hello,

    I would also like to point out that when creating the filesystem for your soon-to-be-new-real-server you should take care not to create the root filesystem (the one you’ll be mounting as your /) with an inode size greater than 128 bytes.

    Otherwise GRUB won’t be able to read your kernel/initrd and so it won’t be able to boot up your real machine later. You can force the creation of a filesystem with an inode size of 128 bytes by setting the inode_size parameter to 128 into /etc/mke2fs.conf.

    All of your others filesystems (/usr, /var, /home, etc) can be created using 256 bytes inode sizes, which is the default inode size used by mke2fs in my system (Debian unstable).


  4. kpartx only works with RAW disk images. qemu-nbd can work with any QEMU supported file format, but is awkward to setup and with various timing issues can be difficult to script.

    Instead take a look at qemu-diskp (http://www.saulsbury.org/software/virtualization.html)

    This uses FUSE to make files out of each of the partitions on a disk held in any QEMU supported disk format. These partition files can then be loop-back mounted or operated on by utilities like mke2fs etc.

Deixe um comentário

Preencha os seus dados abaixo ou clique em um ícone para log in:

Logotipo do WordPress.com

Você está comentando utilizando sua conta WordPress.com. Sair / Alterar )

Imagem do Twitter

Você está comentando utilizando sua conta Twitter. Sair / Alterar )

Foto do Facebook

Você está comentando utilizando sua conta Facebook. Sair / Alterar )

Foto do Google+

Você está comentando utilizando sua conta Google+. Sair / Alterar )

Conectando a %s