Unifying Access to Virtual Disk Images – The EaaS Way

Tuesday, March 01st, 2016 | Author:

With the multitude of emulators EaaS currently supports – what is the best/optimal/ideal/future-proof file format for disk images? Until now, we had a simple answer to that: The best option is the format your desired emulator supports. If you want to use Qemu, you have a very wide range of choices available. Since most emulators support RAW disk images, should that be the preferred choice in general? Unfortunately, if you want to use VirtualBox, you cannot use RAW images, your image needs to be in VDI or VMDK format. You can still use RAW images for archiving purposes and convert images on demand from RAW to VDI when the environment is started, for instance using qemu-img. This, however, implies copying (migrating) the RAW file, and in the case of a typical Windows XP images this means copying about 20GB of data. This not only is inefficient in terms of disk and bandwidth use (most of the times, the environment will not need all of the 20GB), it also impairs the user experience as we have to wait for the image migration to finish before starting the environment. A further set of problematic images, are forensically packaged images in EnCase (EWF) or AFF format. For the preservation community, these image formats are of interest because they embed fixity information as part of the disk image itself. In order to start such an image with VirtualBox, we first have to extract the RAW image and convert it to VDI in a second step.

As emulation is what we are good at, it would of course be nice to just to emulate all the different image file formats for our emulators and have simple storage file format such as RAW or EWF.

xmount is a FUSE driver that emulates different disk image formats (VDI/VHD, VMDK and, of course, RAW) without the need for a complete copy. Hence, one can FUSE-mount a RAW image file as VDI using xmount. The resulting VDI file, however, is only virtual. All conversions required to turn the RAW file into a VDI are made on-the-fly. Unfortunately, the input formats supported by xmount are limited as it only supports EWF, AFF and RAW images as input. Additionally, all data written by the emulator to the virtual VDI file are stored separately in a proprietary block-diff file which was initially designed to be discarded once the image is unmounted again. Nevertheless, xmount is a good start but not yet sufficient for the purposes of EaaS.

The EaaS image archive exports images read-only. As a first step, a writeable layer is required to make a disk image usable with an emulator. To this purpose, a new empty qcow2 disk image is created, linking to a backing file from our image archive. Any data written by the emulator is stored in the qcow2 image, any modified data is then also read from the qcow, but unmodified data is read directly from the image archive.

The following example creates an empty writeback image in the qcow2 format:
qemu-img -f qcow2 -o backing_file=http://archive/object.raw writeback.cow

To support this workflow, xmount required some extensions. To support qcow2 and all its features, for instance using a HTTP URL as backing file, we have added the Qemu block driver to xmount as an input library. Through this, support of input formats of xmount has be widely improved, allowing us to directly mount all image file types supported by Qemu to be exported either as RAW, VDI/VHD or VMDK.

Because we want changes to be stored back into our qcow2 image to create derivatives of an environment (e.g. by installing additional software to a bare operating system image), we also added a write through option to xmount. This causes write operations not to end in xmount’s own proprietary file but to go back through Qemu’s block driver. In case of a qcow2 image as input, this means changes are written to this same image, preserving the link between the modified environment and its backing file.

The following command:

mkdir mountDir
xmount --in qemu writeback.cow --out vdi --cache writethrough --inopts qemuwritable=true mountDir

first creates a target directory (a mountpoint) and then mounts the writeback.cow image to mountDir. In this directory a file called object.raw.vdi can be found and can be used as a virtual disk by VirtualBox, e.g. reading data from the file http://archive/object.raw. Any data written is stored in the writeback image.

Source code at Github: https://github.com/eaas-framework/xmount

There are also pre-built debs for Ubuntu 14.04 available. First add our repository to your sources list:

echo "deb http://packages.bw-fla.uni-freiburg.de/ trusty bwfla" > /etc/apt/sources.list.d/bwfla.list

Prioritize the installation of our packages

echo -e "Package: *\nPin: release c=bwfla\nPin-Priority: 1001" > /etc/apt/preferences.d/pin-bwfla.pref

And finally update the packages list and install xmount and its dependencies

apt-get update
apt-get install qemu-utils qemu-block-extra libxmount-input-qemu xmount libcurl3 libcurl3-gnutls

Tags »

Trackback: Trackback-URL | Comments Feed: RSS 2.0
Category: R&D

Responses are currently closed, but you can
Trackback the post from your own site.