EaaS Ubuntu Packages (14.04 LTS) BETA

Tuesday, June 07th, 2016 | Author:

Installing EaaS has always been a difficult task, since many different software components need to installed and configured. To ease this process we have created Ubuntu packages (currently 14.04 LTS only, 16.04 is on the way).

To retrieve our  packages, you need to add our repository to your APT-sources list. Create a file /etc/apt/sources.list.d/bwfla.list containing the following line:

deb http://packages.bw-fla.uni-freiburg.de/ trusty bwfla

Furthermore, you need to prioritize our packages, as we provide patched versions of some system packages. Add a file /etc/apt/preferences.d/pin-bwfla.pref with the following content:

Package: *
Pin: release c=bwfla
Pin-Priority: 1001

Finally update your packages database (apt-get update).


The EaaS system consists of different components:

  1. eaas-common (dependency of all packages, will be installed automatically)
  2. eaas-emulators
  3. eaas-gateway
  4. eaas-image-archive
  5. eaas-fits
  6. eaas-object-archive
  7. eaas-software-archive
  8. eaas-server
  9. eaas-workflows

All components can either be installed on a single machine, or deployed individually.

Note: If install or re-configure an EaaS package while the application server is already running, make sure to restart the server to enable new features or settings (service bwfla restart).

EaaS Common
The eaas-common module installs common software dependencies (e.g. xmount, shell scripts) and prepares the system for running EaaS. This package is a dependency of all other EaaS modules and does not need to be installed manually.


Say “yes” if you want to follow to use the installation wizard to setup your EaaS instance, or say no if you want to configure your instance manually.


Set IP address and port number the EaaS server should listen to.

EaaS Emulators
The eaas-emulators package is preparing the system to run emulators and act as an EmuComp service. This package also installs all supported emulators.


To make the system scaleable, the EmuComp service needs to be accessible for any client using emulators. Please provide the IP/port or FQDN how a client can access this service. Usually, this is the same IP/port as entered for eaas-common. Sometimes, however, the service is masked / proxied / etc. and is reachable through a different FQDN / port.


EaaS Gateway
The eaas-gateway takes request from clients and dispatches these requests to EmuComp instances.


Add a list of configured EmuComp as instances separated by spaces, e.g. “,4,8” — meaning that the instance running at port 8080 provides 4 CPU cores and instance running at port 8181 provides 8 CPU cores.



EaaS Image Archive
The eaas-image-archive package installs a basic image archive service.


You need to provide a path to the image archive (currently file-system based).


If the path exists, it is assumed that there is a valid (file-base) image archive. If the path does not exist, a new file-based image-archive will be instantiated and pre-populated with simple example images.


Finally, one needs to decide how the image data is delivered to the emulators. Currently, two options are available: the network block device protocol (NBD) and through HTTP. NBD is usually faster and more bandwidth efficient than HTTP, but requires that EmuComp instance are able to connect to the image-archive through port 10809. Using HTTP requires only that port 80 is accessible for EmuComp instances.

EaaS Object-Archive
The eaas-object-archive package installs a basic object archive facade.


The object archive facade is capable to manage multiple object archives. At this point only the directory has to be set where the individual object-archive configurations can be found.


EaaS Software-Archive
The eaas-software-archive currently provides a thin meta-data layer on top of an object-archive. We will update information on using the software-archive soon. Currently WIP.



EaaS Server
The eaas-server package is also a dependency package, installed by one of the functional modules. The server package installs and configures the EaaS application server. Optionally an upstart service will be installed, such that the EaaS services are started when the machine is booted.


Note: the eaas-server package does not start the server during or after installation. You need to either to reboot the machine or start the server manually (service start bwfla).

EaaS Workflows
The eaas-workflows package installs and configures a web-based UI implementing example workflows.



Configure the EaaS Gateway to be used. Provide IP/FQDN and port to a gateway instance. If the current instance has been configured to act as EaaS Gateway, the values are preset to its current configuration.


Configure the EaaS Image Archive to be used. Provide IP/FQDN and port to a gateway instance. If the current instance has been configured to act as EaaS Image Archive, the values are preset to its current configuration.



Configure the EaaS Object Archive to be used. Provide IP/FQDN and port to a gateway instance. If the current instance has been configured to act as EaaS Object Archive, the values are preset to its current configuration.


Category: R&D | Leave a Comment

EaaS Object Archive

Thursday, March 31st, 2016 | Author:

The EaaS object archive (or technically the EaaS object archive facade) is one key component to support seamless integration of institutional object-repositories with EaaS and to make EaaS a cost-effective and scalable access solution for born-digital content.

By design, EaaS separates an emulated computer system (emulation environment) and user objects. The emulation environment consists of the emulator configuration in combination with a bootable disk image containing an installed operating system with software applications etc. The user objects, on the other hand, are all media objects such as CD-ROMs or individual files that are to be rendered in the emulated computer system. Objects and emulation environment are combined only if a user requests a certain object to be rendered using a suitable emulation environment.

The main rationale for this design are preservation planning complexity and costs: as the number of objects are in the thousands or millions, the number of necessary emulation environments is rather small, typically about 10-20 “base” environments representing typical computer systems of technological epochs. By separating objects from their rendering environments, emulation-based preservation planning strategies can focus on a small set of (emulation environments) that have to be kept alive. The objects remain in a dedicated repository.

To make objects accessible with EaaS we have designed a flexible “object archive facade” to translate between the technical requirements of the emulation framework and a specific object repository.

Interfaces and Data Types

To allow EaaS to retrieve and render user objects an object repository provider has to implement a simple Java interface:

public interface DigitalObjectArchive
    public FileCollection getObjectReference(String objectId);

    // object archive identification 
    String getName();

    // optional, returns a list of object IDs
    public List<String> getObjectList(); 

The most important method to be implemented is getObjectReference()
which takes an object ID and returns a FileCollection description of the object. The FileCollection represents all individual files of an object as referenceable URLs. As a JAXB XML representation looks like:

<FileCollection id="OID1">
       type="CDROM" />

Each FileCollectionEntry represents a media image (currently only floppy, disk and cdrom are supported) as a URL to the data stream. The URL may contain https(s), nbd or file transport protocols and ideally support random access to the data stream (i.e. for HTTP(S), the server has to support HTTP Range Requests). The URLs provided by the FileCollection need to be directly accessible by the EaaS infrastructure. Hence, a repository-specific implementation of getObjectReference() provides accessible links to the internal data streams for a given object, or retrieves the object to a temporary storage and creates filesystem links (in the case of direct file access) to these files.

Category: R&D | Leave a Comment

Curate Gear 2016

Thursday, January 21st, 2016 | Author:

Slides of the this year’s Curate Gear demo are online.

The demo script can be downloaded from our GitHub page.

Category: Events, R&D | Leave a Comment

Boot to Emulation – EaaS as a Local Option (Beta)

Wednesday, October 21st, 2015 | Author:

Complementary to the release of the EaaS Docker containers, we’ve created a self-contained USB live system. The USB live-system boots a computer directly and runs emulated environments on local hardware. Running emulators on local machines (e.g. standard PCs) can be an interesting alternative for reading-room setups or museum displays, where cluster- or cloud-computing options are not suitable. Local execution of emulators allows to connect peripherals, such as joystick, printers, CRT monitors, but also supports an improved user experience for some applications (e.g. games, software based art, etc.) by providing native fullscreen and reduced (input-)latency.

The current live-system offers three different options:
  • A complete self-contained system
  • A self-contained system, which integrates with an existing EaaS setup
  • A boot-to-emulator system, suitable for public displays etc, which directly boots into a preconfigured emulation environment
System requirements
  • at least 2 Gb (4 Gb recommended) of RAM
  • boot option from USB (USB 3.0 recommended)
  • a USB pendrive/stick, at least 8 Gb
  • optional a cable connected network card
The Self-contained EMiL Live-System
First, download the USB image here (http://bw-fla.uni-freiburg.de/usb-demo.img) and write it to an USB pendrive. We recommend to use a fast USB 3.0 stick, with at least 8 GB capacity.

To write the image to the USB drive we recommend Linux and MacOSX users to use „dd“. E.g.

sudo dd if=/home/klaus/usb-demo.img of=/dev/<your usb device>
Windows users my use a tool like the win32 disk image writer (http://sourceforge.net/projects/win32diskimager/) or similar tools.

For a fully self-contained setup, just boot directly from the USB stick. We have preloaded the stick with some simple examples for demo purposes. A short cheat-sheet:
  •         stop an emulator with CRTL-ALT-ESC
  •         toggle between fullscreen and web view CRTL-ALT-F
In the non-fullscreen mode, the user may have options to cite an environment, create a screenshot, change a medium, etc…
Add your own images
If you want to add your own images, you can mount the USB stick on your desktop computer and you’ll see two partitions. The first partition contains a read-only ubuntu based live-distribution, the second partition, called „emil-data“ contains two folders:
  • configs/  contains user-writeable configuration files
  • image-archive/ contains a valid image-archive structure with some examples
You can copy your disk images to the image-archive/images/base folder and create meta-data accordingly. We will write a follow-up article on creating appropriate meta-data.
Currently, the second partition is rather small, but can be resized. Write the USB image to a large pen-drive, delete the second partition and re-creating it. Make sure that you retain the configs and image-archive directory and set the label of the second partition to „emil-data“. Any filesystem supported by Linux should be OK. We’ve chosen the proprietary filesystem exFAT (https://en.wikipedia.org/wiki/ExFAT) to support virtual disk images larger than 4Gb and be compatible with all major desktop operating systems.
Integrate with an EaaS Environment
Furthermore, the USB live-system integrates well with an existing EaaS environment, e.g. for hybrid (local / web) usage or as described in the following example a curation and efficient deployment tool in a reading-room environment.
To maintain your emulated environment centrally, in a first step you need to setup EaaS workflows and image-archive. Ideally you should start using our pre-packaged Docker containers (see also: http://bw-fla.uni-freiburg.de/wordpress/?p=817:
cd image-archive/nbd-export/ 
ln -s ../images/base/doom.raw 
ln -s ../images/base/hatari_tOS206us.img 
ln -s ../images/base/qemu-i386-DOS_6.20_CDROM.raw

Finally run:

./run-full-setup.sh --public-ip-port --docker  eaas/bwfla:demo-august-15 --archive-dir /Users/klaus/Downloads/image-archive

with a valid IP for your machine and archive-dir pointing to your image-archive.

Now you have can use the bwFLA workflows via web browser (e.g. open but you also have a public image-archive running, serving the exported environments. To use this archive from the USB stick:
  • delete the image-archive folder from the second partition
  • edit configs/remote/WorkflowsConf.xml: set the <archiveGW> value to the IP and port of your docker instance. Make sure that the machine booting from USB has a cable network connection and the network is configured via DHCP. Also make sure that the USB machine is able to reach your Docker instance.

Boot to Emulator
Finally, the USB stick can be used for booting directly into a specific environment. For this simply put a file named “environment-id.txt” into the top-level directory of the second (“emil-data”) partition. The file should contain only the ID of the environment to load. You can find the ID of  an environment in its meta-data.
Note: this version is not tamper-proof. It is not recommended to use it for public displays. If you need a tamper-proof version please contact us.

The next steps ™
  • Improve usability and workflows
  • The current version is static in particular w.r.t. emulator curation. The next version will support centrally maintained, containerized components, in particular emulators. When the system starts it will check for updated software packages and will download new components if required.
  • Update of available workflows
  • Deployment for reading-rooms via PXE

Category: bwFLA Projekt, DP Projekte, R&D | Leave a Comment

Imaging (old) IDE disks – Harder than imagined

Wednesday, September 16th, 2015 | Author:

Despite the formerly wide spread use of (parallel) port IDE disks in x86 computers, there seem to be a couple of compatibility issues with these devices. While e.g. the 40pin the physical interface did not change, the logical layer did.


The system to be preserved was an integral part of a large scale local language research program. It was set up in 1993 as a test case for computerized local language research and language/dialect mapping. The setup consisted of one server machine running on OS/2 driving an IBM DB2 database and six client machines offering access to the database over a LAN. The LAN infrastructure was running TCP/IPv4 on TokenRing infrastructure. All machines were x86 hardware, featuring 486DX2 for the clients and a Pentium/Overdrive CPU (upgraded from 486) in the server. The machines were equipped with 8 to 16MByte of RAM. The server hard disk was a SCSI disk of 1,1GByte of size, the client hard disks were IDE of 240MByte of size. All clients were identically equipped and the installation on the clients were originally identical.

Five identical IDE disk, manufactured 1993

Experimental Setup

Parallel port IDE got gradually phased out and has been replaced by SATA in the mid 2000. Thus, this kind of connector typically unavailable in new x86 machines. In the experiments a couple of different IDE implementations where used:

  • Intel 865 chipset with a BIOS from 2005
  • Intel 875 chipset with a BIOS from 2004
  • Two port onboard controller (same mainboard with Intel 875)
  • NVidia nForce chipset of 2005
  • Lindy cable multi (physical) IDE port to USB 2.0 adaptor
  • Davicontrol PCI dual-port IDE adaptor (Silicon Image PCI0680 Ultra ATA 166)
The IDE disks mainly considered where 240 MByte Quantum disks. The disks, taken from five client machines, where numbered 1 … 5.  For the system imaging procedure a fairly recent 3.2 Linux kernel (Ubuntu 12.04 LTS) was used. A preliminary requirement for disk imaging is that the system is recognizing the disk properly:
  • The USB adaptor “saw” the disk, but was unable to produce a proper capacity reading (guessed on 2 TByte)
    [  300.633955] scsi 6:0:0:0: Direct-Access     QUANTUM  19234688         0    PQ: 0 ANSI: 2 CCS
    [  300.634368] sd 6:0:0:0: Attached scsi generic sg2 type 0
    [  300.634944] sd 6:0:0:0: [sdc] Very big device. Trying to use READ CAPACITY(16).
    [  300.635821] sd 6:0:0:0: [sdc] Using 0xffffffff as device size
    [ 300.635839] sd 6:0:0:0: [sdc] 4294967296 512-byte logical blocks: (2.19 TB/2.00 TiB)
  • Newer IDE adaptors, like e.g. the onboard controller did not recognize the disk at all
  • Several disks got recognized by the Intel 865, 875 controllers, but two failed (disk #2, #4)
  • The failed disk 4 got properly recognized on the nForce and Davicontrol controller

The disks did not get properly recognized in every boot-up cycle. They need a certain time to spin up to answer properly. Sometimes they “hang”, which is indicated by a permanently on disk activity LED. To see, if the disks got recognized by the operating system (Linux), the kernel messages give the information on which disks are visible to the system. Later on, run with administrator privileges, the fdisk command should give a proper listing of the disk’s partition table.

Some tips to deal with “hanging” disks:

  • Most of the disks did not get detected with every bootup. Pausing the machine, rebooting usually helps to get it finally done.
  • If a disk gets not properly recognized during bootup, then the unloading and loading of the IDE controller kernel module triggers the recognition. This usually helps.
  • The detection rate on the Davicontrol adaptor was different in different machines. The BIOS and bus device order (initialization in different order) seems to influence the process significantly

The tool of choice to produce identical copies of block devices in Linux/Unix systems is dd (or ddrescue). In standard configuration it reads the block device 512 Byte wise and writes this to a file (if asked to). After proper recognition the disk was present through the highlevel device, e.g. _/dev/sda_ and through the device for each partition e.g. _/dev/sda1…5_ (numbering depending on the partitions detected).

Every disk aside disk 3 got read from beginning to end with dd if=/dev/sda of=image-file. This procedure copies every thing including the master boot record as well as the partition table. dd finished the process without any errors in every run on every machine. The machine log did not show any errors either. Thus, it was concluded, that the process ran flawlessly. Unfortunately the simple partition check on the image file fdisk -l image-file did not produce the proper partition listing. This was assumed to be a deficiency of the tool. After trying to boot the resulted system image in emulator random filesystem errors occurred.

Investigating this issues further showed that each dd run produced a random md5sum. Unfortunately no errors was reported from the system or dd. We cross-checked the results with hexdump, which showed different content for each run. However, we could ruled out faulty HDD drives because the original hardware worked well with the original disks. The only source of errors could within the imaging process.

To rule out a faulty version of the used Linux kernel, the experiments were repeated with similar results on a three year older kernel with similar results. To check for implementation flaws in the IDE driver even older versions were booted, but then the hardware did not get fully recognized and a disk dump was impossible.

After getting repeatedly different results from reading the disks with the i865 chipset (default) other IDE controller where used. The AMD/nForce system from about the same era as the i865 system behaved pretty much the same, but was able to read disk #4 which was un-accessible on the i865 (reason not totally clear: We might have not tried hard enough to register the disk to the machine by restarting, delaying bootup or reloading the required kernel modules). After a couple of runs, fdisk produced different results for the same disk, same for the dd runs. The produced images also differed from the images produced on the i865 system.

The Davicontrol adaptor was a recent addition to our hardware pool and was used next. Up to now it never failed to produce a proper partition table reading of different disks. The produced images were exactly the same in different runs (using md5sum). The partition listing of the resulted image files looked as expected (identical to the reading from the original source). Same was true for mounting the partitions (none of the previous filesystem errors encountered from the other images). Thus, it should be pretty obvious that the setup is producing the expected results and has none of the hardware issues of the setups used before. Nevertheless, there might be still hidden errors around.


dd does exactly the job as expected, imaging block devices to files. If the input is corrupted for some reason (but not reported as an error by the operating system), dd has no means to detect such an issue. The verification process of the system imaging is to be done thoroughly using different tools and methods to ensure correctness.

Leassons learned: 

  1. As long as the md5sum (or similar fixity information) is not gathered from a different source, it is (almost) useless to compare the disk’s md5sum with the imaging result
  2. Do at least two consecutive imaging runs and verify that the results are identical

The encountered errors highlight also once again the importance of a hardware archive offering various options for the same task. Adaptors, controllers and system buses get out of use and result in device obsolescence which might prevent the proper preservation of a system.


by Dirk von Suchodoletz & Klaus Rechert

Category: R&D | Leave a Comment

bwFLA EaaS: Releasing Digital Art into the Wild

Saturday, December 21st, 2013 | Author:

With the bwFLA Emulation-as-a-Service you can enable users to view your (interactive) objects without actually giving the environment+object to the user. This is a nice feature, especially for dig. art and similar: you can provide access to an almost unlimited amount of people being able to view, use and interact with a piece of dig. art without being able to copy it. The owner remains in control of the object and is able to restrict access any time.

Jon Thomson and Alison Craighead (www.thomson-craighead.net) nicely prepared and integrated two of their art pieces for public access using the bwFLA EaaS infrastructure. Please take a look at:




Please be patient it may take a minute or so to load. More information can be found here http://thomson-craighead.blogspot.de/2013/12/thalamus-now-emulated-online.html

Category: bwFLA Projekt, R&D | Leave a Comment

bwFLA Demo

Monday, August 05th, 2013 | Author:

Das bwFLA Projekt hat das Ziel Emulation als Werkzeug für den Langzeitzugriff auf verschiedenartigste (komplexe) digitale Objekte und im Forschungsdatenmanagement nutzbar zu machen. Hierfür wurden Use-Cases u.a. aus dem Bereich Kunst und Wissenschaft identifiziert, entsprechende Workflows entwickelt und beispielhaft umgesetzt. Mit der Fertigstellung der Ingest– und Access– Workflows wurde ein wesentlicher Meilenstein des bwFLA Projekts erreicht und soll nun einer breiteren Nutzergruppe vorgestellt werden. more…

Category: bwFLA Projekt | Leave a Comment

And Now for Something Completely Different

Friday, August 02nd, 2013 | Author:

One Terabyte of Kilobyte Age

by Olia Lialina and Dragan Espenschied


More information on this project can be found:




Category: DP Projekte, R&D | Leave a Comment

Tech-Demo: UFC Migration through Emulation

Tuesday, February 21st, 2012 | Author:

Der UFCMigrate Web Service erlaubt das automatisierte Durchführen von Format-Migrationen mittels Emulation. UFCMigrate wurde im Rahmen des PLANETS Projekts konzipiert und wird seit dem konsequent weiterentwickelt.

Weitere Informationen sind auf den UFCMigrate Webseiten zu finden.

Category: OPF, R&D | Leave a Comment

bwFLA@Joining Forces am 1. Februar in Berlin (Nachtrag)

Wednesday, January 18th, 2012 | Author:

Am 1. Februar 2012 findet fand ein internationaler Expertenworkshop “Joing Forces” zur digitalen Bewahrung am Computerspielemuseum in Berlin statt. Organisiert wird die Veranstaltung neben der einladenden Institution durch die nestor AG Emulation, den AK Langzeitarchivierung und die Gruppe Emulation der deutschen Gesellschaft für Informatik. Neben dem KEEP Project sind ebenfalls bwFLA – Funktionale Langzeitarchivierung und Digitising Contemporary Art (DCA) vertreten.


Category: bwFLA Projekt, Events | Comments off