Boot to Emulation – EaaS as a Local Option (Beta)

Wednesday, October 21st, 2015 | Author:

Complementary to the release of the EaaS Docker containers, we’ve created a self-contained USB live system. The USB live-system boots a computer directly and runs emulated environments on local hardware. Running emulators on local machines (e.g. standard PCs) can be an interesting alternative for reading-room setups or museum displays, where cluster- or cloud-computing options are not suitable. Local execution of emulators allows to connect peripherals, such as joystick, printers, CRT monitors, but also supports an improved user experience for some applications (e.g. games, software based art, etc.) by providing native fullscreen and reduced (input-)latency.

The current live-system offers three different options:
  • A complete self-contained system
  • A self-contained system, which integrates with an existing EaaS setup
  • A boot-to-emulator system, suitable for public displays etc, which directly boots into a preconfigured emulation environment
System requirements
  • at least 2 Gb (4 Gb recommended) of RAM
  • boot option from USB (USB 3.0 recommended)
  • a USB pendrive/stick, at least 8 Gb
  • optional a cable connected network card
The Self-contained EMiL Live-System
First, download the USB image here (http://bw-fla.uni-freiburg.de/usb-demo.img) and write it to an USB pendrive. We recommend to use a fast USB 3.0 stick, with at least 8 GB capacity.

To write the image to the USB drive we recommend Linux and MacOSX users to use „dd“. E.g.

sudo dd if=/home/klaus/usb-demo.img of=/dev/<your usb device>
Windows users my use a tool like the win32 disk image writer (http://sourceforge.net/projects/win32diskimager/) or similar tools.

For a fully self-contained setup, just boot directly from the USB stick. We have preloaded the stick with some simple examples for demo purposes. A short cheat-sheet:
  •         stop an emulator with CRTL-ALT-ESC
  •         toggle between fullscreen and web view CRTL-ALT-F
In the non-fullscreen mode, the user may have options to cite an environment, create a screenshot, change a medium, etc…
Add your own images
If you want to add your own images, you can mount the USB stick on your desktop computer and you’ll see two partitions. The first partition contains a read-only ubuntu based live-distribution, the second partition, called „emil-data“ contains two folders:
  • configs/  contains user-writeable configuration files
  • image-archive/ contains a valid image-archive structure with some examples
You can copy your disk images to the image-archive/images/base folder and create meta-data accordingly. We will write a follow-up article on creating appropriate meta-data.
Currently, the second partition is rather small, but can be resized. Write the USB image to a large pen-drive, delete the second partition and re-creating it. Make sure that you retain the configs and image-archive directory and set the label of the second partition to „emil-data“. Any filesystem supported by Linux should be OK. We’ve chosen the proprietary filesystem exFAT (https://en.wikipedia.org/wiki/ExFAT) to support virtual disk images larger than 4Gb and be compatible with all major desktop operating systems.
Integrate with an EaaS Environment
Furthermore, the USB live-system integrates well with an existing EaaS environment, e.g. for hybrid (local / web) usage or as described in the following example a curation and efficient deployment tool in a reading-room environment.
To maintain your emulated environment centrally, in a first step you need to setup EaaS workflows and image-archive. Ideally you should start using our pre-packaged Docker containers (see also: http://bw-fla.uni-freiburg.de/wordpress/?p=817:
cd image-archive/nbd-export/ 
ln -s ../images/base/doom.raw 
ln -s ../images/base/hatari_tOS206us.img 
ln -s ../images/base/qemu-i386-DOS_6.20_CDROM.raw

Finally run:

./run-full-setup.sh --public-ip-port 192.168.99.100:8080 --docker  eaas/bwfla:demo-august-15 --archive-dir /Users/klaus/Downloads/image-archive

with a valid IP for your machine and archive-dir pointing to your image-archive.

Now you have can use the bwFLA workflows via web browser (e.g. open http://192.168.99.100:8080) but you also have a public image-archive running, serving the exported environments. To use this archive from the USB stick:
  • delete the image-archive folder from the second partition
  • edit configs/remote/WorkflowsConf.xml: set the <archiveGW> value to the IP and port of your docker instance. Make sure that the machine booting from USB has a cable network connection and the network is configured via DHCP. Also make sure that the USB machine is able to reach your Docker instance.

Boot to Emulator
Finally, the USB stick can be used for booting directly into a specific environment. For this simply put a file named “environment-id.txt” into the top-level directory of the second (“emil-data”) partition. The file should contain only the ID of the environment to load. You can find the ID of  an environment in its meta-data.
Note: this version is not tamper-proof. It is not recommended to use it for public displays. If you need a tamper-proof version please contact us.

The next steps ™
  • Improve usability and workflows
  • The current version is static in particular w.r.t. emulator curation. The next version will support centrally maintained, containerized components, in particular emulators. When the system starts it will check for updated software packages and will download new components if required.
  • Update of available workflows
  • Deployment for reading-rooms via PXE

Category: bwFLA Projekt, DP Projekte, R&D | Leave a Comment

Emulation as a Service as a Docker (Beta)

Thursday, October 08th, 2015 | Author:

Finally, Isgandar found some time to play with Docker. Docker or containers in general allow to shift installation configuration effort from end-users (admins) to the developers. As the bwFLA EaaS-framework is not easy to install that seemed a perfect fit.

Some prerequisites:

For this example we chose to divide & conquer all EaaS component, ie. every EaaS component is deployed as an individual container. This way, we can explain the functionality of each container and the configuration becomes readable. All explanations refer to the runner scripts from our github repository.
Common parameters (mandatory for all scripts)
  • We start all scripts with sudo. If you are unsure, you can run the container in non-priviliged mode, however with some functionality not available (e.g. uploading files directly into the emulated environment).
  • Choose the flavour / version the script should start with –docker eaas/bwfla:demo-august-15
  • As we package application servers, each component needs to be network accessible. Docker takes care of temporary setting of ‘iptables’ rules such that client doesn’t need to perform mapping of ‘host <-> guest’ ports (i.e. NAT). Make sure that your Firewall doesn’t interfere with the rules set-up by Docker application. Any IP valid for any network interface of the host machine, including 127.0.0.1 should be good.Example:  –public-ip-port 1.2.3.4:8080
Currently EaaS framework consists of at least four components (actually there are five, but the object-archive has not been packaged yet):
  • EmuComp – A container capable of running emulators. Typically such containers are deployed in cluster or cloud environments.
     sudo ./run-flavor-emucomp.sh --docker eaas/bwfla:demo-august-15 --public-ip-port 1.2.3.4:8081

    This script starts ’emucomp’ module, which is responsible for running individual emulators on dedicated compute nodes (possibly in a cluster/cloud environment). As of now emulators supported in the docker are the following: Qemu, SheepShaver, Basilisk, DosBox, Hatari.

  • EaaS– A container acting as gateway, assigning user-requests to EmuComps in a cloud / cluster deployment. This version only allows only fixed a EmuComp cluster. For dynamic cloud allocation please ask.
     sudo ./run-flavor-eaas.sh --public-ip-port 1.2.3.4:8082 --docker eaas/bwfla:demo-august-15 --emucomp 8.8.8.8:8080,4 --emucomp 9.9.9.9:8080,8

    At least one ‘–emucomp <VALUE>’ has to be specified. Repeat the argument if multiple ’emucomp’ modules have to be connected to this ‘eaas’ module. Value of the arguments is composed of the IP:PORT of the ’emucomp’ to be connected and the number of sessions it should supports (usually set to the node’s CPU-count) coming after comma.

  • Image-Archive– The image archive component manages virtual environments (disk images, as well as corresponding meta-data).
    sudo ./run-flavor-imagearchive.sh --docker eaas/bwfla:demo-august-15 --public-ip-port 1.2.3.4:8083 --archive-dir /mnt/data/image-archive

    This script run the ‘image-archive’ module. It requires the location of the image archive directory as a parameter. The ‘nbd-export’ directory should not contain any symbolic links that point to locations outside of the image-archive directory (i.e. only relative paths). This is due to the fact that the image-archive is mounted inside the docker container, which in turn has no access to the host’s file-system.

    Important note:  make sure that the public-ip-port setting chosen for your image-archive instance is accessible by the workflow module and emucomps, in particular make sure that port 10809 is not firewalled. 

    Update: An example image-archive is available for download (26 Mb). It contains 3 example images and meta-data. To use the archive, unpack and export all image files within the images/base directory using symbolic links (make sure you use relative links, see example below). If your OS does not support symbolic links, just copy the images to the nbd-export directory. 

    tar xf image-archive.tgz  #export images via nbd  
    cd image-archive/nbd-export/ 
    ln -s ../images/base/doom.raw 
    ln -s ../images/base/hatari_tOS206us.img 
    ln -s ../images/base/qemu-i386-DOS_6.20_CDROM.raw

     

  • Workflows – Frontend UI and ready-made workflows. The workflows connect the EaaS gateway (and its configured EmuComps) with user content, such as the image-archive. It contains sample preservation workflows which include archival “ingest/access” of digital objects, full disk images, any accompanying software/libraries. This contains exactly the same functionality as our demo site.
    sudo ./run-flavor-workflows.sh –docker eaas/bwfla:demo-august-15 –public-ip-port 1.2.3.4:8083 –image-archive 1.2.3.4:8083 –eaas-gateway 1.2.3.4:8082 [–object-files test/object-files –base-uri http://1.2.3.4/objects/]This script starts the ‘Workflows’ module, which represents a reference implementation of the bwFLA/EaaS API.The ‘–image-archive’ should point to a location of an image-archive, which contains base image of emulated systems, their derivatives, etc.

    The ‘–eaas-gateway’ should point to a location of an EAAS module, which will serve as a main ‘Facade’ for accepting and performing emulation tasks by using one or more ’emucomp’ modules on dedicated compute-nodes.

    The ‘–object-files’ should point to a directory containing user-objects that will appear in the ‘ingest’ workflow. The directory should contain objects in the form “OBJECT_NAME/OBJECT_NAME.iso”. E.g. ‘test/object-files/OBJECT1/OBJECT1.iso’. The ‘-base-uri’ must be specified iff ‘–object-files’ was specified previously. This argument defines the URL-prefix for the location of the objects via which the object can be download through HTTP protocol (’emucomp’ module needs to be able to download the object to inject it into the environment).

    NOTE: we will publish more information on the object-archive structure here. 

Category: bwFLA Projekt, R&D | Leave a Comment

bwFLA EaaS: Releasing Digital Art into the Wild

Saturday, December 21st, 2013 | Author:

With the bwFLA Emulation-as-a-Service you can enable users to view your (interactive) objects without actually giving the environment+object to the user. This is a nice feature, especially for dig. art and similar: you can provide access to an almost unlimited amount of people being able to view, use and interact with a piece of dig. art without being able to copy it. The owner remains in control of the object and is able to restrict access any time.

Jon Thomson and Alison Craighead (www.thomson-craighead.net) nicely prepared and integrated two of their art pieces for public access using the bwFLA EaaS infrastructure. Please take a look at:

http://www.thomson-craighead.net/docs/thal.html

and

http://www.triggerhappy.org

Please be patient it may take a minute or so to load. More information can be found here http://thomson-craighead.blogspot.de/2013/12/thalamus-now-emulated-online.html

Category: bwFLA Projekt, R&D | Leave a Comment

iPRES 2013, 1. Tag

Tuesday, September 03rd, 2013 | Author:

Der erste Tag der iPRES startete nach der Keynote von Gildas Illien (Bibliothèque nationale de France) mit zwei parallelen Tracks zu “Web Archives” und “Object Preservation”. Der zweite Track startete mit einer Evaluation von zwei Standardwerkzeugen zur Database Preservation, SIARD und Chronos. Das Archivieren einer Datenbank ist üblicherweise nicht einfach nur der Dump des Inhalts in eine oder mehrere Dateien. DB Retirement kann zudem bedeuten, dass (nicht mehr benutzte) Daten schrittweise aus dem RDBMS herausgezogen werden. more…

Category: bwFLA Projekt, Events | Leave a Comment

bwFLA Demo

Monday, August 05th, 2013 | Author:

Das bwFLA Projekt hat das Ziel Emulation als Werkzeug für den Langzeitzugriff auf verschiedenartigste (komplexe) digitale Objekte und im Forschungsdatenmanagement nutzbar zu machen. Hierfür wurden Use-Cases u.a. aus dem Bereich Kunst und Wissenschaft identifiziert, entsprechende Workflows entwickelt und beispielhaft umgesetzt. Mit der Fertigstellung der Ingest– und Access– Workflows wurde ein wesentlicher Meilenstein des bwFLA Projekts erreicht und soll nun einer breiteren Nutzergruppe vorgestellt werden. more…

Category: bwFLA Projekt | Leave a Comment

Workshop zur Langzeitarchivierung (in Hessen)

Wednesday, July 03rd, 2013 | Author:

Heute fand in Frankfurt am Main an der Goethe Universität Campus Westend ein Workshop zur Langzeitarchivierung statt, der mögliche Ansätze und Lösungen für den generellen Umgang mit digitalen Objekten und die Archivierung von Forschungsdaten in Hessen statt. Den Auftaktvortrag bestitt das bwFLA-Projekt und gab einen Überblick zur Idee von Emulation und dem im Projekt implementierten Framework und seine Einsatzmöglichkeiten. more…

Category: bwFLA Projekt, Events | Leave a Comment

Full-System-Preservation eines Motorola Mac

Tuesday, June 11th, 2013 | Author:

Archive erhalten Nachlässe bedeutender Persönlichkeiten, die zunehmend auch digitales Material, wie Datenträger oder auch komplette Geräte umfassen. So wie früher Skizzenbücher, Notizen oder geschriebene Briefe Arbeitsmittel waren, übernehmen Computer zunehmend diese Rolle. Ein Beispiel für einen solchen Nachlass ist das Vilem Flusser Archiv an der Universität der Künste in der Berlin, welches einen Apple Macintosh mit einem bedeutenden Werk und Disketten mit dem ersten elektronischen Buch übernommen hat. more…

Category: bwFLA Projekt | Leave a Comment

nestor Treffen AG Emulation Berlin/Mitte

Monday, June 10th, 2013 | Author:

Heute traf sich erneut die AG Emulation des nestor Kompetenznetzwerks. Die AG sorgt für den regelmäßigen Austausch unter Forscher/PraktikerInnen über aktuelle Entwicklungen und laufende Projekte. Bei diesem Treffen standen in Vorbereitung einer GI-Veranstaltung insbesondere die Verbindung von Funktionaler Langzeitarchivierung und Open Data im Vordergrund. Emulation kann zu einem wichtigen Hilfsmittel für den langfristigen Datenzugriff werden, da die wenigsten Primärdaten aus sich heraus erklärt und interpretiert werden können, sondern auch immer deren Weiterverarbeitung und Darstellung eine Rolle spielt. more…

Category: bwFLA Projekt, DP Projekte, Events | Leave a Comment

IDCC Conference Amsterdam, Tag 2

Wednesday, January 16th, 2013 | Author:

Der zweite Konferenztag startete mit einer Keynote des Los Alamos National Laboratory zu “The Web as infrastructure for scholarly research and communication” und teilte sich anschließend in Parallel-Tracks auf: “Institutional Research Data Management”, “National Perspectives in Research Data Management” und “Repositories/Data Archives”. In den  Sessions zur Vorstellung von Forschungs- und angewandten Themen ging es um “Cloud Services”, “Education & Training” und “Confidentiality/Open Research Data”. more…

Category: bwFLA Projekt, DP Projekte, Events | Leave a Comment

Projektvorstellung am Nationalarchiv der Niederlande

Monday, January 14th, 2013 | Author:

Das Nationalarchiv der Niederlande beschäftigt sich mit verschiedenen Wegen auf betimmte komplexe Objekte, wie beispielsweise alte Buchhaltungsdaten oder Personal Archives mit sehr verschiedenen Objekttypen zugreifen zu können. Emulation wird als eine wesentliche Zugriffsstrategie gesehen. Im Zuge des Austauschs wurde der Zugriff auf eine breite Palette verschiedener Originalumgebungen mit bwFLA-Workflows demonstriert. more…

Category: bwFLA Projekt, DP Projekte | Leave a Comment