1 Device File Palacios Extension
3 Akhil Guliani and William Gross
4 (Advised by Peter Dinda)
5 Northwestern University
9 This is a proof-of-concept implementation of device file
10 virtualization in the spirit of the Paradice system. Unlike Paradice,
11 it is implemented using only a preload library in the guest (no guest
12 modifications) and a kernel module in the host (no kernel
13 modifications). Also, there are no modifications to the Palacios
14 VMM either, as the core mechanisms used are the host hypercall
15 interface and the user-level guest memory access capabilities
16 implemented within Palacios.
18 Note that it is a proof of concept. It does not do many of the things
19 that Paradice can do. For example, a hypercall currently involves a
20 hard stop of the guest core. Address space integration in the shadow
21 process is page granularity only. Pointer detection for system call
22 forwarding is done at a coarse granularity, meaning that ioctls would
23 need ot be hand coded. Select across guest and host fds is
28 The basic idea is that we introduce a preload library into a guest
29 process. This library hijacks system calls. When you open a /dev
30 device that is in the list of devices we are proxying, the libraryio
31 handles the open by converting it to a hypercall, and binding the
32 result of the "open" hypercall and the fd returned. On subsequent
33 system calls involving the fd, the system calls are also converted to
34 hypercalls. The preload library merges fds/syscalls handled by the
35 guest and those handled by the host. The preload library also assures
36 that any data accessed is touched (page table entry exists), and could
37 pin it (currently does not). It also limits any pointer argument to
38 point to an block that fits within one page (e.g., read(1K offset,
39 4K length) turns into read(1K offset, 3K length).
41 The hypercall is directed to the second component, a kernel module.
42 The kernel module swizzles pointers involved in the system call from
43 their GVAs to their GPAs. It then queues them for interaction with
44 host user space process called the shadow process. The kernel module
45 and the shadow share a page used to transfer the system call arguments
46 and the return value and errno. The kernel module signals the shadow
47 that a new swizzled system call is available by letting a poll/select
48 complete. The shadow signals the kernel module that it is finished
49 with the system call via an ioctl.
51 The shadow process maps the guest's physical memory into its address
52 space using the guest memory access mechanisms. It then goes into a
53 select waiting for the kernel module. When it receives a system call
54 from the kernel module, it swizzles any pointer arguments in the
55 system call from their GPAs (provided by the kernel module) to their
56 corresponding HVAs (where the guest is mapped). It then issues the
57 system call, and writes back on the shared page both the return code
58 and the current errno value. It then signals completion via the
61 The kernel module then returns from the hypercall to palacios, which
62 returns to the guest preload library, which copies out the relevant
63 results so that it appears that a system call has completed (on the
66 Note that this model can be potentailly also be used as a general
67 system call forwarding mechanism.
71 In this directory (palacios/gears/services/devfile) you will find the
72 prelaod library and the kernel module. There is also a simple test
73 program for the guest.. In the palacios user space directory
74 (palacios/linux_usr), you will find the shadow process code. The
75 latter is separate as it has dependencies on the guest memory access
76 library and build config.
78 In palacios/gears/services/devfile/scripts, you will find scripts
79 which may help to evaluate the system.
84 [root@v-test-r415-3 linux_usr]# ./v3_devfile_shadow
85 v3_devfile_shadow <vm_device>
87 Shadow process to support device file-level virtualization
88 in the spirit of Paradice.
90 This operates with the devfile_host.ko kernel module
91 in the host and devfile_preload.so preload library in
92 the guest. These can be found, along with example scripts
93 in palacios/gears/services/devfile.
95 The general steps are:
96 1. insmod kernel module into host
97 2. copy preload library into guest
99 4. use v3_hypercall to bind the devfile hypercall (99993)
100 to the kernel module for your guest
101 5. run v3_devfile_shadow
102 6. run process in guest that uses preload library
105 ****Scripts and More Detail Below****
109 Copy the patched_start_guest, patched_mem_script,
110 patched_insert_hypercall, and patched_view_console to the top of the
111 palacios directory then cd to the to top of the directory so that you
112 are at path/to/palacios/ and the scripts are at
113 path/to/palacios/patched_*
119 For this system to work, the guest needs a couple things. It needs to
120 be configured with a CGA console (to get v3_console to work). It needs
121 to have sufficient memory (1024kb seems to work). It needs to accept
122 an LD_PRELOAD library. It needs to be an x86_64 architecture OS. It
123 needs to be running a Linux kernel. It needs a second drive set up as
124 a CD-ROM. Once that second drive is set up, replace the backing file
125 with a handmade loopback file system (perhaps using dd). For our
126 scripts to work exactly, this handmade file system should be called
127 littlefs.dat and an empty directory (for mounting) should be present
128 in the guest folder called tmp. The scripts are short and few, so the
129 relevant paths in them can be modified as necessary to fit your setup.
133 Start Guest with Device File Forwarding:
135 in that terminal source patched_mem_script in other terminal go to the
136 same position and source insert_hypercall if the terminal doesn't open
137 into the guest, resize your terminal appropriately and then source
138 patched_view_console and then from the guest do the following
143 source load_lib_in_guest
145 EXAMPLE: ./test_preload r 10 /dev/urandom
147 This will attempt to read 10 bytes from /dev/urandom, where /dev/urandom
150 If that last argument (/dev/urandom)is present in the
151 devfile_preload.c library, then the system will access the host
152 version of that device, if not it will just perform the regular system
153 calls on the guest's version of that device if it is present.
156 Close Running Guest with Device File Forwarding:
158 To shut down smoothly kill the shadow process run from the first
159 terminal (patched_mem_script) make sure the second terminal has left
160 the guest by pressing \ then in either terminal source
164 Adding Host Devices in /dev to the list of Supported Devices:
166 go to path/to/palacios/gears/services/devfile
167 edit guest_devices.txt
168 then run from the command line 'python guest_device_setup.py'
170 this will update the dev_file_ld_lib.c file, which can be made from
171 that directory the object file is copied over to the guests /dev/hdb
172 drive within the patched_mem_script NOTE: Adding devices requires the
173 preload library (devfile_preload.c) to be made and copied to the guest
174 so, they cannot be added while a guest is running with this
175 feature. Close the guest first, add the new device changes, run the
176 python script, make the /gears/services/dev_file directory, start the
181 Extending the System Call Interface:
183 devfile_preload.c is the LD_PRELOAD library for the guest that hijacks
184 some basic system calls to see if they should be forwarded to the
185 host. To support a broader range of devices, this basic set can be
186 added to. Any addition should follow a similar structure to the read
187 or write system call in there. The things to make sure are that all
188 pointer arguments get pinned into memory and that the file descriptor
189 argument is checked against the set of active fds in the dtrack struct
190 (also in the library file).
