\title{
\includegraphics[height=1.5in]{v3vee.pdf}
-\includegraphics[height=1.5in]{palacios.pdf} \\
+\includegraphics[height=1.5in]{logo6.png} \\
\vspace{0.5in}
Palacios Internal Developer Manual
}
-\author{Jack Lange \hspace{0.5in} Lei Xia}
+
\maketitle
-\section{Organization}
+This manual is written for Internal Palacios developers. It contains
+information on how to obtain the palacios code base, how to go about
+the development process, and how to commit those changes to the
+mainline source tree. This assumes that the reader has read {\em An
+Introduction to the Palacios Virtual Machine Monitor -- Release 1.0}
+and also has a slight working knowledge of {\em git}.
+
+
+\section{Overview}
+
+
+Both Palacios and Kitten follow a hybrid development process that
+uses both the centralized repository and distributed development
+models. A central repository exists that holds the master version of
+the code base. This central repository is cloned by multiple people
+and in multiple places to support various development efforts. A
+feature of git is that every developer actually has a fully copy of
+the entire repository, and so can function independently until such
+time as they need to resync with the master version.
+
+There are typically multiple levels of access to the central
+repository, that are granted based on the type of developer being
+granted access. The three basic developer types and their access
+privileges are:
+
+\begin{itemize}
+\item Core Developers: These are the lead developers and are in
+charge of managing the master repository. They have full read/write
+access permissions to the central repository.
+
+\item Internal Developers: Formal members of the development
+team. These people are capable of pulling directly from the central
+repository, but lack the ability to write directly to it.
+
+\item External Developers: People who are not actual members of the
+development team. These people can only access the public repository
+which is only updated to contain the release versions.
+\end{itemize}
+
+Because internal and external developers cannot write directly to the
+master repository, they need to first submit their changes to a core
+developer before they can be added to the mainline. We will discuss
+that process in Section~\ref{sec:submission}.
\section{Checking out Palacios}
+The central palacios repository is located on {\em
+newskysaw.cs.northwestern.edu} in {\em /home/palacios/palacios}. All
+internal developers have read access to the directory. Each developer
+must create their own local version of the repository, this is done
+with {\em git clone}.
+
+\begin{verbatim}
+git clone /home/palacios/palacios
+\end{verbatim}
+
+This creates a local copy of the repository at {\em ./palacios/}.
+
+
+All development work is done in the {\em devel} branch of the
+repository. The developer can access this branch via:
+
+\begin{verbatim}
+git checkout --track -b devel origin/devel
+\end{verbatim}
+
+or
+
+\begin{verbatim}
+/opt/vmm-tools/bin/checkout_branch devel
+\end{verbatim}
+
+{\em Important:}
+Note that palacios is very actively developed so the contents of the
+{\em devel} branch are frequently changing. In order to keep up to
+date with the latest version, it is necessary to periodically pull the
+latest changes from the master repository by running \verb.git pull..
-Checkout or clone the devel branch of Palacios from the master
-repository. You should have the read permission to these branches.
\section{Checking out Kitten}
-hg clone /home/palacios/kitten
+Kitten is available from Sandia National Labs, and is the main host OS
+we are targetting with Palacios. Loosely speaking core Palacios
+developers are internal Kitten developers, and internal Palacios
+developers are external Kitten developers. Because we have limited
+access to the Kitten repository, we are maintaining a local mirror
+copy in {\em /home/palacios/kitten}.
-git clone /home/palacios/palacios
+Kitten uses Mercurial for their source management, so you will have to
+make sure the local mercurial version is configured correctly.
+Specifically you should add the following python path to your shell environment.
-/opt/vmm-tools/bin/checkout\_branch devel
+\begin{verbatim}
+export PYTHONPATH=/usr/local/lib64/python2.4/site-packages/
+\end{verbatim}
+You can then clone Kitten from the local mirror:
+\begin{verbatim}
+hg clone /home/palacios/kitten
+\end{verbatim}
-\section{Compiling Palacios}
-cd palacios/build/
+Both the Kitten and Palacios clone commands should be run from the
+same direcotyr. This means that both repositories should be located at
+the same directory level. The Kitten build process depends on this.
+{\em Important:} Like Palacios, Kitten is very actively developed so
+source tree is frequently changing. In order to keep up to date with
+the latest version, it is necessary to periodically pull the latest
+changes from the mirror repository by running \verb.hg pull. followed
+by \verb.hg update..
+
+\section{Compiling Palacios}
+Palacios is capable of targeting 32 and 64 bit operating systems, and
+includes a build process that supports both these
+architectures. Furthermore, Palacios has multiple build locations,
+with multiple makefiles: a top level build directory and a Palacios
+specific build directory. The Palacios build process first generates a
+static library that includes the Palacios VMM. This static library is
+then linked into a host operating system. Palacios internally supports
+GeekOS and can generate a complete OS image via a unified build
+process. To combine Palacios with Kitten, it is necessary to first
+compile Palacios and then to compile Kitten externally link it with
+Palacios. The output of the compilation process is a bit more complex
+and generates multiple binaries, and the specifics can be found in the
+Makefiles.
+
+The top level build directory provides a number of high level make
+targets, and is located in {\em palacios/build/}. It supports building
+32 and 64 bit versions of the Palacios library independently as well
+as building an integrated version of GeekOS. The basic targets are:
+\begin{itemize}
+\item \verb.make palacios-full32. -- Generates a 32 bit version of the Palacios static library
+\item \verb.make palacios-full64. -- Generates a 64 bit version of the
+Palacios static library
+\item \verb.make geekos. -- Compiles the GeekOS kernel, and link it with the
+Palacios static library
+\item \verb.make geekos-iso. -- Generate an ISO boot disk image from the
+GeekOS kernel that has been compiled
+\end{itemize}
+
+The second build directory is located at {\em palacios/palacios/build}
+and handles only the Palacios compilation process. It supports a
+differnt set of targets and arguments:
+\begin{itemize}
+\item \verb.make ARCH=32. -- iteratively compiles a 32 bit version of Palacios
+\item \verb.make ARCH=64. -- iteratively compiles a 64 bit version of
+Palacios
+\item \verb.make ARCH=32 world. -- fully recompiles a 32 bit version of
+Palacios
+\item \verb.make ARCH=64 world. -- fully recompiles a 64 bit version of
+Palacios
+\end{itemize}
+
+Both build levels support compilation directives that control the
+debugging messages that are generated by Palacios. These are specified
+by appending a \verb.DEBUG_<COMPONENT>=1. to the end of the
+\verb.make. command. The components that are currently supported are:
+\begin{itemize}
+\item \verb.DEBUG_ALL=1. -- enables debugging for all the VMM components
+({\em Warning:} this generates a {\em lot} of debug information.
+\item \verb.DEBUG_SHADOW_PAGING=1.
+\item \verb.DEBUG_CTRL_REGS=1.
+\item \verb.DEBUG_INTERRUPTS=1.
+\item \verb.DEBUG_IO=1.
+\item \verb.DEBUG_KEYBOARD=1.
+\item \verb.DEBUG_PIC=1.
+\item \verb.DEBUG_PIT=1.
+\item \verb.DEBUG_NVRAM=1.
+\item \verb.DEBUG_GENERIC=1.
+\item \verb.DEBUG_EMULATOR=1.
+\item \verb.DEBUG_RAMDISK=1.
+\item \verb.DEBUG_XED=1.
+\item \verb.DEBUG_HALT=1.
+\item \verb.DEBUG_DEV_MGR=1.
+\item \verb.DEBUG_APIC=1.
+\end{itemize}
-This will build Palacios as a library, libv3vee.a in the palacios/palacios/build/.
\section{Compiling Kitten}
+Kitten requires a 64 bit version of Palacios, so make sure that
+Palacios has been correctly compiled before compiling Kitten.
+
\subsection{Configuration}
-Kitten building can be configured by either text or graph configure interface, which is similar to the Linux kernel configure, By one of the following commands:
+Kitten borrows a lot of concepts from Linux, including the Linux build
+process. As such it must be configured before it is actually compiled.
+The Kitten configuration process is the same as Linux, and can be
+accessed via any of these make targets.
+\begin{itemize}
+\item \verb.make xconfig.
+\item \verb.make config.
+\item \verb.make menuconfig.
+\end{itemize}
+
+There are some specific configuration options that should be disabled
+to work with Palacios. Because Palacios is configured by default to
+provide a guest with direct access to the VGA console, the {\em VGA
+console} device driver should be disbabled in the Kitten
+configuration. Similarly the {\em VM console} driver should be
+disabled as well.
+
+Furthermore, because the VGA console is not being used the {\em Kernel
+Command Line Arguments} must be modified to remove the {\em VGA}
+device from the console list.
+
+The guest OS that is booted as a VM is included as an ISO image in raw
+binary format inside Kitten's {\em init\_task}. To change the guest
+ISO, you must change the makefile for the init\_task. This is located
+in {\em user/hello\_world/Makefile} and the syntax is well commented.
+On {\em newskysaw} a collection of guest ISO images are located in
+{\em /opt/vmm-tools/isos/}.
-make xconfig
-make config
-make menuconfig
-Make sure turn on the network device driver, networking, and input kernel command 'console=serial net=rtl8139'
\subsection{Compilation}
-
-Build Palacios as a module for Kitten
-In the first time, make sure to build Kitten before you building the Palacios as the module to kitten.
-Palacios now is built as a module of the Kitten. You can find the palacios.c and palacios.h in the kitten/palacios/. Enter the directory, build the palacios module.
-
-cd kitten/palacios
+After Kitten has been configured the compilation can be done. The
+general process is to compile a reference build of Kitten, followed by
+compiling Palacios support as a kernel module, and then doing a new
+full recompilation of Kitten.
+
+The specific compilation steps are run from the top level Kitten directory:
+\begin{verbatim}
+make
+cd palacios
make -C .. M=`pwd`
cp built-in.o ../modules/palacios-mod.o
-Build Kitten
-Go back to kitten root directory, and build the Kitten again.
+cd ..
+make
+make isoimage
+\end{verbatim}
+
+This generates an ISO boot image containing Kitten, Palacios, and the
+guest that will be run as a VM. The ISO image is located at {\em
+./arch/x86\_64/boot/image.iso}.
-make isoimage
\section{Running Palacios/Kitten}
-Run the whole stuff built above in Qemu using following command:
+Kitten and Palacios are capable of running under Qemu, which makes
+debugging much simpler.
+
+The basic form of the command to start the Qemu emulator is:
+\begin{verbatim}
+/usr/local/qemu/bin/qemu-system-x86_64 -smp 1 -m 1024 \
+ -serial file:./serial.out \
+ -cdrom ./arch/x86_64/boot/image.iso \
+ < /dev/null
+\end{verbatim}
+
+The command starts up a single processor emulated machine, with 1gig
+of RAM and a cdrom drive loaded with the Kitten ISO image. Furthermore
+all output to the serial port is written directly to a file called
+{\em serial.out}. This command can be copied into a shell script for easy access.
-/usr/local/qemu/bin/qemu-system-x86\_64 -smp 1 -m 1024 -serial file:./serial.out -cdrom ./arch/x86\_64/boot/image.iso -net tap, ifname=tap0 < /dev/null
+\section{Development Guidelines}
+32/64 bit compatibility
+name space
+coding guidelines
-\section{networking}
+
+\section{Code Submission}
+\label{sec:submission}
+\subsection{Palacios}
+
+\includegraphics[height=3.5in]{dev_chart.pdf}
+
+stacked git
+
+\subsection{Kitten}
+
+mercurial queues
+
+\section{Networking}
\section{Configuring the development host's Qemu network}
Set up Tap interfaces:
\subsection{Configuring Kitten}
+To enable networking in Qemu, networking needs to be enabled in the configuration.
+
+Make sure turn on the network device driver, networking, and input
+kernel command 'console=serial net=rtl8139'
+
How to set ip address in kitten:
Kitten ip address setting is in file drivers/net/ne2k/rtl8139.c, in the code below which is located in function rtl8139\_init.
+
+
For more questions, talk to Jack or Lei.
\end{document}
