Grant-Table User-Space Device | Progress Report

My proposal was selected for Google Summer of Code 2016 under FreeBSD. The primary goal of my project is to write a character device that will expose Xen Grant-Tables to userspace. You can find more details about the project in the proposal, here. This post will act as a quick progress update on the work I am doing.

Community Bonding Period

April 23 – May 5

• Examinations till April 25.
• Semester project submissions till May 2.
• Reached home – May 5.

May 6

• Search for workstation continues. For the project I need an Intel box with VTx and VTd, and with serial output (to debug bugs inside of Xen).
• Found my old laptop. It’s a i7-720QM. Specs confirm it supports VTx and VTd (though I am yet to confirm MoBo support for VTd). It has an ExpressCard/54 slot that I can use to add a serial port (Amazon).
• Backed up HDD, and installed FreeBSD.
• No WiFi – Broadcom BCM43224 🙁
• Does not boot. Can’t enter BIOS, or boot from USB. (weird) Laptop stuck at POST test.

May 7

• Time to open up the laptop.
• CMOS reset – Check.
• Take out the HDD and boot from USB – Works perfectly.
• Ralink RT 5370 USB WiFi adaptor works.
• Borrowed an HDD from another laptop. Install FreeBSD-11.0-CURRENT. Again, same issue. 🙁
• Weirdly, both HDDs boot fine inside the other laptop.
• Tried a lot of configuration options, ACPI, etc. but to no avail.
• It boots fine from a USB HDD, which is a GUID partition table having a single partition. Maybe, installing to the entire HDD with a similar configuration, instead of a single partition will work.
• Yay! Finally FreeBSD running on the target laptop.

May 8

• Time to install Xen.
• Stuck in a boot loop. No logs on screen.
• Ah! No logs because I added “console=com1” in /boot/loader.conf
• Quickly glanced through logs on screen (now I know the importance of serial output). CPU Panic because can’t allocate 2GB memory for Dom0.
• Loose RAM connection. I should have been careful when taking out the HDD yesterday.
• Xen boots!!
• Time to create a Ubuntu guest.
• Config file:

memory = 1024
vcpus = 2
name = "ubuntu"
disk = [ '/root/xen_files/ubuntu.img,raw,hda,w',
         '/root/xen_files/ubuntu.iso,raw,hdc:cdrom,r' ]
boot = "c" # Boot to hard disk image
# boot = "d" # Boot to ISO image
usbdevice = 'tablet'
vnc = 1
vnclisten = '0.0.0.0'
vncdisplay = 10
vncpasswd = '******'
vif = [ 'bridge=bridge0' ]

• Can connect using vnc. Ubuntu install on ubuntu.img done and boots up perfectly.
• Network in guest doesn’t work. wlan0 bridged to guest. Will set up NAT if need be. (would have pci-passthrough-ed the Broadcom WiFi chip if it was supported)

May 12

• Work on task started.
• Checked out FreeBSD code. (~30 minutes with 2 svn failures :O )
• Let’s build the GENERIC kernel once, and install it. Will make future builds faster.
• >>> stage 2.1: cleaning up the object tree 🙁 Should have known about -DKERNFAST.
• Still building…
• Build completed in 1hr12min. 😀
• Installing… Done.

May 14

• Patched <xen/error.h> to use the public Xen error-code header.
• Recompiled and tested the kernel. Boots up fine. No unexpected messages in log.
• Guest boots up fine.
• Diff-patch generated and sent.

May 17

• Patch modified according to requirements. Removed error-codes not defined in the public header.
• Reading existing Grant-Table code.
• <xen/gnttab.h> is missing the #define in the include guard. Reported.

May 21

• Git is now a perfectly valid option (Perforce, my first choice, wasn’t available). Yay!!!
• Forked FreeBSD tree on Github. Development on branch grant_table.
• https://github.com/akshay1994/freebsd/tree/grant_table
• Patch 1: Rewrite Xen error translation using the public Xen error code header.
• Patch 2: Add missing #define in include guard.

May 19 – June 1

• Holiday @ London.
• I carried the 6kg development machine along with me everywhere. But it was put to very little use.

June 3 – June 4

• Lots of reading.
• Newbus API – https://www.freebsd.org/doc/en_US.ISO8859-1/books/arch-handbook/newbus.html
• Kobj and object oriented system – https://www.freebsd.org/doc/en_US.ISO8859-1/books/arch-handbook/kernel-objects.html
• Privcmd user-space device – https://github.com/akshay1994/freebsd/commit/919b7d8b7c042461296e10914c4a96ee74240c0a
• Xenbus device – https://github.com/akshay1994/freebsd/tree/master/sys/xen/xenbus
• Event-channel user-space device.
• Current Grant table code.

June 5

• Understanding how Grant-Table works, the data structures and processes involved, etc.
• Thorough reading of grant_table.h
• gnttab.h
• Reading Newbus device implementation of grant table @ grant_table.c (also viewing history of the code, etc.).
• Question 1: Why no interface (.m) file for the newbus device?
• Question 2: Should we shift to version 2 of grant table entry structure?
• Question 3: Should I add ioctls and mmaps to this device itself, exposing functionalities to userspace?

June 6

• More and more code reading.
• In file xen-os.h, line 121 should be #undef NBPL, NBPL defined on line 99. Couldn’t find symbol NPBL defined anywhere in codebase.
• Will create a patch and send.
• Patch 3: xen: Correct typo in #undef for symbol NBPL
• Committed as r301511. 🙂
• Questions answered.
• New device preferred, instead of adding ioctls and mmaps to the internal device.
• Version 2 of grant table entry structure is dead.
• Instead of integrating gnttab_* functions inside a newbus interface, it is preferred to do so inside the BUSDMA API.
• Reading about BUSDMA API.

June 7

• Day spent setting up development environment.
• Development can’t happen on the machine on which the code is supposed to be run.
• IDE looks tempting, especially … symbol search, etc.
• Fired up Netbeans. Load freebsd code. IDE hung at parsing. xD
• Increase Java Heap size to 5gigs. Parsing takes 20 minutes :O
• Configured remote build host to be the freebsd target development machine.
• First build took time (uploading all code files).
• Had to set up include paths.
• IDE up 🙂

June 8

• Let’s start writing the device interface.
• xen/gntdev.h will describe the userspace device interface.
• The ioctl interface is kept in sync with the linux interface, and is the union of the linux gntalloc and gntdev interfaces.
• Write device attachment/registration functions.

June 9

• This can’t work. It tries uploading the whole code every other time, and that takes ages.
• Share on network filesystem?
• Somehow Mac Samba share can’t be mounted on FreeBSD.
• I’ll share NFS from FreeBSD and mount on Mac. Connect them both by LAN to ensure speed.
• Works!

June 10

• Add some more code.
• Let’s compile and boot.
• Fast compile (-DKERNFAST) doesn’t touch the newly added file. Why? I’ve added it to sys/conf/files.
• Hmm, maybe it just tests for changes to files touched before.
• Compile without DKERNFAST.
• sys/dev/xen/gntdev.c compiled !!!
• Boots fine!
• Device visible at /dev/xen/gntdev
• Yay! It’s working.

June 11

• Since my work is involved with memory, I must get versed with the FreeBSD virtual memory infrastructure.
• Read code files <vm/vm*.h> to see available data structures and methods.
• Read the chapter on Memory Management (Chapter 6) of the book “The Design and Implementation of the FreeBSD Operating System”.
• That was some nice reading.
• Pages can’t be swapped out. Because Xen deals with Frames (physical mem), and if our kernel swaps the page out, Xen would still be sharing it with other domains and info-leak! I’ll check with Roger.

June 13

• Write code for grant allocators and de-allocators.
• Define internal data structures, like per user private data, per grant-reference data.
• Link these functions to the grant table device @ gnttab.h
• Finally, wrote some worthy code! 😀
• https://github.com/freebsd/freebsd/commit/b593857cd250c7b069d38be17f6abaf39d5b42e4
• https://github.com/freebsd/freebsd/commit/5d65b9880202748ced7b68961ce176d0190d0121

June 14

• Add proper synchronisation.
• Safely delete grant references and pages when foreign access ends.
• Other fixes.
• https://github.com/freebsd/freebsd/commit/0171ba078d3c85135f6ce6caa72bd465d0475105

June 15

• Write deallocator.
• Complete TODOs and fixup dtors.
• https://github.com/freebsd/freebsd/commit/b15b42a28ea46b6f2643fc05eafe4041dc698a6c

June 16 – July 21

I hope everyone is doing great.

First things first. I want to apologise for not updating my blogpost after June 15. Some personal stuff happened on June 17, and since then all my work routines are completely disturbed. Soc-admins know about this, and I hope the community can pardon my irregularity this once.

Now, time for some status update on the project. We have a working Grant Table User-space Device (with some caveats, of course). Yay!!

Check out the code on the pull request: https://github.com/freebsd/freebsd/compare/master…akshay1994:grant_table?expand=1.

Test the device by compiling a custom kernel; from my working branch: https://github.com/akshay1994/freebsd/tree/grant_table

Caveats and work yet to be done:

• A kernel panic happens if you terminate the program without un-mmapping all the mmaps.
• The interface for a grant map is not completely in sync with the Linux interface yet. Notably, we still map and subsequently mmap partially working grants. This is a bug since the pager will fail on a page fault for the non-working-grant.
• I’m yet to add event channel code for the notify ioctls.
• I’m yet to get in touch with the vm-guys. I spent a lot of time understanding the vm structure, but I surely do not have a complete picture. I’m using a MGTDEVICE cdev pager to handle managed, fictitious pages from mapped grants, and a custom PHYS vm object to manage wired physical memory for allocated grants. I have to
  • Confirm if the approach I am using is the best one.
  • The interface differs compared to Linux with regards to multiple mmaps, which I need a little help with.

Looking forward to some community testing of the working parts. Comments are welcome and highly appreciated. 🙂