Docker Done Right

Written by Michael Dexter on .

Yes, that is a bold statement. The Docker application containment architecture is all the rage right now and FreeBSD just may prove to be the ultimate Docker platform thanks to its 15+ years of containment experience and the unrivaled OpenZFS file system.

As one Twitter user put it, “#docker has now had more security issues within a year then
#freebsd #jails has had since 2000. Good job #techbros.”

Indeed, Docker has never been pitched as a security technology but rest assured, Docker on FreeBSD institutionally imprisons and secures Docker images using FreeBSD’s proven Jail infrastructure. FreeBSD Jails have been used in production since their inception to contain applications and full systems and are exactly what Docker needs. Docker itself has migrated away from Linux LXC containers in favor of the cross-platform libcontainer and of all the pluggable choices, FreeBSD’s Jail stands out as one of the best. FreeBSD also offers the bhyve and Xen hypervisors to provide you yet more options for containing your Linux-native and FreeBSD-native Docker deployments.

Then comes storage. Docker images are designed to be read-only and disposable until instructed otherwise. If only there were a file system that institutionalized lightning-fast snapshotting and cloning…

That file system exists! It’s called OpenZFS and FreeBSD has supported it since FreeBSD 7.0. This not only means you get the institutionalized snapshotting and cloning that suit Docker so well, but also the unrivaled data integrity protection that OpenZFS offers. If you care about your data, you care about OpenZFS.

Hands-on Docker

To try Docker on FreeBSD, you will need a recent snapshot such as 10.2 BETA or 11-CURRENT. Note that you should change “zroot” to match your system’s zpool.

 # pkg install docker-freebsd ca_root_nss
 # zfs create -o mountpoint=/usr/docker zroot/docker
 # service docker onestart
 Starting docker…

 # docker pull centos
 …

 # docker images
 REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
 centos latest 7322fbe74aa5 4 weeks ago 172.2 MB

 # docker run -t -i centos /bin/bash
 [root@ /]# uname -a
 Linux 2.6.32 FreeBSD 11.0-CURRENT #5 r285594: Tue Jul 14 23:30:11 EDT 2015
 x86_64 x86_64 x86_64 GNU/Linux

Suddenly… CentOS!

Where the wheels really hit the pavement is with a peek under the hood at the Jail and ZFS output of our Docker Jail and OpenZFS dataset:

 # jls
   JID IP Address Hostname Path
     3 172.17.0.3 /usr/docker/zfs/graph/920bc5fbb45c

 # zfs list
 …
 zroot/docker
       119M 107G 6.02M /usr/docker
 zroot/docker/03a7a57df9197f242484375c4bc2149248ded5aaafc4feb8e472d6774d495530
       8K 107G 112M legacy
 zroot/docker/03a7a57df9197f242484375c4bc2149248ded5aaafc4feb8e472d6774d495530-
 init 128K 107G 112M legacy
 …

This output should be familiar to FreeBSD users and is becoming familiar to more and more GNU/Linux users every day.

For an expanded example of Docker on FreeBSD, consult the FreeBSD Wiki:

https://wiki.freebsd.org/Docker

FreeBSD is poised to be go-to Docker platform thanks to FreeBSD’s proven Jail and OpenZFS features and iXsystems has shipped over ten thousand systems with the best support for these features available anywhere. We can also build out your GNU/Linux-based Docker deployment and ship thousands of GNU/Linux systems every year. Give us a call to learn how we can take your Docker deployment to the next level and beyond.

Michael Dexter

FreeNAS 10: Early M2 Preview

Written by dru on .

In the last post, Jordan provided an overview of the FreeNAS 10 middleware and architecture. In this post, we’ll provide a sneak peak at the upcoming M2 milestone by demonstrating how to install and navigate this version of FreeNAS 10. This is useful if you are curious about the status of the M2 milestone and it is required if you wish to spin up a development environment. The next post will demonstrate how to install the necessary development tools once you have M2 installed.

As of this writing, the M2 milestone is expected to be complete on August 15, at which time a public demo will be available. This means that M2 is under rapid development and that any screenshots and instructions shown in this blog post may change. While the M1 milestone, which ended in mid-June, concentrated on the design of the middleware, the primary goals of the M2 milestone are to provide the following functionality:

  • Installer.
  • Network configuration.
  • Pool creation.
  • The ability to create at least one type of share, but hopefully both NFS and AFP.
  • Usable dashboard, including drag-around functionality.
  • System configuration as well as power management, reboot/logout, and update management.

Mockups are available so that you can preview the layout of each UI element and you can track the status of the ongoing work at the FreeNAS 10 bug tracker.

It is recommended to install M2 into a virtual environment as it is still very much in the testing/preview stage. When creating the virtual machine, make sure that the virtual disk to install into is at least 16GB in size and that you create at least one more virtual disk to use as storage.

Installation images are available from http://download.freenas.org/10/Nightlies/M2 and, at this time, the installation is identical to the instructions for 9.3. Once installed, the system should boot into a prompt:

initial

You may or may not automatically receive an IP address. In this example, the system did not. In this case, type shell which will open up a root command prompt (this is similar to clicking Shell on a FreeNAS 9.3 system). If the network does have a DHCP server, use these commands to try to get a lease, replacing em0 with the name of the interface:

killall dhclient

/sbin/dhclient em0

If the DHCP lease fails or there is not a DHCP server, manually set an IP address that is appropriate for the network and will not conflict with other IP addresses. Again, replace em0 with the interface name.

/sbin/ifconfig em0 192.168.1.2

Once you have an IP address, point your browser to it. You should receive a login screen.

initial1

Type root into the “Username” field and press enter. The icon in the far upper right with a green dot and the word “root” indicates that you are logged in as the root user. Click that icon if you wish to “Logout”.

The left frame contains the configurable elements. Those in white text are under active development and their functionality will increase and bugs will decrease as M2 progresses. If an element looks different than its mockup, its layout is still under active development. Any element with grey text has not been added yet, meaning that it will appear as development on M2 progresses.

If you click “API Docs”, a new tab will open with links describing the RPC Interfaces, Events, Tasks, and JSON schemas used by FreeNAS 10.

If you click “Toggle Debug Tools”, a new frame will open at the bottom of the browser:

initial2

Click “Terminal” to open a root shell. A drop-down menu is available in that tab for selecting the shell type.

As M2 matures and stablizes, future blog posts will demonstrate its features.

 

 

 

 

 

FreeNAS 10: A developer’s perspective

Written by jkh on .

One of the greater challenges in software engineering, particularly once you’ve actually released your latest magnum opus to the world, is deciding what to do next.  How long do you continue to support version X, carefully fixing bugs and polishing the rock, before the list of architectural limitations and # of tickets requesting radical, substantial new features (all of which are potentially disruptive) begin to suggest that simple rock-polishing isn’t going to cut it as your only investment in engineering time and resources?

We reached that tipping point with the FreeNAS 9.x branch about 6 months ago.  While putting the finishing touches on FreeNAS 9.3, a release which was a substantial improvement over previous 9.x releases, it was clear that we were still fundamentally limited by various design decisions made during the 2nd major rewrite of FreeNAS (FreeNAS 8).

This is not to say that those decisions were wrong or bad – all software technologies evolve and hindsight is always 20-20 – but it’s still fair to say that the world is constantly changing and FreeNAS as a product has little choice but to change with it.

First, just to set the context for the description of FreeNAS 10 which follows, a quick Architectural overview of FreeNAS 9 is in order:

Figure1(figure 1)

The observant will quickly notice that the box marked “Middleware” is fairly large on that diagram.  This is because it is, at least from a conceptual standpoint!   In FreeNAS 9, the middleware is a monolithic entity that contains the javascript necessary to create the UI in the browser, the Django Web framework (a Model-View-Controller architecture used to view and modify data in the sqlite database), and all of the scaffolding necessary for talking directly to the underlying FreeBSD OS and a small constellation of helper tools (samba and netatalk being some of the most prominent, as they are used to implement CIFS and AFP file sharing) from the FreeBSD ports collection.

The biggest issue with the middleware, however, isn’t even the fact that it is a monolithic chunk of code with many categories of functionality mixed together with no clear separation of concerns or recognizable abstraction layers. The biggest issue is that the middleware implementation is synchronous, meaning that it can only do one UI-facing thing at a time and must wait for long-running operations to finish before the user can do anything else.   Being monolithic, the middleware also does not exist as a separate process outside the context of the current (and, in fact, only) UI session, the implications of which will become more obvious as you read on.

Let’s start with the most obvious impact of being synchronous, however.  Let’s say the user navigates to the Plugins UI, clicks on the Installed tab, and then tries to upgrade one of their plugins.  They will get the following behavior:

Figure2(figure 2)

As you can see, the UI is fully blocked while it downloads this plugin update.  There isn’t even an option to cancel the operation because the middleware is busy downloading the plugin and no longer truly interacting with the UI.  Should the user get bored and simply choose to navigate away from this page, the results are essentially undefined.

In some areas of the FreeNAS 9 UI, unexpectedly navigating away or closing the browser during a long-running operation can even leave the system in an inconsistent state.  This is not particularly robust behavior, to say the least, all again the result of having a single, highly-privileged middleware instance that both provides the UI and the underlying code for manipulating the state of the underlying machine as part of the user’s session.

As mentioned earlier, there can also only be one such session, which is to say that only one user can be logged into the box and make changes at a time.  What happens if two users log in to the same FreeNAS box and start making changes simultaneously?  Nothing good, in all likelyhood, because each middleware instance thinks that it is the only one (in true Highlander fashion), and has no awareness of the actions of the other.  This lack of overall global awareness, or any persistent management of the system after the user logs out, has other implications for the middleware and for the overall User Experience of FreeNAS 9.  Again, let’s use the existing UI as an illustration:

Figure3(figure 3)

Here we see 3 S.M.A.R.T. tests scheduled at different intervals using the Tasks UI.  Because the middleware may not be around to schedule these tasks (or, for that matter, any Cron jobs or Rsync tasks that the user might schedule), it has to be done by some other agent which has no direct relationship with the middleware.  In this case, that agent is the smartd daemon, the middleware itself (and configuration database) merely tracking the fact that those tests need to be scheduled with it.

This all works well enough, but should one of those S.M.A.R.T. tests fail or be otherwise deemed invalid by smartd, there is absolutely no way for the user to tell from the UI session because the middleware itself has no awareness of this, just as it has no awareness of the success or failure of rsync or cron tasks it schedules.  The middleware is essentially trusting that things it has asked other parts of the system to do on its behalf are functioning as expected.

Again, the middleware was basically and fundamentally not designed to deal with asynchronous events, and the tendency has therefore been to simply not even try to display real-time sources of data other than the most basic alert status and progress information, all of which has to be polled for in fairly painful fashion.   This has impacted the quality of the UI feedback that FreeNAS 9 is able, or even attempts, to provide in countless different ways and is one of the more fundamental shortcomings of the current user experience.

This blog post is supposed to be about FreeNAS 10 and not FreeNAS 9, however, so let’s just conclude this history lesson with one last point about the UI technology itself:  The technologies we used to build the 9.3 UI are definitely showing their age, and while toolkits like Dojo have certainly evolved, our UI has not.  We still use tables as the primary (often only) UI element almost everywhere, there are very few graphical elements outside of the Reporting UI (which itself uses only basic strip-charts), and the UI simply conveys far less information, in a far less dynamic fashion, to the user than anyone would wish.

Architectural overview of FreeNAS 10:

Figure4(figure 4)

It was actually necessary to simplify this picture significantly in order to make for a more comprehensible overview.  There are quite a few new moving parts in FreeNAS 10’s design, including a number of open source technologies adapted from Apple’s OS X operating system, which cooperate to create a thoroughly asynchronous world where things can, and do, happen at any time.  As an architectural overview, however, this picture should suffice in giving you the general idea.

What may not be obvious from this diagram is the fact that there can now be multiple web UIs (as well as CLI sessions) attached to the single Middleware Server process which is now always running and constitutes the gateway for all things happening on FreeNAS 10.

Unlike FreeNAS 9, where it is also easily possible to “sneak behind the middleware’s back” and do things on the command line or from other GUI sessions that are subsequently invisible (and potentially harmful) to it, the middleware in FreeNAS 10 knows essentially everything.  Even operations done directly at the CLI layer, say from a remote ssh session, are communicated up from the FreeBSD 10 kernel in the form of notifications to the middleware, allowing it to react and update the relevant UI accordingly.

As you can see, the oval denoting the web browser is much bigger in FreeNAS 10.  That is not simply a graphical accident but the result of the web UI client being much thicker,  the FreeNAS 10 GUI basically being much more of a web application than the simple UI in FreeNAS 9.  Far more sophisticated technologies like ReactFlux and D3 are going into the FreeNAS 10 UI in an effort to create something which is far more dynamic and informative to the user while, hopefully, also being far easier to use and prettier to look at!

Previously mentioned in passing, but far from insignificant, is also the use of a number of Apple OS X technologies like launchd, notifyd, ASL, and libdispatch, all of which are being used to deal with process management, asynchronous notifications of various important kernel, system and application events, and creating far more structured and machine-parseable log events.  Someone not familiar with any or all of those technologies (though any developer owning a Mac or an iPhone probably is) needs simply to know that they are being used to create a far more IPC and service-centric architecture, where the middleware is far more aware of the actions of services or commands operating on its behalf.

As we have re-architected FreeNAS, we have also adopted a somewhat more rigorous approach to creating developer documentation and designing our internal interfaces such that they are more self-describing and can also be dynamically introspected.  Everything about FreeNAS 10 was designed to be more developer-friendly as well as more user-friendly, in other words, because it is developers who will eventually drive the creation and refinement of features which directly benefit the users.  While all of this is a work in progress, we have already made substantial improvements to the developer experience in comparison to FreeNAS 9.3.

In no particular order, and obviously subject to change as FreeNAS 10 continues to evolve rapidly, here are some (but by no means all) of the features that are making FreeNAS 10 far friendlier to developers:

  • It is now possible to do release engineering (e.g. build complete, installable releases of FreeNAS 10) separately from doing GUI development . This means a GUI developer can do live development against a running FreeNAS 10 machine with nothing more than an ssh connection and some client-side tools, which we have of course documented.  This is in stark contrast to the FreeNAS 9 UI development process, which is basically really hard.
  • The build tools for FreeNAS 10 itself have been broken out into a separate repository, freenas-build, which does the grunt work of checking out all of the other repositories required during the build process, allowing for an easy and intuitive “bootstrap” of a complete FreeNAS 10 build.
  • The FreeNAS 10 UI now features a built-in debugger which allows developers to observe the progress of background tasks, interrogate the middleware directly, and do a host of other useful things when something has gone sideways and they need a bit more information.  Everything going to and from the middleware also consists of JSON objects, so introspection of the fundamental IPC is easy.
  • There is now a real FreeNAS CLI, with built-in help and tab completion, for driving the middleware server as an alternative to the GUI.  The CLI is also designed to be able to run remotely, on any platform with a python interpreter, allowing for remote debugging and administration where a full web UI might be impractical or simply inefficient (this is not simply a developer feature, but also essentially a power user feature).
  • The middleware documents its APIs using a tiny web server of its own (this may eventually be folded into the primary Node.js server) listening on port 8180 – this can also be accessed using the API Docs button in the UI.

We are essentially trying particularly hard to promote and publicise the open source development process from the very beginning with FreeNAS 10, and not just to encourage external developers to take an interest in and join the project but to also make on-boarding our own new hires that much easier!  Some of the key resources any developer will need to know about are, and again in no particular order:

  • FreeNAS 10 development is currently taking place on two repositories on github.  One is the freenas10/development branch, which is being used to do all of the GUI and middleware development work.  The other is our freebsd10 branch of TrueOS, our internal code name for the special version of FreeBSD 10.1 we are basing FreeNAS 10 on.  Both can be built on a suitable FreeBSD 10 or FreeBSD 11 machine (or VM) and there are instructions for doing so provided in the repositories themselves.  Anyone wishing to contribute to either repository can simply fork the relevant repo(s) and send us pull requests!
  • All of the tickets tracking the work to be done (as we have currently broken it down, at least) are in the freenas 10 bug tracker.  Just as with FreeNAS 9’s bug tracker, anyone can sign up for an account and get access to it.  We highly encourage people to contribute ideas in the form of new tickets, or comments on existing ones, as this project evolves since we are highly driven by what is in the bug tracker.
  • The UI for FreeNAS 10 is still very very green, but what UI mockups we have are all on github (just follow the link). Please feel free to comment – they’re just mockups and absolutely nothing is frozen in stone. Even after the initial UIs are implemented, we’re still very much open to suggestions and ideas since we certainly don’t expect the first iteration of anything we do to be perfect, and the UI in FreeNAS 10 is far easier to hack on and change (or even, hopefully soon, customize on a per-user basis).
  • No one would call FreeNAS 10 currently even close to usable – we’re still ramping up the initial development team on the project as I write this – but we are releasing nightly builds of our progress regardless, just so developers can see the progress in the most tangible possible way.

FreeNAS 10 development is also being organized into 5 milestones and we are currently nearing the end of Milestone 2.  This is why you will see sub-directories (and target versions in the bug tracker) like “M1″, “M2″ and so on.  The first public demo will be at the end of M2, which is currently scheduled for August 15th, 2015.  It’s still very early in our development cycle, in other words, and therefore an excellent time to jump on board as a developer, especially if you are highly opinionated and want to have the opportunity to influence the direction of this project!

Daily conversations about FreeNAS 10 development also occur on the #freenas10 IRC channel on irc.ixsystems.com.  If you are an active FreeNAS 10 developer, please feel more than welcome to join that channel and tell us what you’re working on!   Those who are more on the end-user side of things and interested in FreeNAS 10 from a bigger picture perspective should join the #freenas channel on the FreeNode IRC network – FreeNAS’s developers also tend to hang out there.

This has by no means been a complete overview of all of the various cool technologies we are bringing to FreeNAS 10 (I did not even have the opportunity to talk about the future of plugins in the form of AppCafe, or bhyve Virtual Machine support, or the new iocage based jail infrastructure, or S3 compatible object storage, or or or – so many other things!), but I hope that it has given you at least some insight into the very early days of FreeNAS 10’s development process and where to get the bits, in both source and binary form.  We are very excited by FreeNAS 10 as well as having a lot of fun working on it – we hope you can join us!

– Jordan