FreeNAS 10: A developer’s perspective

Written by jkh on .

One of the greater challenges in software engineering, particularly once you’ve actually released your latest magnum opus to the world, is deciding what to do next.  How long do you continue to support version X, carefully fixing bugs and polishing the rock, before the list of architectural limitations and # of tickets requesting radical, substantial new features (all of which are potentially disruptive) begin to suggest that simple rock-polishing isn’t going to cut it as your only investment in engineering time and resources?

We reached that tipping point with the FreeNAS 9.x branch about 6 months ago.  While putting the finishing touches on FreeNAS 9.3, a release which was a substantial improvement over previous 9.x releases, it was clear that we were still fundamentally limited by various design decisions made during the 2nd major rewrite of FreeNAS (FreeNAS 8).

This is not to say that those decisions were wrong or bad – all software technologies evolve and hindsight is always 20-20 – but it’s still fair to say that the world is constantly changing and FreeNAS as a product has little choice but to change with it.

First, just to set the context for the description of FreeNAS 10 which follows, a quick Architectural overview of FreeNAS 9 is in order:

Figure1(figure 1)

The observant will quickly notice that the box marked “Middleware” is fairly large on that diagram.  This is because it is, at least from a conceptual standpoint!   In FreeNAS 9, the middleware is a monolithic entity that contains the javascript necessary to create the UI in the browser, the Django Web framework (a Model-View-Controller architecture used to view and modify data in the sqlite database), and all of the scaffolding necessary for talking directly to the underlying FreeBSD OS and a small constellation of helper tools (samba and netatalk being some of the most prominent, as they are used to implement CIFS and AFP file sharing) from the FreeBSD ports collection.

The biggest issue with the middleware, however, isn’t even the fact that it is a monolithic chunk of code with many categories of functionality mixed together with no clear separation of concerns or recognizable abstraction layers. The biggest issue is that the middleware implementation is synchronous, meaning that it can only do one UI-facing thing at a time and must wait for long-running operations to finish before the user can do anything else.   Being monolithic, the middleware also does not exist as a separate process outside the context of the current (and, in fact, only) UI session, the implications of which will become more obvious as you read on.

Let’s start with the most obvious impact of being synchronous, however.  Let’s say the user navigates to the Plugins UI, clicks on the Installed tab, and then tries to upgrade one of their plugins.  They will get the following behavior:

Figure2(figure 2)

As you can see, the UI is fully blocked while it downloads this plugin update.  There isn’t even an option to cancel the operation because the middleware is busy downloading the plugin and no longer truly interacting with the UI.  Should the user get bored and simply choose to navigate away from this page, the results are essentially undefined.

In some areas of the FreeNAS 9 UI, unexpectedly navigating away or closing the browser during a long-running operation can even leave the system in an inconsistent state.  This is not particularly robust behavior, to say the least, all again the result of having a single, highly-privileged middleware instance that both provides the UI and the underlying code for manipulating the state of the underlying machine as part of the user’s session.

As mentioned earlier, there can also only be one such session, which is to say that only one user can be logged into the box and make changes at a time.  What happens if two users log in to the same FreeNAS box and start making changes simultaneously?  Nothing good, in all likelyhood, because each middleware instance thinks that it is the only one (in true Highlander fashion), and has no awareness of the actions of the other.  This lack of overall global awareness, or any persistent management of the system after the user logs out, has other implications for the middleware and for the overall User Experience of FreeNAS 9.  Again, let’s use the existing UI as an illustration:

Figure3(figure 3)

Here we see 3 S.M.A.R.T. tests scheduled at different intervals using the Tasks UI.  Because the middleware may not be around to schedule these tasks (or, for that matter, any Cron jobs or Rsync tasks that the user might schedule), it has to be done by some other agent which has no direct relationship with the middleware.  In this case, that agent is the smartd daemon, the middleware itself (and configuration database) merely tracking the fact that those tests need to be scheduled with it.

This all works well enough, but should one of those S.M.A.R.T. tests fail or be otherwise deemed invalid by smartd, there is absolutely no way for the user to tell from the UI session because the middleware itself has no awareness of this, just as it has no awareness of the success or failure of rsync or cron tasks it schedules.  The middleware is essentially trusting that things it has asked other parts of the system to do on its behalf are functioning as expected.

Again, the middleware was basically and fundamentally not designed to deal with asynchronous events, and the tendency has therefore been to simply not even try to display real-time sources of data other than the most basic alert status and progress information, all of which has to be polled for in fairly painful fashion.   This has impacted the quality of the UI feedback that FreeNAS 9 is able, or even attempts, to provide in countless different ways and is one of the more fundamental shortcomings of the current user experience.

This blog post is supposed to be about FreeNAS 10 and not FreeNAS 9, however, so let’s just conclude this history lesson with one last point about the UI technology itself:  The technologies we used to build the 9.3 UI are definitely showing their age, and while toolkits like Dojo have certainly evolved, our UI has not.  We still use tables as the primary (often only) UI element almost everywhere, there are very few graphical elements outside of the Reporting UI (which itself uses only basic strip-charts), and the UI simply conveys far less information, in a far less dynamic fashion, to the user than anyone would wish.

Architectural overview of FreeNAS 10:

Figure4(figure 4)

It was actually necessary to simplify this picture significantly in order to make for a more comprehensible overview.  There are quite a few new moving parts in FreeNAS 10’s design, including a number of open source technologies adapted from Apple’s OS X operating system, which cooperate to create a thoroughly asynchronous world where things can, and do, happen at any time.  As an architectural overview, however, this picture should suffice in giving you the general idea.

What may not be obvious from this diagram is the fact that there can now be multiple web UIs (as well as CLI sessions) attached to the single Middleware Server process which is now always running and constitutes the gateway for all things happening on FreeNAS 10.

Unlike FreeNAS 9, where it is also easily possible to “sneak behind the middleware’s back” and do things on the command line or from other GUI sessions that are subsequently invisible (and potentially harmful) to it, the middleware in FreeNAS 10 knows essentially everything.  Even operations done directly at the CLI layer, say from a remote ssh session, are communicated up from the FreeBSD 10 kernel in the form of notifications to the middleware, allowing it to react and update the relevant UI accordingly.

As you can see, the oval denoting the web browser is much bigger in FreeNAS 10.  That is not simply a graphical accident but the result of the web UI client being much thicker,  the FreeNAS 10 GUI basically being much more of a web application than the simple UI in FreeNAS 9.  Far more sophisticated technologies like ReactFlux and D3 are going into the FreeNAS 10 UI in an effort to create something which is far more dynamic and informative to the user while, hopefully, also being far easier to use and prettier to look at!

Previously mentioned in passing, but far from insignificant, is also the use of a number of Apple OS X technologies like launchd, notifyd, ASL, and libdispatch, all of which are being used to deal with process management, asynchronous notifications of various important kernel, system and application events, and creating far more structured and machine-parseable log events.  Someone not familiar with any or all of those technologies (though any developer owning a Mac or an iPhone probably is) needs simply to know that they are being used to create a far more IPC and service-centric architecture, where the middleware is far more aware of the actions of services or commands operating on its behalf.

As we have re-architected FreeNAS, we have also adopted a somewhat more rigorous approach to creating developer documentation and designing our internal interfaces such that they are more self-describing and can also be dynamically introspected.  Everything about FreeNAS 10 was designed to be more developer-friendly as well as more user-friendly, in other words, because it is developers who will eventually drive the creation and refinement of features which directly benefit the users.  While all of this is a work in progress, we have already made substantial improvements to the developer experience in comparison to FreeNAS 9.3.

In no particular order, and obviously subject to change as FreeNAS 10 continues to evolve rapidly, here are some (but by no means all) of the features that are making FreeNAS 10 far friendlier to developers:

  • It is now possible to do release engineering (e.g. build complete, installable releases of FreeNAS 10) separately from doing GUI development . This means a GUI developer can do live development against a running FreeNAS 10 machine with nothing more than an ssh connection and some client-side tools, which we have of course documented.  This is in stark contrast to the FreeNAS 9 UI development process, which is basically really hard.
  • The build tools for FreeNAS 10 itself have been broken out into a separate repository, freenas-build, which does the grunt work of checking out all of the other repositories required during the build process, allowing for an easy and intuitive “bootstrap” of a complete FreeNAS 10 build.
  • The FreeNAS 10 UI now features a built-in debugger which allows developers to observe the progress of background tasks, interrogate the middleware directly, and do a host of other useful things when something has gone sideways and they need a bit more information.  Everything going to and from the middleware also consists of JSON objects, so introspection of the fundamental IPC is easy.
  • There is now a real FreeNAS CLI, with built-in help and tab completion, for driving the middleware server as an alternative to the GUI.  The CLI is also designed to be able to run remotely, on any platform with a python interpreter, allowing for remote debugging and administration where a full web UI might be impractical or simply inefficient (this is not simply a developer feature, but also essentially a power user feature).
  • The middleware documents its APIs using a tiny web server of its own (this may eventually be folded into the primary Node.js server) listening on port 8180 – this can also be accessed using the API Docs button in the UI.

We are essentially trying particularly hard to promote and publicise the open source development process from the very beginning with FreeNAS 10, and not just to encourage external developers to take an interest in and join the project but to also make on-boarding our own new hires that much easier!  Some of the key resources any developer will need to know about are, and again in no particular order:

  • FreeNAS 10 development is currently taking place on two repositories on github.  One is the freenas10/development branch, which is being used to do all of the GUI and middleware development work.  The other is our freebsd10 branch of TrueOS, our internal code name for the special version of FreeBSD 10.1 we are basing FreeNAS 10 on.  Both can be built on a suitable FreeBSD 10 or FreeBSD 11 machine (or VM) and there are instructions for doing so provided in the repositories themselves.  Anyone wishing to contribute to either repository can simply fork the relevant repo(s) and send us pull requests!
  • All of the tickets tracking the work to be done (as we have currently broken it down, at least) are in the freenas 10 bug tracker.  Just as with FreeNAS 9’s bug tracker, anyone can sign up for an account and get access to it.  We highly encourage people to contribute ideas in the form of new tickets, or comments on existing ones, as this project evolves since we are highly driven by what is in the bug tracker.
  • The UI for FreeNAS 10 is still very very green, but what UI mockups we have are all on github (just follow the link). Please feel free to comment – they’re just mockups and absolutely nothing is frozen in stone. Even after the initial UIs are implemented, we’re still very much open to suggestions and ideas since we certainly don’t expect the first iteration of anything we do to be perfect, and the UI in FreeNAS 10 is far easier to hack on and change (or even, hopefully soon, customize on a per-user basis).
  • No one would call FreeNAS 10 currently even close to usable – we’re still ramping up the initial development team on the project as I write this – but we are releasing nightly builds of our progress regardless, just so developers can see the progress in the most tangible possible way.

FreeNAS 10 development is also being organized into 5 milestones and we are currently nearing the end of Milestone 2.  This is why you will see sub-directories (and target versions in the bug tracker) like “M1″, “M2″ and so on.  The first public demo will be at the end of M2, which is currently scheduled for August 15th, 2015.  It’s still very early in our development cycle, in other words, and therefore an excellent time to jump on board as a developer, especially if you are highly opinionated and want to have the opportunity to influence the direction of this project!

Daily conversations about FreeNAS 10 development also occur on the #freenas10 IRC channel on irc.ixsystems.com.  If you are an active FreeNAS 10 developer, please feel more than welcome to join that channel and tell us what you’re working on!   Those who are more on the end-user side of things and interested in FreeNAS 10 from a bigger picture perspective should join the #freenas channel on the FreeNode IRC network – FreeNAS’s developers also tend to hang out there.

This has by no means been a complete overview of all of the various cool technologies we are bringing to FreeNAS 10 (I did not even have the opportunity to talk about the future of plugins in the form of AppCafe, or bhyve Virtual Machine support, or the new iocage based jail infrastructure, or S3 compatible object storage, or or or – so many other things!), but I hope that it has given you at least some insight into the very early days of FreeNAS 10’s development process and where to get the bits, in both source and binary form.  We are very excited by FreeNAS 10 as well as having a lot of fun working on it – we hope you can join us!

– Jordan

FreeNAS in Production

Written by Annie Zhang on .

I work as a Network Administrator in a shop where we deploy 60+ ESXi VMs on a half dozen host servers.  We heard about FreeNAS from my predecessor who implemented it where he is now employed.  My boss asked me to evaluate FreeNAS as it is time to replace one of our existing SANs.  I installed FreeNAS 9.3 on an old IBM server and took the four 3-hour FreeNAS classes from Linda Kateley. I then dove in head first with a new 4U SuperMicro SAN with dual Intel CPUs, 128GB EEC RAM, twenty 4TB SAS HDDs and a handful of consumer grade SSDs.  The important lessons learned are:

  1. Buy big RAM chips so you have available slots if you need more RAM.
  2. eMLC SSDs cost more but they are definitely better than off the shelf SSDs.
  3. There is a 4X & 5X FreeNAS rule that states you want 4 or 5 times as many gigabytes of SSD capacity for cache as you have gigabytes of RAM.  I always go big so we deployed the 8X rule.
  4. Download FreeNAS and install it on a test platform and then take Linda Kateley’s four paid and one free interactive online class to get started.  The FreeNAS forum is wonderful as long as you research your questions before posting.
  5. RAID cards are not needed but if you want an LSI RAID card make sure you flash it to IT (pass through mode).  LSI support & the FreeNAS forum were great in guiding me to the correct utilities I needed for my LSI SAS 9211-8i Host Bus Adapter.
  6. If you want the lights to work correctly on the front of a SuperMicro SAN use only SAS HDDs.

A very knowledgeable admin on the FreeNAS forum sent me a quote that is quite appropriate:  “It’s like a learning cliff.  We know.”  That said, Linda’s class made it much more like a steep hill and very manageable.
Currently I have the fastest SAN I have ever had the pleasure of working with and all the hard work pays off when you see you have maxed out your state of the art fiber gigabit network.

Dale Josephson
Network Administrator
Karuk Tribe

My FreeNAS Lab

Written by Casey on .

My FreeNAS Lab

I started using FreeNAS a couple of years ago at work when we started looking into backup storage solutions for our various (pricey) production arrays. I first experimented with 5-6 year old spare desktop computers of the Intel Core 2 CPU/DDR2 memory vintage and whatever hard drives that we had laying around. We were so impressed with FreeNAS’s versatility (CIFS, AFP, iSCSI) and performance that we decided to purchase enterprise class hardware for a custom built production backup storage solution using FreeNAS. It ended up costing about $10K for 40 TB initially and we have since added another 40 TB shelf to the same server as our storage needs have grown.

I have also been able to upgrade the hardware I use for my FreeNAS/VMware test lab and it is about this that I will give a more detailed account. I have two FreeNAS boxes, 1 to host VMware datastores over iSCSI and the other to receive backups for the first system and for CIFS and AFP (time machine) shares.

System 1

System 1

Motherboard – Asus P8Z68-V LE

CPU – Intel Core i5 2500K 3.30 GHz quad core

Memory – 32 GB G.SKILL Ripjaws X Series DDR3 1600 PC3 12800 (4 x 8GB)

Hard Drives – 6 x 600GB WD VelociRaptor

iSCSI NIC – 4-port gigabit Intel E1G44HT PCI-Express 2.0 I340-T4

OS drive – SanDisk Cruzer Fit 8 GB

Build – FreeNAS-9.3-STABLE-201412312006

Volume – RAIDZ2 2.2 TB available

System 2

system 2

Motherboard – Asus P7P55D-E LX

CPU – Intel Core i5 760 2.80 GHz quad core

Memory – 16 GB G.SKILL Ripjaws X Series DDR3 1600 PC3 12800 (2 x 8GB)

Hard Drives – 8 x 2 TB Hitachi Deskstar

OS drive – SanDisk Cruzer Fit 8 GB

Build – FreeNAS-9.3-STABLE-201412312006

Volume – RAIDZ2 10 TB available

Performance

I know that iXsystems doesn’t like to post performance numbers of their TrueNAS systems, but I’m going to post my numbers.  I do it mainly as a point of comparison because I recently swapped out the 4-port gigabit NIC for a 10 Gbps Intel X540-T1. Here are the CrystalDiskMark numbers inside a Windows guest on an ESXi 5.1 iSCSI datastore:

(MB/s – IOPS) 4 x 1 Gbps 10 Gbps
Read Seq 103 538
Write Seq 100 864
Read 512K 94 576
Write 512K 92 760
Read 4K 10 – 2,352 14 – 3,340
Write 4K 14 – 2,683 30 –7,226
Read 4K QD32 111 – 27,184 282 – 68,823
Write 4K QD32 88 – 21.678 394 – 96,097

There’s a 1 Gbps bottleneck and it must be because the ESXi free license doesn’t have MPIO support. Going to 10 Gbps simplifies networking and has a huge performance increase; I highly recommend it.

 

Conclusion

FreeNAS is awesome!!! The more I use it, the more confident I am in its stability, performance, and ability to meet all my needs. I’m looking forward to FreeNAS 10 and you can be sure I’ll upgrade.

Thanks FreeNAS community and iXsystems.

     -David L. DevOps Engineer

     Imagine Learning Provo, UT