Yes, You Can Virtualize FreeNAS

Written by Joshua Paetzel on .

FreeNAS is the world’s most popular open source storage OS, and one of the more popular questions I get asked is, “How do I run FreeNAS as a VM?” Due to the number of caveats required to answer that question, I would typically short-circuit the conversation by recommending against it, or only recommend it for test environments since the prerequisite knowledge required to “do it right” can’t be passed on quickly. Somehow over time, this message morphed into a general consensus that “you cannot (or shouldn’t) virtualize FreeNAS at all under any circumstances”, which wasn’t my intention. So, I’m here to set the record straight once and for all: You absolutely can virtualize FreeNAS.

virtualization

Whether you are test driving the functionality of FreeNAS, testing an upgrade for compatibility in your environment, or you want to insulate your FreeNAS system from hardware faults, virtualization can provide many well understood benefits. That said, while FreeNAS can and will run as a virtual machine, it’s definitely not ideal for every use case. If you do choose to run FreeNAS under virtualization, there are some caveats and precautions that must be considered and implemented. In this post I’ll describe what they are so that you can make well-informed choices.

Before we get started though, I should probably start with a disclaimer…

Warning

If best practices and recommendations for running FreeNAS under virtualization are followed, FreeNAS and virtualization can be smooth sailing. However, failure to adhere to the recommendations and best practices below can result in catastrophic loss of your ZFS pool (and/or data) without warning. Please read through them and take heed.

Ok, phew. Now that that’s over with, let’s get started.

1. Pick a virtualization platform

When developing FreeNAS we run it as a VM. Our virtualization platform of choice is VMware, and it’s the platform in which the FreeNAS developers have the most experience. FreeNAS includes VMware tools as well.

virtualization3

Our second choice for a virtualization platform is CItrix XenServer. FreeNAS has no tools built in for XenServer, but you get a solid virtualization experience nonetheless. Other hypervisors such as bhyve, KVM, and Hyper-V also work, but the development team does not use them on a daily basis.

2. Virtualizing ZFS

ZFS combines the roles of RAID controller, Volume Manager, and file system, and since it’s all three in one, it wants direct access to your disks in order to work properly. The closer you can get ZFS to your storage hardware, the happier ZFS is, and the better it can do its job of keeping your data safe. Things like native virtual disks or virtual disks on RAID controllers insulate ZFS from the disks, and therefore should be avoided whenever possible. Using a hypervisor, you typically have a disk on a RAID controller presented to a hypervisor which creates a datastore with a disk on it running FreeNAS. This places two layers between ZFS and the physical disks which warrants taking the following precautions.

Precautions

  1. If you are not using PCI passthrough (more on that below), then you must disable the scrub tasks in ZFS. The hardware can “lie” to ZFS so a scrub can do more damage than good, possibly even permanently destroying your zpool.
  2. The second precaution is to disable any write caching that is happening on the SAN, NAS, or RAID controller itself. A write cache can easily confuse ZFS about what has or has not been written to disk. This confusion can result in catastrophic pool failures.
  3. Using a single disk leaves you vulnerable to pool metadata corruption which could cause the loss of the pool. To avoid this, you need a minimum of three vdevs, either striped or in a RAIDZ configuration. Since ZFS pool metadata is mirrored between three vdevs if they are available, using a minimum of three vdevs to build your pool is safer than a single vdev. Ideally vdevs that have their own redundancy are preferred.

3. Consider the Use Case

Is this a production or non-production FreeNAS application? The answer to this question has significant implications to the subsequent recommended practices.

Non-Production

If your use case is a test lab, science experiment, pre-upgrade checks of a new version, or any other situation where real data that you care about isn’t at stake, go ahead and virtualize. Create a VM with 8GB of RAM, two vCPUs, a 16GB install disk, and a single data disk of whatever size is appropriate for your testing, and go to town.

Production

This is where things get serious. If you’re using FreeNAS in an application that’s relied on for daily operations, this is considered a “Production Environment”, and additional precautions must be followed closely to avoid downtime or data loss.

If you use PCI passthrough (aka DirectPath I/O), you can then use FreeNAS just like it’s installed on physical hardware. The PCI device passthrough capability allows a physical PCI device from the host machine to be assigned directly to a VM. The VM’s drivers can use the device hardware directly without relying on any driver capabilities from the host OS. These VMware features are unavailable for VMs that use PCI passthrough:

  • Hot adding and removing of virtual devices
  • Suspend and resume
  • Record and replay
  • Fault tolerance
  • High availability
  • DRS
  • Snapshots

To use PCI passthrough, you need to use a host bus adapter (HBA) supported by FreeNAS (we recommend LSI controllers of the 2008 chipset variety, which are 6Gb/s and well supported in FreeNAS) as a PCI passthrough device that is connected directly to your FreeNAS VM. The FreeNAS VM then has direct access to the disks. Make sure to adhere to the guidelines on using PCI passthrough. If you use PCI passthrough, it is as if you aren’t virtualizing at all so you’ll be safe to use FreeNAS in a production scenario.

4. Other Considerations

If you are still interested in virtualizing FreeNAS, pay attention to the following:

Virtualization Requirements

Adhere to the FreeNAS hardware recommendations when allocating to your FreeNAS VM. It goes without saying that virtualized FreeNAS is still subject to the same RAM and CPU requirements as a physical machine. When you virtualize FreeNAS your VM will need:

    • At least two vCPUs
    • 8GB or more of vRAM, at least 12GB of vRAM if you use jails/plugins
    • Two or more vDisks
    • A vDisk at least 16GB in size for the OS and boot environments
    • One or more vDisks at least 4GB in size for data storage, at least 3 are recommended
    • A bridged network adapter

Striping your vDisks

In this configuration, ZFS will be unable to repair data corruption but it will be resilient against pool corruption caused by damage to critical pool data structures causing loss of the entire pool. If you are using a SAN/NAS to provision the vDisk space, then three striped 1TB drives will require 3TB of external LUN usable space.

RAIDZ protection of your vDisks

In this configuration, ZFS will repair data corruption. However, you will waste an additional virtual disk worth (or two if RAIDZ2 is used) of space since the external storage array protects the LUN and RAIDZ creates parity to protect each vDisk. If you are using a SAN/NAS to provision the vDisk space, then three RAIDZ1 1TB drives will require 4.5TB of external LUN usable space.

Disk space needed for provisioning

With striping, you’ll be required to provision 3TB of space from the SAN/NAS storage array to get 3TB of usable space. If you use RAIDZ protection, it will use one of the virtual disks for parity, and you will be required to provision 4.5 TB of space from the SAN/NAS storage array to get 3TB of usable space. Depending on the $/GB for your SAN/NAS this additional 1,500 GB can get quite expensive.

TL;DR Summary

I have attempted to share some best practices that the engineering team at iXsystems has used while virtualizing, and I hope that I have not missed anything big. With so many different hypervisors, it is difficult to give you specific instructions. You need to take some precautions to utilize your setup in a production environment safely:

      • PCI passthrough of an HBA: This is the best case and ideally recommended
      • If using a RAID controller/SAN/NAS, Write cache: Disabled
      • FreeNAS scrub tasks: Disabled unless PCI passthrough is used
      • Disk configuration
        • Single disk: Vulnerable to pool metadata corruption, which could cause the loss of the pool. Can detect — but cannot repair — user data corruption.
        • Three or more virtual disks striped (even if they are from the same datastore!): Resilient against pool corruption. Can detect — but cannot repair — corrupted data in the pool. Depending on what backs the vDisks you may be able to survive a physical disk failure, but it is unlikely that the pool will survive.
        • Three or more virtual disks in RAIDZ: Can detect and repair data corruption in the pool, assuming the underlying datastore and/or disks are functional enough to permit repairing by ZFS’ self-healing technology.
        • Never ever run a scrub from FreeNAS when a patrol read, consistency check, or any other sort of underlying volume repair operation, such as a rebuild, is in progress.

Some other tips if you get stuck:

      • Search the FreeNAS Manual for your version of FreeNAS. Most questions are already answered in the documentation.
      • Before you ask for help on a virtualization issue, always search the forums first. Your specific issue may have already been resolved.
      • If using a web search engine, include the term “FreeNAS” and your version number.

As an open source community, FreeNAS relies on the input and expertise of its users to help improve it. Take some time to assist the community; your contributions benefit everyone who uses FreeNAS.

To sum up: virtualizing FreeNAS is great—the engineering organization and I have used it that way for many years, and we have several VMs running in production at iXsystems. I attempted to provide accurate and helpful advice in this post and as long as you follow my guidance, your system should work fine. If not, feel free to let me know. I’d love to hear from you.

Josh Paetzel
iXsystems Senior Engineer

What’s Improved in FreeNAS 9.3-BETA | State of the Union Video | In-depth OwnCloud Tutorial Video

Written by Annie Zhang on .

Hello FreeNAS Users,

This time DCIG ranked both our TrueNAS Z20 and Z30 arrays “Excellent” and “Best-in-Class” in hardware in the newest guide in their series, the “Midsize Enterprise Hybrid Storage Buyer’s Guide”, putting it ahead of competitive offerings from EMC, Hitachi, HP, Dell EqualLogic, Nimble Storage, Fujitsu, and others.
Also by popular request, we have a guide for running FreeNAS as a VM and a tutorial demonstrating the setup wizard introduced in FreeNAS 9.3.

Cheers,
The FreeNAS Team

DCIG Ranks TrueNAS Excellent Again
Good news – our TrueNAS Z20 and TrueNAS Z30 were ranked “Excellent” by DCIG in their 2015-2016 DCIG Midsize-Enterprise Hybrid Storage Buyer’s Guide. Twenty-seven different products were evaluated for over 90 different criteria. Both of our systems were awarded “Excellent” for software and the Z30 was named “Best in Class” for hardware. Read More>>
6 Reasons Why TrueNAS is replacing NetApp and EMC – Free Webinar
We invite you to join Matt Olander, Co-Founder of iXsystems, in a free webinar about TrueNAS. Find out why people are making the switch from big-name, legacy storage vendors to TrueNAS. Read more >>

FreeBSD Journal
Yes, You Can Virtualize FreeNAS
Running FreeNAS as a virtual machine is complicated so it’s not something we generally recommend. However, it is possible as long as you take the right precautions. Josh Paetzel, FreeNAS developer and iXsystems Senior Engineer, explains the steps you need to take to safely virtualize FreeNAS in this blog post.
FreeNAS 9.3 Setup Wizard Tutorial
In our newest video tutorial, we demonstrate how to use the initial setup wizard to configure a volume and set up a share. Check it out>>
FreeNAS at Linuxfest Northwest
The FreeNAS team is back from this year’s LFNW with a recap of the events and photos from Day 1 & Day 2 of the event.
FreeNAS Certification Classes
We offer a free Intro to FreeNAS class that runs every day. For those of you interested in learning more about advanced topics, we also offer paid, fully interactive classes. Read more >>
Live Events

TechTip #17
FreeNAS 9.3 jails don’t just support plugins, they also support FreeBSD packages in general.
Join the Team
iXsystems, the company that sponsors FreeNAS, is looking for a few good people to join our team. Interested? The full list of available positions can be found on our website.
Links of the Month

The History (and Future) of FreeNAS & TrueNAS

Written by Annie Zhang on .

The FreeNAS project got its start way back in 2005, when Olivier Cochard-Labbé wanted to turn his old PC into a home server. There wasn’t an open source project that fit all of his needs, so he did what any self-respecting software developer would do: he sat down and wrote his own. Just like that, the software that would eventually become the world’s most popular open source software defined storage was born.

History of FreeNAS

Development continued until 2009, when one of the project developers proposed moving FreeNAS to a Debian Linux based system. This move would have meant losing access to the FreeBSD community and the overall quality of its software, and FreeNAS would also lose its native ZFS support, since the ZFS On Linux project didn’t even exist at that time. iXsystems had used FreeNAS for many years and sold servers specifically made for FreeNAS, so Matt Olander, one of the iXsystems founders, reached out to Olivier and offered to take over FreeNAS development on FreeBSD. Olivier gave his blessing, and iXsystems started immediately working on FreeNAS 8. In order to modernize FreeNAS, the development team at iXsystems rewrote almost all the code and replaced the m0n0wall PHP code with a full featured, easy-to-use webGUI.

From there, iXsystems continued to improve the FreeNAS code and add new features. A new plugin system was introduced in FreeNAS 8.3 that allows users to install ports, packages and PBI’s and extend the use of the system. In March 2013, with the release of 8.3.1, FreeNAS became the first and only open source storage project to offer encryption with ZFS. In the most recent 9.3 release, we redesigned the UI again, moved to ZFS completely, and added the ability to boot from multiple boot environments and roll-back updates or other configuration changes. It also added the ability to automatically check for updates, added support for NFSv4, supports booting from multiple boot environments, and makes it easier to roll-back updates or apply other configuration changes.

There’s a smooth continuum between FreeNAS development and our for-profit work. The time we put into open source supports projects around the globe. It also supports the company because the software is directly incorporated into our storage offerings. The feedback and bug reports we get from the FreeNAS open source community allow us to refine our software and deliver that refinement to our customers – other storage companies only wish they had access to the QA resources we do!

The first generation of the TrueNAS storage appliance launched in August 2011 and, since then, we have launched the second generation of TrueNAS appliances. In late 2014, we also unveiled our all-flash TrueFlash system.

There are very big things planned for the future of FreeNAS. FreeNAS 10 will feature a dramatic overhaul of the user interface, completely rewritten underpinnings, and a re-base on FreeBSD 10.x. Overall, it will be far more responsive and intuitive to use, feature many more Enterprise level storage features, and offer greatly enhanced reporting and monitoring options. We think long-time FreeNAS users will be quite impressed by it.

We are also releasing frequent incremental updates for FreeNAS 9.3 while the community waits for FreeNAS 10 to hit its initial release milestones in 9-12 months. We promise the wait will be worth it!

Jordan Hubbard
FreeNAS Project Manager and iXsystems CTO