Posts Tagged ‘FreeNAS’

FreeNAS: A Worst Practices Guide

Written by Mark VonFange on .

FreeNAS Worst Practices2

There are many best practices guides for managing storage solutions out there, but a lot of how you administer your storage depends on your specific use case and what you’re trying to accomplish. While we have created a best practices for FreeNAS, we also decided to take a look at what you don’t want to do; things that will leave you hurting either immediately or down the road.

In that spirit, we’ve put together a worst practices guide for FreeNAS based on years of experience with systems in the field. The easiest way to avoid these pitfalls is to simply purchase a TrueNAS system from the experts at iXsystems, who can help set up your systems for optimal performance and functionality. For those who prefer the DIY approach, here are some things to look out for when setting up and managing your own FreeNAS system.

Using Hardware RAID with ZFS

When setting up a RAID array, common knowledge says that hardware RAID is preferable to software RAID. This is something of a misconception as all RAID is software RAID. If you’re using a hardware RAID controller, it has its own independent operating system that communicates with your disks and often has caches to improve read and write performance. This was a good idea in the distant past, and improved RAID performance substantially, but operating systems and the hardware they run on have come a long way since those days.

FreeNAS uses the ZFS file system and is designed to communicate directly with your disks using its own volume manager. ZFS includes a sophisticated yet efficient strategy for providing various levels of data redundancy, including the mirroring of disk and the “ZFS” equivalents of hardware RAID 5 and higher with the ability of losing up to three disks in an array. If a given set of disks is provided to ZFS using a hardware RAID card, ZFS will not be able to efficiently balance its reads and writes between them or rebuild only the data used by any given disk. Hardware RAID cards typically rebuild disks in a linear manner from beginning to end without any regard for their actual contents.

The “one big disk” that hardware RAID cards provide limits some of ZFS’s advantages, and the read and write caches found on many hardware RAID cards are how risk gets introduced. ZFS works carefully to guarantee that every write it receives from the operating systems is on disk and checksummed before reporting success. This strategy relies on each disk reporting that data has been successfully written, but if the data is written to a hardware cache on the RAID card, ZFS is constantly misinformed of write success. This can work fine for some time but in the case of a power outage, catastrophic damage can be done to the ZFS “pool” if key metadata was lost in transit. Such failures have been known to carry five-figure price tags for data recovery services. Unlike hardware RAID, you will not suffer from data loss that can occur from interrupted writes or corrupt data returned from a hardware cache with ZFS.

Finally, most hardware RAID cards will mask the S.M.A.R.T. disk health status information that each disk provides. Very simply, each disk is connected to the hardware RAID controller card and the disks become invisible to the standard S.M.A.R.T. monitoring utility “smartctl”. Without access to this information, the user is left unaware of classic warning signs of impending disk failure, like reallocated sector count or unusually high temperature. Even the time it takes to run smartctl can be indicative of an impending problem.

While some hardware RAID cards may have a “pass-through” or “JBOD” mode that simply presents each disk to ZFS, the combination of the potential masking of S.M.A.R.T. information, high controller cost, and anecdotal evidence that any RAID mode is about 5% slower than non-RAID “target” mode results in zero reasons for using a hardware RAID card with ZFS.

Long story short, using hardware RAID on FreeNAS can lead to anything from corrupted writes to fatal errors that require you to invest in costly data recovery services.

Setting up Deduplication without Adequate Planning

Deduplication is a much-desired feature for storage solutions. On any given system, more than half your data may be duplicates of data elsewhere in your storage pool, causing a greater storage consumption. Deduplication reduces capacity requirements significantly and improves performance by tracking duplicate data with a ‘deduplication table’, eliminating the need to write and store duplicate information. ZFS stores this table on disk, which means that, if the host has to refer to the on-disk tables regularly, performance will be substantially reduced because of the slower speeds of standard spinning disks.

This means you need to plan to fit your entire deduplication table in memory to avoid major performance and, potentially, data loss. This generally isn’t a problem when first setting up deduplication, but as the table grow over time, you may unexpectedly find its size exceeds memory. This splits the deduplication table between memory and hard disk, turning every write into multiple reads & writes, slowing your performance down to a crawl. In an enterprise environment, this can cause significant productivity decreases and angry staff workers.  If this happens, the best solution is to add more system memory so that the pool will be able to import back to memory. Unfortunately, this can sometime take days to perform, and, if your hardware already has maxed out its memory capabilities, would require migrating the disks to a whole new system to access the data.

The general rule of thumb here is to have 5 GB of memory for every 1TB of deduplicated data. That said, there may be instances where more is required, but you will need to plan to meet the maximum potential memory requirements to avoid problems down the road. To get a more precise estimate of the required memory for deduplication do the following: run the ‘zdb -b (pool name)’ command for the desired pool to get an idea of the number of blocks required, then multiply the ‘bp count’ by 320 bytes to get your required memory. If it’s less than 5GB, still use the 5GB per terabyte of storage rule. If it’s higher, go with that number per terabyte.

For must use cases, it is recommended to just utilize lz4 compression for data consumption savings, as there’s no real processing cost. In fact, due to of the advances in CPU speeds, compression actually improves disk performance because writing uncompressed data to disk takes longer than compressed data. To be safe, always use compression instead of deduplication unless you know exactly what you are doing.

Striping Without Redundancy

ZFS offers all the typical forms of RAID redundancy and more, including ZFS striping (RAID 0), ZFS mirroring (RAID 1), RAID 10, and RAID-Z levels that allow for 1, 2 or 3 disk failures without affecting your storage pool.  ZFS striping can speed up your performance by spreading out writes across multiple disks and combining all your disks into one large pool. This can seem appealing to the new user because of its maximum speed and capacity, but if any of your disks has a failure, your entire pool will be lost. While, with secondary storage or non-critical data, this may not prove to be a catastrophic loss, losing your storage pool is always a big deal and it’s always recommended to configure your storage pool with some level of redundancy.

Using a SLOG for asynchronous write scenarios

The ZFS filesystem can tier cached data to help achieve sizable performance increases over spinning disks. Users can set up flash-based  L2ARC read cache and SLOG (Separate ZFS Intent Log, sometimes called a ZIL) ‘write cache’ devices. While an L2ARC read cache will speed up reads in most use cases, the SLOG only speeds up synchronous writes.

The ZIL caches writes to guarantee their completion in the case of a power failure or system crash. The ZIL normally exists as part of the ZFS pool, but with a SLOG, it resides on a separate, dedicated device. This speeds up performance by batching data together for synchronous writes for more efficiency. These performance gains help with database operations, NFS operations such as virtualization where the operating system explicitly requests synchronous writes. If you aren’t using something that is known to use synchronous writes like NFS or databases, chances are your SLOG will not help performance. A potential solution here is to set your pool to “sync=always”. This ensures that every write goes to the write cache, improving write performance.

Too Many Snapshots.  

Snapshots give users the ability to rollback to previous system states to retrieve lost files or go back to a configuration that worked properly, while only saving the file system’s blocks that have changed since the last snapshot. This results in near instant snapshot tasks. Snapshot tasks can be set for regular intervals and stay stored as long as desired.

While ZFS generally boasts that you can save unlimited snapshots, there are some practical limits to this. Some users may decide to have periodic updates every few minutes for multiple datasets and make their lifetime indefinite. Taking one snapshot every five minutes will require over 100,000 snapshots each year, creating some substantial performance loss. If you have thousands of snapshots, this means you will have thousands of blocks accumulating. Depending on the capacity of the disk, this can cause slowdowns when you list snapshots, possibly across the entire ZFS pool.

Upgrading your FreeNAS version with a full boot device

FreeNAS makes upgrading to the latest version, switching between nightly and release versions and rolling back to earlier versions very easy by storing snapshots of the OS on your boot device. However, if you fill your boot device beyond its capacity, updating your OS version may result in the upgrade process mysteriously failing. Fortunately, FreeNAS will give you an alert when your boot device exceeds 80% capacity, so you should know when your boot drive is getting full and deleting version snapshots is easy to do.

Just go into your System>>Boot tab and select the image you would like to delete and click on the delete button on the bottom of the page.


Rebuilding your ZFS array incorrectly

FreeNAS gives users the ability to set up ZFS arrays and resilver disks in the case of a drive failure. If you remove the wrong disk and try to rebuild, you can end up losing your entire pool. It is important to remember that the physical arrangement of the drives on your hardware may not correspond to your device numbers (ada0, ada1, ada2, etc). To counter this, we recommend writing down the serial numbers for each disk along with which slot they’re in, as the GUI will give you associated serial numbers in the case of a drive failure.

In addition, if you try to rebuild a ZFS array with a disk that is too small, your rebuild will fail. This can happen if you use a smaller capacity drive, say a 2TB instead of a 3TB, but it can also happen between different drives of the same listed capacity. Different drive manufacturers may create each drive with a slightly different total capacity, making the effective capacity of your replacement drive slightly higher or lower than the disk you replaced. If the capacity is slightly higher, your rebuild will succeed, but if it is slightly lower, it will not.

If a failure occurs on drives with the same listed capacities, there is a workaround available from the FreeNAS web user interface. Just access your system>>advanced menu and temporarily change your Swap Size to 0 before rebuilding. Once your rebuild is complete, make sure to change it back though (usually the default of 2GiB). The extra 2GiB should accommodate any small difference in drive capacity but do try to use identical drives whenever possible.


Other Issues to Watch For

There are a couple of common issues with Active Directory that can cause problems. The first is if the system clock is out of sync. Make sure you’re using a time server as AD/CIFS is very time sensitive. Second, having the domain name entered incorrectly can cause your Active Directory to have big problems. Ideally, your domain should have a reverse DNS entry, which you can determine easily enough:

Also, whenever possible, try not to mix sharing services on the same dataset. Differences in permissions between Unix (NFS) and Windows (CIFS) sharing formats can create some conflicts, so try and avoid this when you can. If you need users from multiple operating systems to have access to the same datasets, CIFS/SMB is your best choice.  If you need to have multiple sharing protocols, you will want to separate your datasets between NFS & CIFS/SMB.

Finally, filling your storage pool over 80% of capacity will cause degraded performance. Try to plan your storage pool size to accommodate for this.


When deploying any server or storage system, setting up your system properly can help prevent headaches and even catastrophes down the road. As they say, an ounce of prevention is worth a pound of cure. While there are many aspects to setting up any given use case, this guide should avoid most of the major pitfalls people run into while setting up their FreeNAS storage. And if you’re looking for even greater assurance, visit, call us at 1-855-GREP-4-IX or email us at, for information on our qualified, professionally supported TrueNAS appliances. We look forward to hearing from you!


Docker Done Right

Written by Michael Dexter on .

Yes, that is a bold statement. The Docker application containment architecture is all the rage right now and FreeBSD just may prove to be the ultimate Docker platform thanks to its 15+ years of containment experience and the unrivaled OpenZFS file system.

As one Twitter user put it, “#docker has now had more security issues within a year then
#freebsd #jails has had since 2000. Good job #techbros.”

Indeed, Docker has never been pitched as a security technology but rest assured, Docker on FreeBSD institutionally imprisons and secures Docker images using FreeBSD’s proven Jail infrastructure. FreeBSD Jails have been used in production since their inception to contain applications and full systems and are exactly what Docker needs. Docker itself has migrated away from Linux LXC containers in favor of the cross-platform libcontainer and of all the pluggable choices, FreeBSD’s Jail stands out as one of the best. FreeBSD also offers the bhyve and Xen hypervisors to provide you yet more options for containing your Linux-native and FreeBSD-native Docker deployments.

Then comes storage. Docker images are designed to be read-only and disposable until instructed otherwise. If only there were a file system that institutionalized lightning-fast snapshotting and cloning…

That file system exists! It’s called OpenZFS and FreeBSD has supported it since FreeBSD 7.0. This not only means you get the institutionalized snapshotting and cloning that suit Docker so well, but also the unrivaled data integrity protection that OpenZFS offers. If you care about your data, you care about OpenZFS.

Hands-on Docker

To try Docker on FreeBSD, you will need a recent snapshot such as 10.2 BETA or 11-CURRENT. Note that you should change “zroot” to match your system’s zpool.

 # pkg install docker-freebsd ca_root_nss
 # zfs create -o mountpoint=/usr/docker zroot/docker
 # service docker onestart
 Starting docker…

 # docker pull centos

 # docker images
 centos latest 7322fbe74aa5 4 weeks ago 172.2 MB

 # docker run -t -i centos /bin/bash
 [root@ /]# uname -a
 Linux 2.6.32 FreeBSD 11.0-CURRENT #5 r285594: Tue Jul 14 23:30:11 EDT 2015
 x86_64 x86_64 x86_64 GNU/Linux

Suddenly… CentOS!

Where the wheels really hit the pavement is with a peek under the hood at the Jail and ZFS output of our Docker Jail and OpenZFS dataset:

 # jls
   JID IP Address Hostname Path
     3 /usr/docker/zfs/graph/920bc5fbb45c

 # zfs list
       119M 107G 6.02M /usr/docker
       8K 107G 112M legacy
 init 128K 107G 112M legacy

This output should be familiar to FreeBSD users and is becoming familiar to more and more GNU/Linux users every day.

For an expanded example of Docker on FreeBSD, consult the FreeBSD Wiki:

FreeBSD is poised to be go-to Docker platform thanks to FreeBSD’s proven Jail and OpenZFS features and iXsystems has shipped over ten thousand systems with the best support for these features available anywhere. We can also build out your GNU/Linux-based Docker deployment and ship thousands of GNU/Linux systems every year. Give us a call to learn how we can take your Docker deployment to the next level and beyond.

Michael Dexter

Apple’s Jordan Hubbard Joins iXsystems

Written by Ben Milman on .

FreeBSD Co-Founder Wraps Up Historic Tenure at Apple to Become iXsystems CTO

iXsystems®, Inc. is delighted to welcome Jordan Hubbard, former Director of UNIX Technology at Apple®, as Chief Technical Officer. Hubbard will lead engineering and development at iXsystems, take the reins of FreeNAS® (the wildly popular open source storage platform) and work to bring the TrueNAS® Unified Storage Appliance to a wider audience by adding new capabilities, such as object storage, simpler cloud integration, and high-performance real-time data deduplication, to name a few. He will also work to create future products and services which better serve the needs of emerging enterprise and consumer markets.

At Apple, Hubbard led the development of many BSD and Unix technologies at the core of Mac OS® and iOS over the last 12 years. His primary areas of focus were on modernizing the Unix platform, creating better and more fundamental security technologies, increasing performance and power efficiency, and tirelessly promoting the common interests and exchange of technologies between the OSS community and Apple.

With Hubbard’s knowledge and decades of experience, iXsystems is poised to expand further into the enterprise storage market, increasing support for industry standards while continuing to develop new and innovative technology. TrueNAS Unified Storage is already used around the world supporting big data, virtualization, and cloud computing infrastructures and, under Hubbard’s leadership, TrueNAS will further solidify its position as the most powerful and easy to use storage option on the market.

“I’m very excited to have this opportunity to help guide iXsystems through its next phase of professional and technological development,” says Hubbard. “This is not just a great opportunity for me to be part of a company known for its strong support of FreeBSD and other open source software, but I’m also looking forward to helping it achieve new levels of success with the TrueNAS storage appliance and having the opportunity to create future innovative products.”

Hubbard and iXsystems are connected through the FreeBSD® Project and many years of history. Hubbard co-founded the FreeBSD project in 1993 and was a long-time executive employee of iXsystems’ corporate ancestors, BSDi and Walnut Creek CD-ROM. BSDi became iXsystems in 2001, and Walnut Creek became the FreeBSD Mall in 2002, before re-joining iXsystems in 2005.

iXsystems is a champion of FreeBSD, making both financial and technical contributions to the project as well as running the FreeBSD Mall and representing FreeBSD at open-source conferences and trade shows around the world. Hubbard is the ideal figure to lead the development of FreeBSD-based systems at iXsystems and spearhead the company’s contributions to FreeBSD, carrying on the long tradition of being a leading supporter of the open source community.

Matt Olander, Co-founder and CTO Emeritus of iXsystems, is thrilled to welcome Jordan back to the iX family. Apple has a well-deserved reputation for making some of the highest quality, easiest to use products in the world, and we’re thrilled to have someone with this kind of experience join our team in a leadership role. No one else has Jordan’s enthusiasm and skill for bringing great technology to a wide audience, and his long history with us and with FreeBSD makes bringing him on board just that much sweeter. We’re looking forward to a grand new era of innovation, and Jordan will help lead the way!

Jordan Hubbard will assume the duties of iXsystems CTO on July 15, 2013.

About iXsystems®
iXsystems builds rock solid enterprise-class servers and storage solutions. All of our products are assembled, tested, and shipped from our company headquarters in Silicon Valley. Technical support is provided in-house by the same engineers that build the systems. Thousands of companies, universities, and U.S. Government departments have come to rely on iXsystems’ customer-first commitment to excellence. iXsystems champions the cause of Open Source technology by dedicating extensive resources to several FreeBSD community projects: FreeNAS, PC-BSD®, FreeBSD, and TrueOS®.

Apple, Mac OS, and OS X are registered trademarks of Apple, Inc.
FreeBSD is a registered trademark of the FreeBSD Foundation, used with permission.
UNIX® is a registered trademark of The Open Group.
IOS is a trademark or registered trademark of Cisco in the U.S. and other countries and is used under license.