In part two of this “Building a 20 disk FreeNas fileserver” blog, I will cover the internals of the system. Be sure to have read the first part of this build log first!

Updated 13-Nov-2016

Supermicro X10SLH-F

Price: € 230,70

The motherboard is usually the first thing I choose for a new build and this time was no exception. For this build I had a number of motherboard requirements…

Support for ECC memory

Reading the FreeNas forums, this requirement get repeated very often. In my old file server I did not use ECC memory and I never had an issue, but since the fileserver will now really carry a lot of data, the inconvenience of having to restore everything far out-weighs the added cost of ECC memory. This requirement meant I ended up in the server boards range.

IPMI included, please

Having IPMI is just so useful. I have a server in a datacenter “somewhere” and I’ve gotten used to have IPMI. This fileserver will be “at home” but I do have a VPN setup that will allow me to reach the IPMI connection safely from anywhere. Since the ECC memory requirement had already pushed me into the server boards category, this requirement was easy.

Gigabit network, at least

The 10mbps network device on my on fileserver was a real bottleneck at times. 1gbps network is pretty standard on motherboards today and should be sufficient for the near future at least. Since my network infrastructure is also gigabit based, it makes no sense to get a faster network device. This requirement was easy.

Two x8 PCI-E 3.0 slots

For 20 disks, I will need to add two 8 disk HBA cards. The IBM M1015 cards require x8 PCI3 slots, so my motherboard needs two of those. Of course a x16 slot can be used too. This requirement did prove to be a lot harder to fulfill. and is the main reason I went for the X10SLH-F board.

On-board graphics

This is “nice to have” since it means I do not need to purchase a graphics card. That would be necessary because the cpu’s that fit on this motherboard usually do not even have an option for integrated graphics and it would make no sense to pay extra for it. Also, since I am using two x8 PCI slots for the HBA’s, I would most likely need to get a “power gaming” motherboard to be able to add a graphics card. 😉

Needless to say, the Supermicro motherboard I went for has on board VGA graphics. And to be honest, the graphics option is not even being used: since I have IPMI to reach the system, I have not hooked up a monitor. But I guess it is nice to have the option if I ever need it.

Enough sata ports available

Of course, the two 8 drive HBA cards only give me 16 drives. So I need an additional 4 Sata (6Gbps) ports to feed all of the 20 data drives (plus one more for the boot device).

I know that a lot of FreeNas systems boot from a USB thumb drive. Indeed, my old fileserver did that too. However, I have had a number of corrupted USB sticks in the past and I decided to move away from that. The Supermicro motherboard has 6 sata ports on board and fulfills the requirement nicely.

Sata DOM power connector

Not going for the USB thumb drive boot device means going for an extra HDD or SSD. The motherboard has the sata data ports available but a regular drive would need an additional sata power connector too and sadly, my setup has just enough power connectors to power the disk cages. I could have sourced a power lead with an extra power connector, but while looking for these, I found out about the Sata DOM (disk on module) solution: mini SSD drives you can plug directly in the sata connector and, more importantly, are able to get power using dedicated connectors on the motherboard. The X10SLH-F motherboard has an on-board 5v power connector for a Sata DOM.

Well liked by the community

I also wanted to be certain that the motherboard I used was a known entity. If any issues arise, it helps if the component is known and used by many others. This Supermicro X10SLH-F board turns out to be a bit of a favorite in the FreeNas community so mission accomplished: we have a motherboard!

Intel Xeon E3-1241 v3 @ 3.50GHz

Price: € 284,45

Once the motherboard is selected, a choice for the CPU is next. My primary requirement here was actually price. It is easy to spend way too much money on the biggest, fastest cpu you can find, but since this is a home fileserver, concurrent usage will be rather low which means four cores would be plenty.

I do want to use CIFS file sharing and since CIFS is single threaded, it is recommended to have a reasonably high clock frequency. This meant I did not go for the cheapest four-core E3, but went one step up.

The E3 cpu’s have a very nice performance vs power ratio. This is not unimportant for a fileserver that will be running 24/7. And E5 or E7 would of course be faster, but both the purchase price and running cost would be significantly higher. And to be honest, the E3 has plenty of power for a fileserver which will always be rather IO bound due to the slow hard disks. The only reason I might go for an E5 is if I have all SSDs or do a lot of encryption. As far as I know, the E3 has less hardware acceleration when it comes to encryption. But I am not going to encrypt my ZFS so there is simply no need to spend extra on a bigger processor.

I also have no problem going to an Intel processor. Yes I like AMD fine too: my HTPC runs an AMD A10 and I am very happy with it. But the requirements for the motherboard only yielded Intel options, so…

Update (13-Nov-2016): I received feedback on this section from Stux on the Freenas forums. The E3 does have the same encryption capabilities as the E5/E7 variants, so there is no reason to drop the E3 CPU’s, even if encryption is important for you. Do go and read Stux’s post for the full story!

Kingston 8GB PC3-12800 DDR3-1600MHz ECC Unbuffered CL11 1.35V

Price: € 90,75

Choosing the memory was easier than expected: the supermicro site only has a small number of certified DIMMs listed and these were good value for money. Sadly, even though these can run low-voltage, the motherboard is not able to provide it, so they run at regular voltages.

Just in case you are wondering, this is not an issue: all low voltage DIMMs can run regular voltage too. I contacted Kingston support before buying and they confirmed. 🙂

FreeNas recommends a strict minimum of 8GB RAM and more RAM usually means better performance. I also want to run a number of services on this fileserver (think DNS/DHCP/Backup/OwnCLoud/…) so I decided to not stop at 16GB RAM but got 4 of these DIMMs.

IBM ServeRAID M1015

Price: € 77,42*

Another easy choice was the HBA (host bus adapter). I used one of these in my old server so I knew how the worked and how reliable they are. Since I wanted to build the new fileserver without tearing the old one down, I purchased two new cards. The card from the old fileserver will serve a s a nice backup, just in case a card breaks down.

The prices for these cards are impossible to predict since most people source these from ebay. Personally I got mine from www.aliexpress.com. It’s always a bit of a risk to go this way, but I have been very happy with aliexpress in the past. I was able to buy two cards for € 163,47, including shipping.

*Now I am not saying you will be able to find them for this price (you might even be able to source them cheaper), but I do not want to use the € 400,00+ prices I have seen for these cards since I would never have gotten them at that price.

One thing to look out for when you are using multiple cards in one system is to make sure that the card IDs are different. The UUIDs that are generated for your disks will also include the pci slot, but I find it comforting to know the card IDs are different so there is no risk. I may be a bit paranoid here, but please allow me my phobia! 🙂

Changing the ID is easy to do using the flash tools. You will need those tools anyway because you will want to re-flash the HBA card to “IT” mode. Basically, this removes all the raid intelligence from the card. You might think that’s a bit of a shame, but the reasoning is that FreeNas and ZFS do a better job with just “dumb” drives so the advice is to do this.

Supermicro SuperDOM 32GB

Price: € 65,96

As touched upon earlier, I decided to shy away from using a USB thumb drive this time. Yes they work fine and yes they are much cheaper but yes, they also go bad quite quickly. I’ve had USB stick go bad on me within a few months and the worst part of it is that you only notice it when suddenly “strange things” start happening.

When I discovered the Sata DOMs, I decided to go for that and I do not regret it. They appear to be quite stable and of course they are much faster than a USB stick.

I was tempted to go for the 16GB version instead, but when looking at the FreeNas documentation, they now seem to be advising to use a 32GB boot device. The price difference was not that large so I decided to go for it.

I actually also debated going for two of these DOMs in a mirror setup. FreeNas now has good support for mirrored boot devices. The main reason that held me back was the power requirements: the supermicro motherboard has one on board power connector for a DOM. The second module would require a connection from the power supply. And since I am critically short on power connectors… 🙂

A second reason not to go for two DOMs is that even though the module clicks directly into a sata port on the motherboard, they are a little bit wider that the port which means that when your sata ports are side-by-side, as they are on my motherboard, one DOM will block the sata port next to it.

Of course this can be solved by putting the DOMs on small Sata extension cables but at this point, it all got to be too much: an extra power cable, two extra sata cables… No thanks. Let’s just go with a single module: the way FreeNas works makes it easy to temporarily use a USB thumb drive if the DOM should happen to fail. And since a DOM is unlikely to fail, I should be fine.

Since Supermicro sells these modules, I just went for them, even though as always, Supermicro does charge a little extra… At least the quality should be good. 🙂

By the way, in case you are wondering: yes the little 5v cable to power the DOM module is included with the module. This is a little detail I could find nowhere so I just ordered and hoped for the best. But the cable does indeed come with the “SuperDom”. 🙂

The Data Storage Devices

Price: € 108,95 (3TB), €155,50 (4TB)

Well, you cannot have a fileserver without disks, so… 🙂

The old fileserver had ten 3TB disks which I now have re-purposed for the new file server. I also purchased 10 new 4TB disks to complete the build. I have had excellent experience with WD Red drives in the past so it was a no-brainer for me to go that way again.

I debated going for 6TB drives but decided the price difference is not worth it just yet. I will be replacing the 3TB drives with 4TB drives when they fail, so eventually they will all become 4TB drives.

Funnily enough, this may take longer than expected due to the excellent support from Western Digital. 🙂 Any drive that fails withing 3 years is replaced free of charge. They even pay for the shipping.

I even had a few drives that failed just after the warranty expired and they still got replaced. Since my fileserver is not installed in “data center” conditions regarding power stability and cooling, a good warranty service is very welcome.

The final (price) picture

That’s all for the components that make up my new fileserver. Assembling is nothing special; anyone who has ever put together a PC will have no trouble with this. Just be careful: a tower with 20 HDDs is a pretty heavy thing to “shlep around”. 🙂

        Case: Antec twelve hundred V3                      186,99
   HDD cages: 4x Icy Dock FatCage MB155SP-B  4x  97,50 =   390,00
Power Supply: Seasonic Platinum 660W                       154,90
 Motherboard: Supermicro X10SLH-F                          230,70
         CPU: 4 Core Intel(R) Xeon(R)                      284,45
              E3-1241 v3 @ 3.50GHz
      Memory: 4x Kingston 8GB PC3-12800      4x  90,75 =   363,00
              DDR3-1600MHz 
              ECC Unbuffered CL11 1.35V 
              (P/N 9965525-139.A00LF)
         HBA: 2x IBM ServeRAID M1015         2x  77,42 =   154,84*
 Boot device: Supermicro SuperDOM 32GB                      65,96
              (SSD-DM032-PHI)
     Storage: 10 x 3TB WD Red (WD30EFRX)    10x 108,95 = 1.089,50
              10 x 4TB WD Red (WD40EFRX)    10x 155,50 = 1.555,00

                                                 Total € 4.475.34

*Your price for the HBA's will probably be different!

So there you have it. I did not add the costs for the F-Clamps I needed since, well… I didn’t want to. :p But about 60% of the total price is going to the actual storage devices, the disk drives, I guess that is not too bad.

In the next and probably last installment, I’ll add more info on the FreeNas installation, including the pool setup and other stuff I am now running in this machine. Until then, how much usable disk space do you think this system offers in the end?