my build: https://pcpartpicker.com/b/8hgJ7P#cx4069854
To assist ZFS in discovering controller errors, each drive from a given mirror zvol is connected to different controller. There are 2 controllers in use: one on-board (AHCI) and one in the form of PCIe card from LSI. Each has 4 ports, so ample for the 6 drives installed.
- SAS 3008 models: https://forums.servethehome.com/index.php?threads/lsi-raid-controller-and-hba-complete-listing-plus-oem-models.599/
- bought from the art of server on ebay
- uses the
mprdriver
- IT mode flashing: https://github.com/EverLand1/9300-8i_IT-Mode
- which was not necessary because the card came already pre-flashed with latest IT firmware
nas% sudo sas3flash -listall
Avago Technologies SAS3 Flash Utility
Version 16.00.00.00 (2017.05.02)
Copyright 2008-2017 Avago Technologies. All rights reserved.
Adapter Selected is a Avago SAS: SAS3008(C0)
Num Ctlr FW Ver NVDATA x86-BIOS PCI Addr
----------------------------------------------------------------------------
0 SAS3008(C0) 16.00.12.00 0e.01.00.07 08.37.00.00 00:01:00:00
Finished Processing Commands Successfully.
Exiting SAS3Flash.
The LSI MEGARAID SAS 9240-4I 4-PORT PCIE 6GB card does not support disks larger than 2 TiB.
- https://groups.google.com/g/mailing.freebsd.questions/c/Zw6L8PM_POY?pli=1
- https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=147572
- https://web.archive.org/web/20110303074426/http://kb.lsi.com/KnowledgebaseArticle16399.aspx
- https://www.servethehome.com/lsi-sas-2008-raid-controller-hba-information/
root@nas[~]# sas2flash -listall
LSI Corporation SAS2 Flash Utility
Version 16.00.00.00 (2013.03.01)
Copyright (c) 2008-2013 LSI Corporation. All rights reserved
Adapter Selected is a LSI SAS: SAS2008(B2)
Num Ctlr FW Ver NVDATA x86-BIOS PCI Addr
----------------------------------------------------------------------------
0 SAS2008(B2) 16.00.00.00 10.00.00.06 No Image 00:01:00:00
Finished Processing Commands Successfully.
Exiting SAS2Flash.
The data set needs to be created with SMB sharing option. It seems that it deals with case sensitivity and other data set options.
https://www.youtube.com/watch?v=mCfX4sqDmzs is a how to for setting up SMB shares on FreeNAS.
https://www.ixsystems.com/blog/users-permissions-acls-on-freenas/ also mentions SMB shares.
For the NAS to be visible under 'Network' for macOS 10.13.6 it is necessary to enable the NetBIOS advertisement in the Sharing.
Time machine shares should have a quota - it eventually governs the number of "snapshots" created by Time machine. Multiple machines using single Time machine data set is bad idea - it would eventually lead to
macOS catalina does not seem to be supporting Time machine over SMB. Thus AFP has to be used.
To set up the Time machine AFP share, create a data set and edit its properties, change its quota and ownership to the user who will be using it.
On macOS catalina it is necessary to add cnid scheme = tdb to the Auxiliary parameters in the AFP share settings.
ref: https://www.ixsystems.com/community/threads/time-machine-over-afp-problem-permission-denied.85359/
The CPU fan is connected to the motherboard via a cable which limits the speed. This bumps into the BMC limits which results into PagerDuty alerts to be created. To avoid that, I tried to limit the low thresholds for the CPU fan via IPMI:
sudo ipmitool sensor thresh CPU_FAN1 lower 250 300 350
Interestingly in the sensor list the values are then displayed as 300/300/400. However, for some reason this made the BMC produce several warnings per minute about the thresholds being reached. So, I opted for simple Molex to 4-pin (or rather 2-pin) adapter cable to avoid the fan speed to be monitored. Also, it seems that the threshold setting was not persistent across power off. Anyhow, the functionality of the CPU fan can be deduced from the CPU temperature.
- Alerting via system speaker: https://github.com/vladak/truenas_beep
- Grafana dashboard: https://gist.github.com/vladak/e3707bfc2f81307de0268530691e9572
Here's the graph displaying the CPU fan being turned on at one point:
The CPU temperature alert in Grafana is set to use 42C as the threshold for the median values of the 4 CPU metrics. If the fan ever goes bad again, it will clearly signal this condition.