#ZFS on #Linux observations:
1. ZFS on #Solaris is awesome.
2. My experience with ZFS on Linux has been terrible.
I'm using a Dell #R720 configured as a NAS server, with a Dell PERC H310 controller that natively supports JBOD, running Gentoo Linux. The Dell replaced a succession of two SunFire X4540s, both of which were absolutely rock-solid as NAS servers (until their system controller boards failed) and never once had a ZFS error reported except when a drive physically failed. With the R720, I get hot and cold running errors reported. I'm using all Samsung 870 Evo solid-state drives, in two #RAIDZ arrays, one of eight drives and one of six. I am at this very moment in the process of cleaning up the arrays ... again.
What I can't figure out is why.
— Is ZFS on Linux really that terrible?
— Does ZFS on Linux just somehow not work well with SSDs?
— Does the PERC controller in the R720 not work well with SSDs?
I wasn't originally running SSDs in this array; my first attempt was using 2.5" spinning rust drives. I rapidly discovered two things:
1. As far as I can determine, all 2.5" mechanical hard drives 2TB or larger on the market are SMR drives;
2. OH MY GOD, SMR DRIVES (especially, I am told, in ZFS) ARE UTTERLY FUCKING HORRIBLE except on WORM (read once, write many) applications in which you don't really care how slow the original write is. RAIDZ write performance on the Dell on brand new 2.5" SMR drives was four to six times slower than RAIDZ write performance on the X4540 with older and slower CMR drives on older and slower SCSI/SAS controllers. Despite newer, "faster" drives on a newer, faster controller, the SMR array was utterly unusable.
Now, I'm not experiencing any problems with SSDs in any of my other systems, Windows or Linux, INCLUDING the R720, except with ZFS. The boot drives on the R720 are an mdraid mirror formatted XFS and have never thrown a single error.
So this is really leading me to wonder a crucial question:
Is there something I don't know about #ZFSonLinux that causes it to not work well with #SSD drives? Do I need to just forget about running ZFS on my NAS and let the PERC controller create hardware RAID5 volumes?
(And if anyone wonders "why don't you just run a commercial NAS appliance?", well, I tried that route. I tried one of the very latest generation QNAP servers that run ZFS storage on a Linux OS. Oh my god, I can't even begin to speak to how horribly bastardized it was. QNAP may well be a good NAS choice if you only care about Windows and SMB and never ever want to look under the hood or try to accomplish anything except through the web front-end, and don't already have an existing backup solution that you want to continue using.)