Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Thoughts about ROOT on ZFS
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Gentoo Chat
View previous topic :: View next topic  
Author Message
lcsavb
n00b
n00b


Joined: 11 Jun 2020
Posts: 30
Location: Sao Paulo / Brasil

PostPosted: Tue Jun 16, 2020 12:32 pm    Post subject: Thoughts about ROOT on ZFS Reply with quote

Recently I've been in a lot of trouble to use the gentoo root on zfs.

I've found a detailed how-to: https://www.funtoo.org/ZFS_as_Root_Filesystem, BUT there's a warning:

Quote:
The Funtoo Linux project cannot currently recommend ZFS on Linux for use in production. In fact, we recommend that you do not use ZFS in production. We have had a number of issues with this filesystem, even after making some very significant effort to make it work more reliably. Please use BTRFS instead. It's better than you've heard.


In fact I'd like to do that because its snapshotings abilities and I was planning to make a RAID0 with two NVME M2 drives - I mess with the system a LOT and often break it.

So, it's not an absolute need and I can appoint three caveats: the RAID0 scheme wouldn't bring any tangible performance improvement, it would reduce reability of my system (the SSDs alone are pretty fast) and I could achieve the same goal with with mdadm and BTRFS. Actually I've found that ZFS lags behind BTRFS and XFS in terms of write/read times.

What's your thoughts about running gentoo root on zfs? Is it worth it or even safe? For sure it is not easy - I'm on the brink of giving up.
Back to top
View user's profile Send private message
Morality124
Tux's lil' helper
Tux's lil' helper


Joined: 20 Feb 2018
Posts: 78

PostPosted: Tue Jun 16, 2020 3:34 pm    Post subject: Reply with quote

That wiki page has not been updated since November 2019... since then there have been two major ZoL releases that have fixed many issues, most specifically compiling with newer kernels without incurring major performance loses. Furthermore, the recommendation of Btrfs as an alternative is... spurious at best as Btrfs is notorious for having major data corruption bugs with seemingly every new kernel release, whereas ZFS is simply a more robust codebase.

I have had no troubles patching manually ZFS support and running it in my kernels, and I trust my ZFS partitions far more than the Btrfs ones because of the proven codebase.

It is correct that ZFS can be slower than other file systems... but then ZFS is not a "performance file system" like XFS, it is file system for maintaining data integrity.

Admittedly I still have to convert my system to use ZFS as root, but knowing someone else who has already done so with an initramfs, I don't have worries that I'll be able to accomplish this.

In regards to RAID and consumer SSDs, there was a proc setting to set to ensure more reliable operation that I can't recall at the moment. (EDIT: echo 200 > /sys/block/sd[letter]/[device]/timeout - gives enough time for the drives to report errors to the kernel).

If data integrity is not your top priority, and you don't have a need for the many ZFS-specific features like send, fixed snapshots, etc, then sure use another file system if you want something faster and more convenient.
Back to top
View user's profile Send private message
ali3nx
l33t
l33t


Joined: 21 Sep 2003
Posts: 676
Location: Winnipeg, Canada

PostPosted: Tue Jun 16, 2020 6:51 pm    Post subject: Re: Thoughts about ROOT on ZFS Reply with quote

lcsavb wrote:
Recently I've been in a lot of trouble to use the gentoo root on zfs.

I've found a detailed how-to: https://www.funtoo.org/ZFS_as_Root_Filesystem, BUT there's a warning:

Quote:
The Funtoo Linux project cannot currently recommend ZFS on Linux for use in production. In fact, we recommend that you do not use ZFS in production. We have had a number of issues with this filesystem, even after making some very significant effort to make it work more reliably. Please use BTRFS instead. It's better than you've heard.


In fact I'd like to do that because its snapshotings abilities and I was planning to make a RAID0 with two NVME M2 drives - I mess with the system a LOT and often break it.

So, it's not an absolute need and I can appoint three caveats: the RAID0 scheme wouldn't bring any tangible performance improvement, it would reduce reability of my system (the SSDs alone are pretty fast) and I could achieve the same goal with with mdadm and BTRFS. Actually I've found that ZFS lags behind BTRFS and XFS in terms of write/read times.

What's your thoughts about running gentoo root on zfs? Is it worth it or even safe? For sure it is not easy - I'm on the brink of giving up.


Gentoo on zfs root works great especially if you have the capability to use uefi boot.

https://wiki.gentoo.org/wiki/User:Ali3nx/Installing_Gentoo_Linux_EFISTUB_On_ZFS

Using grub to boot zfs root while widely provided as the recommended example setup for many people makes using zfsroot setups a real pain in the arse that can entrely be avoided by not using grub.

My own server runs a 50 ish Terabyte raidz2 pool, two sata ssd mirror pools and has been very reliable.

One possible complication you may have depending on the motherboard used is attempting to use mirrored or raid nvme may not behave due to pci express link design of the motherboard used but you'll only discover this by attempting to configure such a setup. I attempted to run mirrored samsung 960 pro nvme ssd's using an asus z270-a skylake motherboard and the results were not stable due to hardware design limitations.

Really overall raid 0 especially would be best avoided and morso with nvme due to the possibility of raid nvme being unstable. If you were going to use zfs for nvme creating a mirror pool from two nvme devices would be highly preffered. the performance gains of raid 0 on nvme would be largely insignificant compared to the feature benefits and data integrity zfs would provide.

Having data integrity rely on raid 0 makes the point of using it more or less irrelevant.
_________________
Compiling Gentoo since version 1.4
Thousands of Gentoo Installs Completed
Emerged on every continent but Antarctica
Compile long and Prosper!
Back to top
View user's profile Send private message
Dorsai!
Apprentice
Apprentice


Joined: 27 Jul 2008
Posts: 280
Location: Bavaria

PostPosted: Tue Jun 16, 2020 7:36 pm    Post subject: Reply with quote

Instead of a striped mirror (RAID10) I on my pool I use one raidz2 vdev with 6 common HDDs and one special vdev in the form of a mirror of 2 Samsung 970 Pro NVMe SSDs.

By default the special vdev will be used for metadata, which already speeds the system up a fair bit due to fewer small reads and writes to the HDDs. Additionally I've set the special_small_blocks property everywhere to 8k so all small blocks are also allocated to the SSDs, which further reduces random access to the HDDs.

But this is where it gets awesome because on my root filesystem I have set special_small_blocks=128k, which effectively allocates my entire root fs (only 8GB including portage) on the SSD resulting in blazingly fast boot and app startup.

I have never read about anyone else exploiting this allocation classes feature in that way to get real hybrid storage.
Back to top
View user's profile Send private message
lcsavb
n00b
n00b


Joined: 11 Jun 2020
Posts: 30
Location: Sao Paulo / Brasil

PostPosted: Tue Jun 16, 2020 11:08 pm    Post subject: Re: Thoughts about ROOT on ZFS Reply with quote

ali3nx wrote:

Gentoo on zfs root works great especially if you have the capability to use uefi boot.

https://wiki.gentoo.org/wiki/User:Ali3nx/Installing_Gentoo_Linux_EFISTUB_On_ZFS



That's a great guide! Thanks!

I'll give it another shot!
Back to top
View user's profile Send private message
ali3nx
l33t
l33t


Joined: 21 Sep 2003
Posts: 676
Location: Winnipeg, Canada

PostPosted: Wed Jun 17, 2020 12:56 pm    Post subject: Re: Thoughts about ROOT on ZFS Reply with quote

lcsavb wrote:
ali3nx wrote:

Gentoo on zfs root works great especially if you have the capability to use uefi boot.

https://wiki.gentoo.org/wiki/User:Ali3nx/Installing_Gentoo_Linux_EFISTUB_On_ZFS



That's a great guide! Thanks!

I'll give it another shot!


Thanks for the compliment. How's the progress progressing? :)
_________________
Compiling Gentoo since version 1.4
Thousands of Gentoo Installs Completed
Emerged on every continent but Antarctica
Compile long and Prosper!
Back to top
View user's profile Send private message
lcsavb
n00b
n00b


Joined: 11 Jun 2020
Posts: 30
Location: Sao Paulo / Brasil

PostPosted: Wed Jun 17, 2020 4:07 pm    Post subject: Re: Thoughts about ROOT on ZFS Reply with quote

ali3nx wrote:

Thanks for the compliment. How's the progress progressing? :)


I'm almost there, but the computer is freezing in early boot. https://forums.gentoo.org/viewtopic-p-8471726.html#8471726.

I've rsynced my working installation and built the zfs into the kernel.

Not sure what's the problem and how to debug it.

Code:

➜  ~ zfs list
NAME                USED  AVAIL     REFER  MOUNTPOINT
rpool              49.5G   400G       24K  none
rpool/ROOT         49.5G   400G       24K  none
rpool/ROOT/gentoo  49.5G   400G     49.5G  /
rpool/home           48K   400G       24K  none
rpool/home/tux       24K   400G       24K  /home/tux
rpool/usr            24K   400G       24K  none
rpool/var            48K   400G       24K  none
rpool/var/lib        24K   400G       24K  none


Code:

sysresccd /efi # efibootmgr -v
BootCurrent: 0009
Timeout: 1 seconds
BootOrder: 0000,0003,0008,000A,0009,0002,0007
Boot0000* Gentoo ZFS 5.4.38 HD(1,GPT,39cfc197-109d-4a54-a216-0ccca39327fe,0x800,0xf4000)/File(efi\gentoo\vmlinuz-5.4.38-gentoo.efi)r.o.o.t.=.Z.F.S.=.r.p.o.o.l./.R.O.O.T./.g.e.n.t.o.o. .d.o.z.f.s. .i.n.i.t.r.d.=.\.e.f.i.\.g.e.n.t.o.o.\.i.n.i.t.r.a.m.f.s.-.5...4...3.8.-.g.e.n.t.o.o...i.m.g.
Boot0002  CT2000MX500SSD1 BBS(HD,,0x0)..BO
Boot0003* Samsung SSD 970 EVO Plus 500GB BBS(HD,,0x0)..BO
Boot0007  Samsung SSD 970 EVO Plus 500GB BBS(HD,,0x0)..BO
Boot0008* Seagate Expansion 0710 BBS(Floppy,,0x0)..BO
Boot0009* UEFI: ADATA USB Flash Drive 1100, Partition 1 PciRoot(0x0)/Pci(0x8,0x1)/Pci(0x0,0x3)/USB(3,0)/HD(1,MBR,0xde1d416a,0x800,0x1ce7000)..BO
Boot000A* ADATA USB Flash Drive 1100 BBS(HD,,0x0)..BO


And, finally, genkernel output:

Code:

* WARNING... WARNING... WARNING...
* Additional kernel parameters that *may* be required to boot properly:
* - Add "dozfs" for ZFS volume management support
*   and either "root=ZFS" to use bootfs autodetection or "root=ZFS=<dataset>"
*   to force booting from a specific dataset
*
* Hostid '53257e1f' is embedded into initramfs.
* If you will use this initramfs for a different system you MUST set 'spl_hostid=<hostid>' parameter to overwrite embedded hostid!
*
* If importing ZFS pool is slow, add dozfs=cache or dozfs=force to kernel commandline.
* "man genkernel" explains "dozfs" in detail.

* Do NOT report kernel bugs as genkernel bugs unless your bug
* is about the default genkernel configuration...
Back to top
View user's profile Send private message
ali3nx
l33t
l33t


Joined: 21 Sep 2003
Posts: 676
Location: Winnipeg, Canada

PostPosted: Wed Jun 17, 2020 6:19 pm    Post subject: Reply with quote

perhaps you could also post parted -l and zpool status?

If you created a mirror pool the method used to partition the disks used to create a mirror pool for zfs is important. the UUID''s for every partition used must be unique or the zfs pool will fail to import.

Some older linux raid configuration guides offer to users that using sfdisk to clone partition tables is a useful benefit but that doesn't work for zfs pools because sfdisk will not create unique partition uuid identifiers.

I've also had some initial problems with initramfs functioning properly on initial boot attempts due to zfs attempting to import the root pool using a zfs cache file but more often those issues are resolvable by generating a new cache file and recreating the initramfs.

The completion output from genkernel is offered every time genkernel completed creating an initramfs and is normal.
_________________
Compiling Gentoo since version 1.4
Thousands of Gentoo Installs Completed
Emerged on every continent but Antarctica
Compile long and Prosper!
Back to top
View user's profile Send private message
mrbassie
l33t
l33t


Joined: 31 May 2013
Posts: 607

PostPosted: Thu Jun 18, 2020 4:21 pm    Post subject: Re: Thoughts about ROOT on ZFS Reply with quote

lcsavb wrote:


And, finally, genkernel output:


It could be your initramfs. I used to use genkernel to generate the initramfs but iirc it broke when they released 0.7.0. I switched to blissinitramfs and never looked back.
Back to top
View user's profile Send private message
ali3nx
l33t
l33t


Joined: 21 Sep 2003
Posts: 676
Location: Winnipeg, Canada

PostPosted: Thu Jun 18, 2020 4:29 pm    Post subject: Re: Thoughts about ROOT on ZFS Reply with quote

mrbassie wrote:
lcsavb wrote:


And, finally, genkernel output:


It could be your initramfs. I used to use genkernel to generate the initramfs but iirc it broke when they released 0.7.0. I switched to blissinitramfs and never looked back.


The other option i've used is dracut which does work but when dracut didn't want to work it was due to a dracut created initramfs booting too fast causing the zfs module to fail to load before the zpool import was attempted. Because of this anomaly having occurred i omitted recommending use of a dracut initramfs in my zfsroot setup guide.

That occurrence was caused by using mirrored ssd rpool however genkernel fixed that issue for me. Initial attempts to resolve that issue i created a bloated initramfs including the entire linux firmware directory compressed with zl4 to attempt to slow down the initramfs processing which worked but I later settled on just using genkernel compression defaults of xz as the lz4 compressed initramds didnt make a lot of difference for my own nas server and the xz compressed initramfs was 150MB smaller :lol:

Using lz4 seemed to make sense given zfs dataset compression widely prefers or recommends lz4.

I've never needed to use Fearedbliss git repo to make zfsroot function but its great he's contributed what he was. The livecd's Fearedbliss releases provide one of the only sources of zfs capable linux livecd's with modern zfs software revisions i'm aware of.
_________________
Compiling Gentoo since version 1.4
Thousands of Gentoo Installs Completed
Emerged on every continent but Antarctica
Compile long and Prosper!
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Gentoo Chat All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum