Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
RAID vs rsync. Your preferences, experiences?
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2  
Reply to topic    Gentoo Forums Forum Index Gentoo Chat
View previous topic :: View next topic  
Author Message
steve_v
Apprentice
Apprentice


Joined: 20 Jun 2004
Posts: 177
Location: New Zealand

PostPosted: Mon Aug 24, 2020 8:54 am    Post subject: Reply with quote

pjp wrote:
The only way I'll do 15 minute snapshots is if I can skip performing them if there are no changes. And I don't know if that is possible with rsync.
Yeah, I wouldn't try it with rsync either, it's too slow for a start. ZFS snapshots are atomic, extremely fast, and consume no space if there are no changes.
You could probably snapshot every 10 seconds if you wanted, when you pay the COW performance tax upfront snapshots are essentially free.

Code:
time zfs snapshot tank@test

real    0m0.395s
user    0m0.008s
sys     0m0.014s


pjp wrote:
When the daily snapshot is completed, then the 15min/hourly or whatever increment snapshots would be cleaned up, depending on available space.
I just call zfs-auto-snapshot from cron, with --keep= set to an appropriate number.
So rolling windows for each snap interval - 15 minute snapshots going back a day, daily snapshots going back a month, whatever suits on a per-dataset basis. It's pretty much fire-and-forget TBH.
_________________
Once is happenstance. Twice is coincidence. Three times is enemy action. Four times is Official GNOME Policy.
Back to top
View user's profile Send private message
Anon-E-moose
Advocate
Advocate


Joined: 23 May 2008
Posts: 4866
Location: Dallas area

PostPosted: Mon Aug 24, 2020 10:03 am    Post subject: Reply with quote

pjp wrote:
As for space, those would all be temporary., and I'd probably skip certain files or directories.


I do use the exclude option of rsync, there's always directories that are full of "tmp" files, etc that I don't care about backing up or they're in a category that I backup separately.

Code:
rsync -aix --delete --exclude={"music/","movies/","music-video/","tv/","torrents/","download/","don/virtual/","don/git/","gentoo/"} root@don1:/n/ /mnt/backup/don1/n


/n drive is my spinning rust hold lots of things drive :lol:
_________________
PRIME x570-pro, 3700x, RX 550 - 5.8 zen kernel
Acer E5-575 (laptop), i3-7100u - i965 - 5.5 zen kernel
---both---
gcc 9.3.0, profile 17.1 (no-pie & modified) amd64-no-multilib, eudev, openrc, openbox, palemoon
Back to top
View user's profile Send private message
pjp
Administrator
Administrator


Joined: 16 Apr 2002
Posts: 18556

PostPosted: Mon Aug 24, 2020 2:58 pm    Post subject: Reply with quote

@steve_v:

Yes, I'd prefer ZFS. :( But I still want to go through patch maintenance on kernel upgrades to see if it is worth it. I don't want to be relying on ZFS and decide it isn't a long term solution. I'm almost willing to use a "kitchen sink" kernel just to avoid having to maintain one at all. And the location of the snapshots on an hourly basis or less would be on my workstation / primary "server". I don't expect rsync to work for that.


@Anon-E-moose:

I'm mainly thinking of easily reproducible stuff. Either because I build it or download it, but also of some arbitrarily large size that I haven't yet defined. --max-size=SIZE seems to be the solution if the concept of snapshots is otherwise workable. It might be reduced to sending the rsync file list to tar, which would probably need to have the max size in single digit MB.



The overnight transfer finished a little while ago with only one error.
Code:
Time value of <path/to/file>  truncated on receiver.
Unfortunately not much available on that error, other than the obvious. All 3 local copies are in agreement with a wonky modify time in 2414. rsync modified it on the backup host to 2006. I don't like tools that do that. I use options "-avzhHEi" and -a includes -t "preserve time" which it did not preserve. Unfortunately sftp didn't either, and scp didn't preserve any times. Maybe the problem is due to the backup system currently being booted using sysresccd 5.0.4. Hopefully.
_________________
Your lips move, but I can't hear what you're saying.
Back to top
View user's profile Send private message
Anon-E-moose
Advocate
Advocate


Joined: 23 May 2008
Posts: 4866
Location: Dallas area

PostPosted: Mon Aug 24, 2020 3:33 pm    Post subject: Reply with quote

This was a response to a question someone asked about rsync and weird time values.

Quote:
Furthermore, it's outside the range 1970–2038, which is the range of dates that are universally supported.

Then he mentions that ext4 will allow values from 1901–2514, so if on that filesystem, then the value is correct as far as its concerned, but lots of software won't understand it. I don't know what ranges are valid for other filesystems.
_________________
PRIME x570-pro, 3700x, RX 550 - 5.8 zen kernel
Acer E5-575 (laptop), i3-7100u - i965 - 5.5 zen kernel
---both---
gcc 9.3.0, profile 17.1 (no-pie & modified) amd64-no-multilib, eudev, openrc, openbox, palemoon
Back to top
View user's profile Send private message
pjp
Administrator
Administrator


Joined: 16 Apr 2002
Posts: 18556

PostPosted: Mon Aug 24, 2020 5:12 pm    Post subject: Reply with quote

ext4 for what I'm using. So now I'm wondering when ext4 gained that support. wikipedia says 1901 - 2446 and references the following, so that does seem to be the issue, not rsync:
Quote:
Some older kernels might have written pre-1970 dates with 1,1 in the
extra bits. This patch treats those incorrectly-encoded dates as
pre-1970, instead of post-2311, until kernel 4.20 is released.
sysrescuecd uses 4.9.

As long as rsync isn't the cause, I'm good. I don't actually care about that date (or the file, really, it'll eventually be removed), but I didn't want a tool I couldn't trust.

Thanks for finding that, I hadn't seen that information!
_________________
Your lips move, but I can't hear what you're saying.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Gentoo Chat All times are GMT
Goto page Previous  1, 2
Page 2 of 2

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum