Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
[SOLVED] Heavy disk writes cause swapping? Why?
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware
View previous topic :: View next topic  
Author Message
ExZombie
Apprentice
Apprentice


Joined: 29 May 2004
Posts: 170

PostPosted: Fri Oct 02, 2020 1:57 pm    Post subject: [SOLVED] Heavy disk writes cause swapping? Why? Reply with quote

Hello folks,

Just when I thought that the bad old days of GUI stuttering or the system even becoming totally unresponsive when writing to a slow medium were behind us, I'm shocked to find that this crap still happens, although now it's weirder than ever. Needless to say, searching the intarwebz on this topic yields tons of results from the beginning of the decade, but nothing that would be relevant for my particular case.

In short, when writing a lot of data, GUI starts stuttering. How to reproduce (I hope someone can):


  • start atop in a terminal
  • create a 100M random file in memory, e.g.
    Code:
    dd if=/dev/urandom of=/tmp/somedata bs=1M count=100

  • replicate the data 100 times (yielding 10G) and write it to disk
    Code:
    yes /tmp/somedata | head -n100 | xargs cat | pv -s 10G > ~/lotsofdata

  • wait a while (depending on how much RAM you have), GUI starts stuttering (including mouse pointer!), atop reports a large number of swout in the PAG line
  • disable swap with
    Code:
    swapoff -a

  • repeat the exercise, see the data written out with no issues


So, what the heck? Why does writing a file cause active programs to be paged out? My system has


  • 16G RAM, of which 6.5G was used by programs was used during the test
  • Swap was mostly empty during the whole test, only 50M-ish was used
  • NVME drive, so this is not the slow-as-molasses-USB-stick issue from a decade ago
  • I have LUKS+LVM+ext4 set up, I don't know if this affects anything
  • I have ruled out the effects of the usual suspects (dirty_ratio, dirty_background_ratio, vfs_cache_pressure)
  • swappiness was set to 0


So, any ideas? I would prefer to have swap enabled, because (a) it is required for hibernation; (b) sometimes, I run 3 VMs and if I then start compiling C++ code, well … it's good to have swap :lol:


Last edited by ExZombie on Sun Oct 11, 2020 10:47 am; edited 1 time in total
Back to top
View user's profile Send private message
alamahant
Guru
Guru


Joined: 23 Mar 2019
Posts: 557

PostPosted: Fri Oct 02, 2020 10:05 pm    Post subject: Reply with quote

Decrease the swappiness value in
/proc/sys/vm/swappiness
First get your current value and the echo a lower value to the file
Code:

cat /proc/sys/vm/swappiness
echo <lower-value-maybe-10-or-20> > /proc/sys/vm/swappiness

If you are satisfied you can make it permanent by making an entry in
/etc/sysctl.d/swappiness.conf
Code:

vm.swappiness=<10-or-20-or-something-low>

Lower values mean your system will be less inclined to use swap.
:D
Back to top
View user's profile Send private message
Etal
Veteran
Veteran


Joined: 15 Jul 2005
Posts: 1829

PostPosted: Fri Oct 02, 2020 10:26 pm    Post subject: Reply with quote

alamahant wrote:
Decrease the swappiness value in
/proc/sys/vm/swappiness


OP said swappiness was set to 0... you can't decrease it any further ;-)
Back to top
View user's profile Send private message
szatox
Veteran
Veteran


Joined: 27 Aug 2013
Posts: 1962

PostPosted: Sat Oct 03, 2020 12:21 am    Post subject: Reply with quote

I suppose that you don't have 10G of RAM available, but with swap on your cat thinks you do and tries to swallow that file 100 times exceeding memory.
And since you have some swap space, kernel is trying to satisfy all allocation requests.

In the second case cat may notice there is not enough RAM to swallow all the data at once ( E.g. there is no RAM so allocating it fails), so the head of your cat is long gone by the time you see it's tail. I'd expect OOMKiller to kick in and put your pipeline down at this point though.
Anyway, try to find out how many files cat opens in parallel. (hint: ls /proc/$pid/fd), maybe that would shed some light.
Back to top
View user's profile Send private message
Maitreya
Guru
Guru


Joined: 11 Jan 2006
Posts: 440

PostPosted: Sat Oct 03, 2020 8:50 am    Post subject: Reply with quote

I thought swapiness 0 meant "not using disk".
And using 1 would be the lowest value for disk swapping.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 46746
Location: 56N 3W

PostPosted: Sat Oct 03, 2020 11:47 am    Post subject: Reply with quote

ExZombie,

The kernel has several ways to swap without using a swap partition.
The swap partition is only ever used for dynamically allocated RAM. Everything else has a permanent home home on disk and can either be flushed to disk and reloaded , or if its code, just dropped and reloaded.

Your write fills all of RAM with dirty buffers. They have a permanent home on disk so will not be written to swap, that would just be silly. Writing them to the wrong place.

The kernel needs more RAM for for your huge write, so it drops code, that's on disk and can be reloaded.
Now you begin to notice. Some of the code for your running apps has been dropped to make way for your huge write.
RAM is full of dirty buffers, waiting to be flushed to disk. Some of this flushing needs to happen to reload the code you want to execute.
Everything slows down due to RAM thrashing.

I'm not sure what swapiness controls. If its just the paging of dynamically allocated RAM, that only delays the inevitable.

The OOM won't kick in until the kernel wants some RAM for itself and there is none.
Then it will kill something.

That's what's happening to your system. I don't know the fix.
You can achieve the same effect writing a file that's too big for RAM to any device that can't keep up with the rate data becomes available in RAM.
Although, last time I wrote half a dozen Bluray rips to my media server over NFS, I didn't see the problem.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
Etal
Veteran
Veteran


Joined: 15 Jul 2005
Posts: 1829

PostPosted: Sat Oct 03, 2020 4:01 pm    Post subject: Reply with quote

Maitreya wrote:
I thought swapiness 0 meant "not using disk".
And using 1 would be the lowest value for disk swapping.


That's not correct. According to kernel docs:

Quote:
swappiness

This control is used to define how aggressive the kernel will swap
memory pages. Higher values will increase aggressiveness, lower values
decrease the amount of swap. A value of 0 instructs the kernel not to
initiate swap until the amount of free and file-backed pages is less
than the high water mark in a zone.


Not sure what the "high water mark in a zone" is.
Back to top
View user's profile Send private message
ExZombie
Apprentice
Apprentice


Joined: 29 May 2004
Posts: 170

PostPosted: Sun Oct 04, 2020 7:34 am    Post subject: Reply with quote

NeddySeagoon wrote:

The kernel has several ways to swap without using a swap partition.
The swap partition is only ever used for dynamically allocated RAM. Everything else has a permanent home home on disk and can either be flushed to disk and reloaded , or if its code, just dropped and reloaded.

Your write fills all of RAM with dirty buffers. They have a permanent home on disk so will not be written to swap, that would just be silly. Writing them to the wrong place.


Yeah, I know what you mean; it's a good and insightful guess, but I'm afraid that's not it. If the problem was just paging out of executables and libraries et al., I think the problem would persist even when the swap partition is disabled. It does not, it goes away if I disable swap. Moreover, the atop man page explicitly states that the swout number is about swap space.
Back to top
View user's profile Send private message
ExZombie
Apprentice
Apprentice


Joined: 29 May 2004
Posts: 170

PostPosted: Sun Oct 04, 2020 7:39 am    Post subject: Reply with quote

szatox wrote:
I suppose that you don't have 10G of RAM available, but with swap on your cat thinks you do and tries to swallow that file 100 times exceeding memory.


cat doesn't work like that (AFAIK none of these tools buffer more data than needed for stream IO), and anyway the amount of total RAM used does not change while the command is running. I don't think userspace us the problem here.
Back to top
View user's profile Send private message
Maitreya
Guru
Guru


Joined: 11 Jan 2006
Posts: 440

PostPosted: Mon Oct 05, 2020 11:17 am    Post subject: Reply with quote

Etal wrote:
Maitreya wrote:
I thought swapiness 0 meant "not using disk".
And using 1 would be the lowest value for disk swapping.


That's not correct. According to kernel docs:

Quote:
swappiness

This control is used to define how aggressive the kernel will swap
memory pages. Higher values will increase aggressiveness, lower values
decrease the amount of swap. A value of 0 instructs the kernel not to
initiate swap until the amount of free and file-backed pages is less
than the high water mark in a zone.


Not sure what the "high water mark in a zone" is.


Did you try it?
As far as my experiments go, 0 = disable , 1 = minimal.
Back to top
View user's profile Send private message
ExZombie
Apprentice
Apprentice


Joined: 29 May 2004
Posts: 170

PostPosted: Mon Oct 05, 2020 12:43 pm    Post subject: Reply with quote

Maitreya wrote:

Did you try it?
As far as my experiments go, 0 = disable , 1 = minimal.


Yes. Sadly, no difference in my case :(
Back to top
View user's profile Send private message
ExZombie
Apprentice
Apprentice


Joined: 29 May 2004
Posts: 170

PostPosted: Mon Oct 05, 2020 6:53 pm    Post subject: Reply with quote

Etal wrote:
Not sure what the "high water mark in a zone" is.


Well, you can see the value in /proc/zoneinfo and yeah, with vm.swappiness at 0, swapping will start when the number of free pages falls below it. So now, I found a partial mitigation that makes the behavior "softer; thanks for the hint!

  • Increase vm.watermark_scale_factor to 100 or more (but 1000 is a bit much …), which makes the low and high watermarks more widely spaced
  • Enable zswap

This makes the lag and jerkiness bearable, kind of. But the question remains: why does the file cache cause useful pages to be pushed to swap for no reason?
Back to top
View user's profile Send private message
ExZombie
Apprentice
Apprentice


Joined: 29 May 2004
Posts: 170

PostPosted: Sun Oct 11, 2020 10:47 am    Post subject: Reply with quote

Heh. I upgraded from kernel 5.4 (current stable) to 5.8, and the issue is gone. Monitoring with atop shows that swout is non-zero only occasionally. Machine doesn't freeze, mouse pointer remains smooth. Yay!

Thanks everyone for you input :)
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum