Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
umount hangs on network shares during a reboot [SOLVED]
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Networking & Security
View previous topic :: View next topic  
Author Message
don quixada
l33t
l33t


Joined: 15 May 2003
Posts: 744

PostPosted: Sat Jul 25, 2020 12:08 am    Post subject: umount hangs on network shares during a reboot [SOLVED] Reply with quote

Hi, can anyone please point me in the right direction? I've searched around the web and these forums but a solution isn't apparent. The problem I'm having is this:

When my system reboots and there are network shares that are already detached (from the server side) it just hangs on "Unmounting network shares" and apparently there is an infinity timeout. This results in me needing to power cycle the machine in the middle of the shutdown sequence which obviously isn't very good especially since the hard drives haven't been unmounted yet.

Is there a way to add a timeout that goes to a lazy unmount or something like that. I found posts from 15 years ago but everything has changed since then so they weren't very helpful. Thanks in advance!

dq


Last edited by don quixada on Sun Jul 26, 2020 6:26 pm; edited 2 times in total
Back to top
View user's profile Send private message
Ralphred
Tux's lil' helper
Tux's lil' helper


Joined: 31 Dec 2013
Posts: 117

PostPosted: Sun Jul 26, 2020 10:38 am    Post subject: Reply with quote

With the default behaviour it will try to unmount an indefinite number of times, you can stop this and limit it to 2 retries (3 for UDP mounts) by specifying the soft option on mount.
Once soft is set, there are 3 things that determine the actual timeout:
    the timeo mount option (defaults to 600, or 1 minute)
    the retrans mount option (defaults to 2)
    the kernels tcp_syn timeout (governed by the kernel)

So we end up with a timeout of tcp_syn_timeout*retrans for unmounting when the server is no longer available, or timeo*retrans when the server is available but the exports are not.

The easiest thing to do is specify the options in fstab, such as
Code:
10.0.0.1:/media/films                           /media/films            nfs     timeo=60,soft                        0 0

and create a file to lower the number of tcp_syn_retries before shutdown
Code:
echo -e '#!/bin/bash\necho 1 > /proc/sys/net/ipv4/tcp_syn_retries' >/etc/local.d/tcp_syn_retries.stop
chmod +x /etc/local.d/tcp_syn_retries.stop


If the exports aren't always accessible (like on a laptop when you are away from home) then add noauto to the options, or bg as I do (bg will cause a timeo delay on boot per mount before backgrounding the mount process)
If you mount them read/write then you should read the manpage for nfs about the soft option, there are data corruption implications (that IMHO are less than those of hard resetting a pc during shutdown, and seem on par with umount -fl TBQH)
Back to top
View user's profile Send private message
don quixada
l33t
l33t


Joined: 15 May 2003
Posts: 744

PostPosted: Sun Jul 26, 2020 1:56 pm    Post subject: Reply with quote

Thanks for your reply. Does this also work for cifs/Samba? Or are there different flags?

dq
Back to top
View user's profile Send private message
Ralphred
Tux's lil' helper
Tux's lil' helper


Joined: 31 Dec 2013
Posts: 117

PostPosted: Sun Jul 26, 2020 3:00 pm    Post subject: Reply with quote

don quixada wrote:
Does this also work for cifs/Samba

cifs does have some similar flags, like echo_interval instead of timeo, and it will also be affected by tcp_syn_retries.

If it still hangs post again, I have a script that lazy force unmounts shares/exports from dead servers, it's a bit meh as it works by pinging, but you could run that at shutdown in a pinch.
Back to top
View user's profile Send private message
don quixada
l33t
l33t


Joined: 15 May 2003
Posts: 744

PostPosted: Sun Jul 26, 2020 6:25 pm    Post subject: Reply with quote

Hmm, well it works now, although it think it was your script because even though I enabled the cifs flags (echo_interval=60 and soft) in fstab, those flags are enabled by default. Anyway, thank you for the script things are much better now.

dq
Back to top
View user's profile Send private message
proteusx
Apprentice
Apprentice


Joined: 21 Jan 2008
Posts: 298

PostPosted: Mon Jul 27, 2020 8:29 am    Post subject: Reply with quote

Ralphred wrote:
I have a script that lazy force unmounts shares/exports from dead servers, it's a bit meh as it works by pinging, but you could run that at shutdown in a pinch.


I would very much like to see your script.
Back to top
View user's profile Send private message
Ralphred
Tux's lil' helper
Tux's lil' helper


Joined: 31 Dec 2013
Posts: 117

PostPosted: Mon Jul 27, 2020 7:26 pm    Post subject: Reply with quote

proteusx wrote:
I would very much like to see your script.

Code:
#!/bin/bash
#Reads /proc/mounts and performs a lazy, forced un-mount of any nfs file systems if the server does not respond to ping
cat /proc/mounts >/tmp/NFSumount_mounts.tmp

DEBUG=false
PRETEND=false

while read line;do
        if ! echo $line |awk '{print $3}'| grep -wq "nfs4\?";then
                continue
        fi
        mountpoint=$(echo ${line}|awk '{print $2}')
        if $DEBUG;then echo -en "NFS mount at ${mountpoint} found, ";fi
        server=$(echo ${line}|grep -wo "addr=[0-2]\?[0-9]\?[0-9]\.[0-2]\?[0-9]\?[0-9]\.[0-2]\?[0-9]\?[0-9]\.[0-2]\?[0-9]\?[0-9]"|sed 's/addr=//')
        if $DEBUG;then echo -en "server is ${server}. ";fi
        if ! ping -w5 -i0.2 -c10 ${server} >/dev/null;then
                if $DEBUG;then echo -ne "$server Did not respond to ping. ";fi
                echo "Unmounting \"${mountpoint}\""
                if ! $PRETEND;then umount -fl ${mountpoint};fi
        else
                if $DEBUG; then echo $server responded to ping;fi
        fi

done < /tmp/NFSumount_mounts.tmp

rm /tmp/NFSumount_mounts.tmp

It's ... applied lazyness. Cifs stores addr=[server ip addr] in /proc/mounts too, so if you changed the grep string to read "nfs4\?\|cifs" it should work for that, but there is no umount.cifs or associated man page, so testing with PRETEND and DEBUG set to true a few times to make sure it's doing what's expected is a must. I used to use it until I stumbled across the tcp_syn_timeout thingy a while back. Before that is was in /etc/local.d/nfs_mounts.stop. It's still there as nfs_mounts.redundant, because looking back at old scripts and seeing how less hacky your newer scripts are is nice :)

Quote:
even though I enabled the cifs flags (echo_interval=60 and soft) in fstab, those flags are enabled by default

Yeah, 'soft' is different, kind of, but echo_interval= is in seconds whereas timeo= is in 10ths of seconds so if I was trying to apply the sameish rules I'd set echo_interval=6.
It's good to know that lowering syn_retries on shutdown helps though, thx for the feedback.
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 16187

PostPosted: Tue Jul 28, 2020 2:35 am    Post subject: Reply with quote

Ralphred wrote:
Code:
cat /proc/mounts >/tmp/NFSumount_mounts.tmp
Do you need a temporary file here? Could this be read directly into the while loop, or would that cause problems with the mount list changing under the script?
Ralphred wrote:
Code:
while read line;do
        if ! echo $line |awk '{print $3}'| grep -wq "nfs4\?";then
                continue
        fi
Since you skip lines that fail this test, why not have this pipeline filter your temporary file and only while read the lines that satisfy the pipeline?
Code:
awk '$3 ~ /\<nfs4?\>/ { print $2, $4; }' < /tmp/NFSumount_mounts.tmp | while read mountpoint mountopts; do
Ralphred wrote:
Code:
        mountpoint=$(echo ${line}|awk '{print $2}')
With my change above, this becomes unnecessary.
Ralphred wrote:
Code:
        server=$(echo ${line}|grep -wo "addr=[0-2]\?[0-9]\?[0-9]\.[0-2]\?[0-9]\?[0-9]\.[0-2]\?[0-9]\?[0-9]\.[0-2]\?[0-9]\?[0-9]"|sed 's/addr=//')
You can use sed to search for the appropriate pattern, so there is no need for grep here. Also, note that your filter will miss IPv6 mounts. Supporting IPv6 requires more than just changing the address filter though, since you would need to use ping6 to probe an IPv6 address.
Back to top
View user's profile Send private message
proteusx
Apprentice
Apprentice


Joined: 21 Jan 2008
Posts: 298

PostPosted: Tue Jul 28, 2020 7:47 am    Post subject: Reply with quote

@Ralphred
Thank you. Your script is much better than my simplistic
Code:
umount -fl -t nfs,nfs4 
which does not always work.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Networking & Security All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum