How to handle umount command failure

Answer

Introduction
Directories
Use fuser and lsof to find files opened by processes
Use genkex and genkld to find files opened by the kernel
Use kdb to find files opened by the kernel
Conclusion

Introduction

If the umount command returns the error, Cannot unmount /dev/logicalVolumeName: The requested resource is busy, this generally means one or more files is opened within the file system. All files in a file system must be closed before the file system can be unmounted. In most cases, files are opened within a file system because one or more processes is currently running with open references to the files. To close the files, these processes must either be terminated, or the programs must be directed to close their open files. Files can also be opened by the AIX kernel, and will not be associated with a specific process. Two types of files that can be opened by the kernel are kernel extensions and shared libraries. The procedure for closing files opened by the kernel varies depending on the type of file. The techniques described in this document, and in Technote T1000188 can be used to find open files and close them so that the file system can be unmounted without having to reboot the system.

Directories

Directories are simply special types of binary files which contain references to other files and directories, and are opened and read by commands such as ls and the shell. Like regular files, directory files must be closed before a file system can be unmounted. Two common examples of open directory files are the current working directory, and directories used as mount points for other file systems.

Current working directory

Setting the current working directory with the cd command will create an open reference to the directory within the shell, so the current working directory should not be set to the mount point for a file system, or to any subdirectory beneath the mount point, before attempting to unmount the file system. Use the pwd command to display the current working directory. For example, if you are attempting to unmount the file system with the mount point /lgfs, and your current working directory is /lgfs/myfiles, you will not be able to unmount the file system.

# pwd
/lgfs/myfiles
# umount /lgfs
Cannot unmount /dev/fslv00: The requested resource is busy.
# cd /
# umount /lgfs
#

Mount point directories

File systems mounted on directories located beneath the mount point for an enclosing file system create an open reference to the mount point directory. Before attempting to unmount a file system, all file systems mounted within that file system must first be unmounted. Use the mount command to view the mount points for all currently mounted file systems.

Use fuser and lsof to find files opened by processes

In most cases, files are opened within a file system because one or more processes have open references to the files. To close the files, these processes must either be terminated, or the programs must be directed to close their open files. If the open file reference is an executable file and the process associated with the executable file is running, the process must be terminated. The fuser and lsof commands can be used to find processes with open file references. The use of these commands to help resolve a problem when unmounting file systems is documented in Technote T1000188.

Use genkex and genkld to find files opened by the kernel

Kernel extensions

Kernel extensions are stored as binary files and are used to extend the capabilities of the kernel. Kernel extensions most often reside in /usr/lib/drivers, but might also exist in other directories outside of /usr. Use the genkex command to view all currently loaded kernel extensions. This example finds kernel extensions that are loaded from files in the /etc/drivers directory.

# genkex | grep " /etc"
         4175000    12020 /etc/drivers/idecdrompin
         40b3000     3678 /etc/drivers/coreprobe.ext
         4037000    3acf0 /etc/drivers/hd_pin_bot
         3fd5000    39540 /etc/drivers/scsidiskpin

See Technote T1000188 for more information about unloading kernel extensions.

Shared libraries

Shared libraries are also stored as binary files and are loaded by the kernel for use by one or more processes. Shared libraries can be located anywhere on the system. Use the genkld command to view all currently loaded shared libraries. This command lists the path to the file so the output can be piped to grep to search for the mount point of the file system you are are trying to unmount. However the path might be listed as a relative path, in which case the mount point might not be included in the path, and thus a grep for the mount point will not return a hit. For example, if the shared library /lgfs/slib/libmyshr.a was loaded with a relative path, the output from genkld might appear as follows:

# genkld | grep /lgfs
#
# genkld | grep libmyshr.a
        d01e4100      6da ./libmyshr.a[shr.o]

If relative paths are listed in the output from genkld and you suspect the file system you are trying to unmount might have an open reference to a shared library, use the find command to search for the file name within the file system. Or, use the kdb command as described in the next section below.

# find /lgfs -name libmyshr.a
/lgfs/slib/libmyshr.a

If genkld shows a loaded shared library that is located within the file system you are trying to umount, you will need to unload the shared library to release the open file reference. To do this you must first terminate all processes using the shared library, and then run the slibclean command to unload the shared library. You can use the genld -l command to search for processes using a specific shared library. For example, to find all processes using the shared library libpthreads.a, you can run the following command:

# genld -l | grep -p libpthreads.a | grep Proc_name
Proc_pid: 1         Proc_name: init
Proc_pid: 86158     Proc_name: rpc.statd
Proc_pid: 94294     Proc_name: dtlogin
Proc_pid: 110712    Proc_name: syncd
Proc_pid: 155652    Proc_name: hostmibd64
Proc_pid: 192682    Proc_name: sendmail
Proc_pid: 204964    Proc_name: radiusd
Proc_pid: 209040    Proc_name: portmap
Proc_pid: 217254    Proc_name: radiusd
Proc_pid: 221350    Proc_name: java
Proc_pid: 225440    Proc_name: radiusd
Proc_pid: 270472    Proc_name: xmwlm
Proc_pid: 278768    Proc_name: genld
Proc_pid: 299194    Proc_name: xmtopas
Proc_pid: 327866    Proc_name: IBM.DRMd

If the command above does not return any output, but genkld shows the library is loaded, it means the library is loaded even though there are no processes currently using the library. Run slibclean to unload the library and release the open reference to the library file.

Use kdb to find files opened by the kernel

If the fuser, lsof, genkex, and genkld commands have not helped locate open files within a file system, you can use the kernel debugger kdb command.

1. Run kdb as root.
   # kdb
   The specified kernel file is a 64-bit kernel
   Preserving 1417366 bytes of symbol table
   First symbol __mulh
              START              END <name>
   0000000000001000 0000000003E10050 start+000FD8
   F00000002FF47600 F00000002FFDC940 __ublock+000000
   000000002FF22FF4 000000002FF22FF8 environ+000000
   000000002FF22FF8 000000002FF22FFC errno+000000
   F100070F00000000 F100070F10000000 pvproc+000000
   F100070F10000000 F100070F18000000 pvthread+000000
   PFT:
   PVT:
   id....................0002
   raddr.....0000000000725000 eaddr.....F200800021B00000
   size..............00040000 align.............00001000
   valid..1 ros....0 fixlmb.1 seg....0 wimg...2
   (0)>
   
   
2. Run the vfs command. This command will list all currently mounted file systems.
   (0)> vfs
                              GFS             DATA TYPE    FLAGS
   
     1 F1000100144CA4B0 0149FD58 F10001001540D080 JFS2    DEVMOUNT
   ... /dev/hd4 mounted over /
     2 F1000100144CA550 0149FD58 F1000100153D8480 JFS2    DEVMOUNT
   ... /dev/hd2 mounted over /usr
     3 F1000100144CA5F0 0149FD58 F100010015344480 JFS2    DEVMOUNT
   ... /dev/hd9var mounted over /var
     4 F1000100144CA410 0149FD58 F100010015402C80 JFS2    DEVMOUNT
   ... /dev/hd3 mounted over /tmp
     5 F1000100144CA730 0149FD58 F1000100185F1480 JFS2    DEVMOUNT
   ... /dev/hd1 mounted over /home
     6 F10001001640C410 0149FDF8 0000000000000000 PROCFS
   ... /proc mounted over /proc
     7 F10001001640C4B0 0149FD58 F1000100185F1880 JFS2    DEVMOUNT
   ... /dev/hd10opt mounted over /opt
     8 F10001001640C550 0149FD58 F100010018601C80 JFS2    DEVMOUNT
   ... /dev/fslv00 mounted over /lgfs
     9 F1000100144CA690 0149FE98 F10001001879C000 NFS3    REMOTE
   ... /software mounted over /software
   
   
3. From the vfs output in step 2, find the index number in the first column for the file system that will not unmount.
   
   
4. Run the following command where N is the value of the index number for the file system.
   (0)> vfs N | awk '{ if ($3 > 0) print $1 " " $3 " " $4 " " $6 }' | more
   
   In this example, the file system which that not unmount is /lgfs with an index of 8.
   (0)> vfs 8 | awk '{ if ($3 > 0) print $1 " " $3 " " $4 " " $6 }' | more
   
   In the output from vfs, search for the text COUNT.  Below the heading is a list of regular files and directory files that are open in the file system. The 2nd column is a count of the number of open paths to the file, and the 3rd column is the address of the gnode for the file. In this example output, there is one regular file and two directories that are open in the file system /lgfs.
   
   ...
   ...
         COUNT             VFSP TYPE
    2     2 F10001001C7944B0  REG
   13     4 F10001001C7C28B0  DIR
   38     1 F100010018611CB0  DIR
   ...
   ...
   
   
5. Choose a file in the list above and run the gnode command on the gnode number.
   (0)> gnode F10001001C7944B0 | egrep "gn_seg|gn_data"
   gn_seg........ 0000000000002E29
   gn_filocks.... 0000000000000000 gn_data....... F10001001C794480
   
   Note the gn_seg and the gn_data values.
   
   
6. Run the scb command with the gn_seg value as an argument. Look for pvproc in the output. If the number is 0, it either means the process has already been killed but the file remains open, or there is no process associated with the open file, meaning the file has been opened by the kernel. If the number is > 0, it means there is a process associated with the file. In most cases, if you have already used fuser and lsof, you will not find processes associated with open files when using kdb.
   (0)> scb 2 0000000000002E29 | grep pvproc
   pvproc pointer           (pvproc)   : 0000000000000000
   
   
7. If the pvproc pointer is > 0, run the following command to find the PID for the process associated with the file.
   (0)> proc pvprocPointer
   
   The PID will be in hex. Convert this to decimal and use the PID to find the process with the ps -efk command after quitting kdb. But before quitting kdb, follow the next step to find the name of the open file.
   
   
8. To obtain the inode for the open file, run the inode command for a JFS file, or the i2 command for a JFS2 file, and pass the gn_data value as an argument (gn_data is listed in the output of step 5 above. The type of file is listed in the TYPE column from step 2).
   (0)> i2 F10001001C794480
       ADDRESS           DEVICE           I_NUM    COUNT   TYPE    FLAG
   F10001001C794480 8000000A0000000A      45131    00002   VREG
   ...
   ...
   
   
9. Quit kdb and use the find command with the inode number to find the name of the file.
   (0)> q
   # cd /lgfs
   # find . -inum 45131
   ./slib/libmyshr.a
   
   
10. In this example we see the open file is a shared library file.
    # genkld | grep libmyshr.a
            d01e4100      6da ./libmyshr.a[shr.o]
    
    Find which processes are using the shared library.
    # genld -l | grep -p libmyshr.a | grep Proc_name
    #
    
    There are no processes currently using this shared library so the library can now be removed with slibclean.
    # slibclean
    
    Note: slibclean will only remove a loaded shared library if there are no processes currently using the shared library.
    
    Verify that the shared library has been unloaded.
    # genkld | grep libmyshr.a
    #
    
    Repeat steps 5 - 9 for all files listed in step 4 and take the appropriate action to close the file references before attempting to unmount the file system.