Filesystems mount options and side effects on APM v8.1.x

Body

Linux filesystems are known to be reliable,resilient and suitable for many different types of scenarios.
The big amount of mount options also provides a consistent way to implement secure environments.
But you need to know that those mount options may have unexpected side effects on some applications, so it is better you learn something more about them.
When you install a Linux environment from scratch, the default options used for mounted filesystems are not expected to cause troubles, because the most "restrictive" flags are not used.

But for sure on production environments it can happen your sysadmin is used to mount filesystems using flags, like noexec or nosuid, that in specific scenario can drive applications to misbehave or to crash.

In this blog article I will show two different cases where the aforementioned flags caused problems to APM v8.1.x
Actually the problems resulted to be with services used by APM more than with APM itself, the first one with DB2, the second with MongoDB.

 

---

APM Installation fails with message : DBI20043E  DB2 failed to install because of restrictive permission settings on the temporary directory /tmp

 

During APM installation, we got message DBI20043E when the installer tried to run DB2 installation program.
Even launching manually the db2setup we received immediately the same message.
We verified that SELInux was disabled and directory /tmp had the correct permission settings.
From previous experience, I knew this message can be issued when the /tmp filesystem is mounted with flag "noexec", but it was not this case.
All looked fine with /tmp itself.
After some investigation, I found this message was indeed misleading.
The root cause is described in this DB2 APAR: IT21713.
-----------------
Description:
In the db2setup script, the test for -noexec mount on the /tmp or DB2TMPDIR will fail if any subdirectory has the -noexec mount, even directories of no interest to DB2.

This error will be present in the install.log:

DBI20043E  DB2 failed to install because of restrictive permission settings on the temporary directory /tmp

The mount command involved lists all the filesystems present (/tmp , /var/tmp, /abcd/bcf/tmp etc)and the grep command currently used will check for all directories that have /tmp as
subdirectory including /tmp instead of specifically /tmp or DB2TMPDIR.  This can cause a customer's installation to incorrectly fail.
-----------------

This APAR is addressed in DB2 v10.5 fp9 or higher.
We implemented the workaround: find and unmount the directory(es) with the -noexec option, and remount it after installation is completed.
Of course the one with -noxec option was not the usual /tmp because we previously already checked it.
There was another subdirectory under /var (so it was /var/tmp) mounted with flag noexec.
Unmounting that filesystem and mounting it again without noexec option allowed the DB2 installation to complete and after that also all the other APM
components have been correctly installed.

So in case you get error DBI20043E during APM installation, extend your verification to other "tmp" folders and related filesystem mount options, do NOT check only the filesystem for /tmp.

MongoDB service does not start.

After a succesfull APM installation, all the services are up and running but MongoDB.
Looking at the mongod.log and at the startup.log we can read:

---------
forked process: 1226808
ERROR: child process failed, exited with error number 48
---------

This error has been observed in the past when mongod process was started with root instead of mongod userID.
In a previous case, this occurred because customer removed the suid bit from mongod binary file.

So this was the very first check we wanted to perform.

Listening the mongod file into /opt/ibm/mongod/bin, we anyway verified that the suid bit was there.

 

-rwsr-sr-x. 1 mongod mongod 36233039 Feb 1 2017 mongod

 

Despite of this, we found the process mongod started as root in the ps command output.
Looking at the /opt/ibm/mongodb path I noticed some files were created as root too, because the mongod process started as
root since the very first time after product installation.

This means that the operating system is not respecting the suid bit despite it is correctly set.
It starts the process with root user (the one we use to run the apm start command) and this caused the mongodb initialization to fail.

There is only one explanation for it to happen.

If the filesystem where the binary file with suid bit resides is mounted using flag "nosuid", then operating system
does not allow set-to-UID to take effect even if suid bit is set.

So, if the output of mount command shows something like:
 

/dev/mapper/opt_ibm_dev on /opt/ibm type xfs (rw,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)

 

it means that any file into APM home directory with suid bit set will not work as expected.

Considering that some files have been already created with wrong owner and that a mongod process was already running with the
wrong userID, I suggested those steps to recover:
 
1) Stop the mongod process if it is still running using the following procedure:

      1a) cd /opt/ibm/mongodb/bin && cp stop-mongodb.sh temp-stop-mongodb.sh
      1b) nano temp-stop-mongodb.sh
      1c) Change this line from
         mongodb_pid=`ps hU mongod -o pid,cmd | grep "$mongodb_home"/bin/mongod | awk '{print $1; exit}'
      to
         mongodb_pid=`ps hU root -o pid,cmd | grep "$mongodb_home"/bin/mongod | awk '{print $1; exit}'

      1d) ./temp-stop-mongodb.sh
 
Note:  the script should indicate that MongoDB is stopped.  Run the ps -ef | grep mongod  command to verify
 
2) Change file owners back to mongod
 
     2a) In install_dir/mongodb directory, if there are any files with owner of root, change owner back to mongod except for these files:
 
           .enabled
           .mongodb-activated
           .started  (if it exists)
           create_index.log
 
          Use this command to change the owner:
           chown mongod <file-name>
 
     2b) In the directories below, issue the chown command to change any files with owner of root back to owner of mongod:

      <apm_install_dir>/mongodb/data
      <apm_install_dir>/mongodb/data/journal
      <apm_install_dir>/mongodb/data/diagnostic.data  

      Command to use:
      chown mongod *
 
     2c) In install_dir/mongodb/logs, change any files with owner of root back to owner of mongod except for the startup.log file per this command
 
      chown mongod <file-name>
 
     2d) In <apm_install_dir>/mongodb/tmp,  delete the mongodb-27000.sock  file if it exists
 
     2e) Run this command
      ls -alR install_dir/mongodb > /tmp/mongodb-filelist.log
 
     2f) Search /tmp/mongodb-filelist.log for root and verify that the only files with root ownership are the exceptions listed above and the files in this
      directory :  <apm_install_dir>/mongodb/scripts/logs
 
3) Unmount the /opt/ibm

This was the filesystem name in our case, in your case of course you have to unmount the filesystem where APM installed, it may have a different name.

4) Mount it again without "nosuid" flag
Also ask sysadmin to modify the default option for this filesystem, otherwise when the system is restarted, it will be mounted again with nosuid
 
5)Start APM

 

After the above actions, mongodb started successfully and also APM was able to complete initialization.

As you can see, mount options noexec and nosuid can have a negative effects on APM if they are used for filesystems where APM is installed or that APM uses during execution, so pay attention to them.

If you want to know more about filesystems mount options, this man page is a good source of information.

http://man7.org/linux/man-pages/man8/mount.8.html

The information regarding mount options are in section called "FILESYSTEM-INDEPENDENT MOUNT OPTIONS"

Hope it helps.

 

 

Subscribe and follow us for all the latest information directly on your social feeds:

 

 

  

Check out all our other posts and updates:
Academy Blogs:	https://goo.gl/U7cYYY
Academy Videos:	https://goo.gl/TLfMoF
Academy Google+:	https://goo.gl/HnTs0w
Academy Twitter :	https://goo.gl/AhR8CL