IBM Support

Docker unresponsive after few minutes. Stopping Planning Analytics Workspace hangs. Containers are stuck in Restarting state.

Troubleshooting


Problem

On a previously working Planning Analytics Workspace environment, Docker is getting unresponsive after few minutes (usually about 30 minutes) whereas all containers are still shown as "Up" (except for bss-init, which normally shuts down itself shortly after Planning Analytics Workspace started up).
When we try to stop containers through Powershell, the whole shutdown process hangs eventually.
Even basic Docker commands are not responding.
After a forced restart of Docker service, the "/scripts/paw.ps1 ps" command is showing neo-idviz and neo-provision as "Restarting", but they never reach the "Up" state.
The Mongo container might be either in "Up" or "Exit" state, but in both cases, the "mongo.log" shows that Mongo DB crashed :
2020-01-27T18:02:48.796+0100 I CONTROL  [thread1] *** unhandled exception (access violation) at 0x00007FF9AD17F020, terminating
2020-01-27T18:02:48.796+0100 I CONTROL  [thread1] *** access violation was a DEP violation at 0x00007FF9AD17F020
2020-01-27T18:02:48.796+0100 I CONTROL  [thread1] *** stack trace for unhandled exception:
2020-01-27T18:02:48.980+0100 I CONTROL  [thread1]                 ???
2020-01-27T18:02:48.980+0100 I CONTROL  [thread1] KERNEL32.DLL    BaseThreadInitThunk+0x14
2020-01-27T18:02:48.980+0100 I -        [thread1]
2020-01-27T18:02:48.980+0100 I CONTROL  [thread1] writing minidump diagnostic file C:\mongodb\bin\mongod.2020-01-27T17-02-48.mdmp
2020-01-27T18:02:49.280+0100 I CONTROL  [thread1] *** immediate exit due to unhandled exception

Cause

This issue is between Docker and VMWare tools. It occurs since VMWare tools were updated to version 11.0.x (and up to 11.0.5). 

Environment

Windows Server 2016
Docker: any versions
Docker-Compose: any versions
Planning Analytics Workspace: any versions

Resolving The Problem

Here is a link that is referring to this docker issue:
However, the problem is not specific to Planning Analytics Workspace, it does affect the dependencies that Planning Analytics Workspace relies on.
Until april 2020, the solutions were either to disable "VMWare Tools" service, or to downgrade from 11.0.1 to a previous version like 10.3.x.
Now the issue with the VMware tools is resolved in VMware tools version 11.0.6. See release notes here:
The problem was caused by a new VMWare Tools plug-in called 'appInfo'. Here was the corresponding defect:
The recommendation is to upgrade to VMware tools version 11.0.6 or higher.
For getting the steps to downgrade or upgrade VMWare tools, contact VMWare Support.
In case a downgrade or an upgrade of VMWare Tools is not a quick and easy option, then here is an temporary solution:

Since the problem between Docker and VMware Tools seems to happen about 30 minutes after reboot:
- Create a Powershell script (.ps1) that contains this line:
stop-service vmtools 
- Open the Windows Task Scheduler
- Create a task named "Stop_vmtools"
- In the Trigger tab, configure the task to run "At startup", with a delay of 15 minutes. Indeed, it might be important to leave this service some time alive, since the IT team or VSphere administrators might need it someday to solve other issues.
- In the Actions tab, select "Start a program", type "Powershell" as the program name, and in the "Add arguments" field, just type the path and name of the stop_vmtools.ps1 script
- Test the scheduled task: start VMWare Tools service, run the task from the Task Scheduler (by right-click, "Run"), and verify this stops the VMWare Tools service.

This solution solves the compatibility issue between Docker and VMWare Tools, since the vmtools service is now stopped before it can interfere with Docker.

Document Location

Worldwide

[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSCTEW","label":"IBM Planning Analytics Local"},"Component":"Planning Analytics Workspace","Platform":[{"code":"PF033","label":"Windows"}],"Version":"All Versions","Edition":"","Line of Business":{"code":"LOB10","label":"Data and AI"}}]

Document Information

Modified date:
12 January 2022

UID

ibm11284508

Page Feedback