Troubleshooting
Problem
Scheduled job Exception with "Communications link failure due to underlying exception" on the customTableDataUpload_80, customTableDataUpload_104, customTableDataUpload_105 or customTableDataUpload_106 trigger. When running Guardium v8.01 Central Manager
Symptom
Scheduled Job exceptions similar to:
>>>>
2012-06-24 09:00:05.0 customTableDataUpload_104 trigger:
customTableJobGroup.customTableDataUpload_104
479 total inserts.
Error occurred while upload on datasource: <collector/aggregator hostname> :
Cannot get connection. 1
<<<<
These exceptions commonly occur every 5 minutes for one or more of the customTableDataUpload_80, customTableDataUpload_104, customTableDataUpload_105 or customTableDataUpload_106 jobs
Cause
This exception is due to one or more of the following reasons:
- the CUSTOM database table ENTERPRISE_NO_TRAFFIC has reached the 4GB limit;
- the CUSTOM database has reached the 4GB limit;
- inter-appliance connection timeout was exceeded
- the appliance listed in the <hostname> is a new aggregator.
- a port is closed in the firewall.
Environment
Guardium v8.01 Central Manager
Diagnosing The Problem
customTableJobGroup.customTableDataUpload_80 9 total inserts. Error
occurred while upload on datasource: <collector/aggregator hostname> :
Communications link failure due to underlying exception: ** BEGIN
AND/OR
customTableJobGroup.customTableDataUpload_104 20 total inserts. Error
occurred while upload on datasource: <collector/aggregator hostname> :
Communications link failure due to underlying exception: ** BEGIN
AND/OR
customTableJobGroup.customTableDataUpload_105 29 total inserts. Error
occurred while upload on datasource: <collector/aggregator hostname> :
Communications link failure due to underlying exception: ** BEGIN
AND/OR
customTableJobGroup.customTableDataUpload_106 2 total inserts. Error
occurred while upload on datasource: <collector/aggregator hostname> :
Communications link failure due to underlying exception: ** BEGIN
Resolving The Problem
Follow these steps to resolve the issue (ALL steps must be performed on the central manager ONLY - however, patch installation should be performed on all appliances):
1. Install the latest GPU (v801p100 or later) and v801p140. The GPU can be downloaded from Fix Central. Patch v801p140 should be requested from Guardium support (as part of step 6 below).
2. Restart the appliance after installing the latest GPU (this is only required if the appliance does not restart after installing the GPU). Restart the GUI after applying v801p140
3.In the GUI go to <Tools>, <Report Builder>, <Customer Table Builder>
(if you receive "Error: This system is not licensed for the Custom Data Upload application; please contact a Guardium account manager." skip ahead to Step 5 below):
- for customTableDataUpload_80 exceptions: select "STAP_INFO" and click purge. Schedule a purge to run daily for all data older than 1 day.
- for customTableDataUpload_104 exceptions: select "CM_SNIFFER_BUFFER_USAGE" and click purge. Schedule a purge to run daily for all data older than 1 day.
- for customTableDataUpload_105 exceptions: select "Enterprise No Traffic" and click purge. Schedule a purge to run daily for all data older than 1 day.
- for customTableDataUpload_106 exceptions: select "ENTERPRISE_STAPS_CHANGED" and click purge. Schedule a purge to run daily for all data older than 1 day.
4.In the GUI go to <Tools>, <Report Builder>, <Customer Table Builder> and:
- for customTableDataUpload_80 exceptions: select "STAP_INFO" and click "Upload Data", click "Modify Schedule". Define a schedule to run no more frequently than once every 15 minutes.
- for customTableDataUpload_104 exceptions: select "CM_SNIFFER_BUFFER_USAGE" and click "Upload Data", click "Modify Schedule". Define a schedule to run no more frequently than once every 15 minutes.
- for customTableDataUpload_105 exceptions: select "Enterprise No Traffic" and click "Upload Data", click "Modify Schedule". Define a schedule to run no more frequently than once every 20 minutes.
- for customTableDataUpload_106 exceptions: select "ENTERPRISE_STAPS_CHANGED" and click "Upload Data", click "Modify Schedule". Define a schedule to run no more frequently than once every 15 minutes.
5. If the appliance in the exception is a new aggregator. Unregister this aggregator from the central manager, run a manual aggregation using "Run Once Now". Once the aggregation has completed, re-register it back with the central manager.
6. Check that all ports are open as required. See this article:
https://www-01.ibm.com/support/docview.wss?uid=swg21973188
Especially check that 3306 is open in both directions.
7. Open a PMR to have someone from L2 Guardium support ensure the CUSTOM database and/or the four tables mentioned above are NOT full.
If you received the "...not licensed for the Custom Data Upload application ..." message in Step 3, please note this in the PMR and ask that these scheduled uploads be disabled in the database.
Note: In Step 3 above, define a purge schedule that will work for you but will NOT fill the CUSTOM database.
Note: In Step 4 above, these are only recommended schedule times, but do not run the data uploads more frequently than suggested.
Was this topic helpful?
Document Information
Modified date:
16 June 2018
UID
swg21607822