Question & Answer
Question
What are Domain-specific Statistical baseline anomaly models?
Answer
Background:
Readymade models for incident detection, diagnosis, resolution, and avoidance. Domain-specific Statistical baseline anomaly models aim to detect alerts and stories for the log data from applications running on specific IBM products. These models help Site Reliability Engineers (SREs) to better react to the Customer Impacting Events (CIEs). As of 3.6, WebSphere and MQ domains are supported.
Architecture:
Domain-specific Statistical baseline anomaly models involve these steps:
- Detecting specific IBM product anomalies without any pre-training. For domain-specific log anomaly detection, such as for IBM MQ or WebSphere, the statistical baseline model already has a baseline for domain-specific logs. Thus, the model can detect potential errors without waiting for 30 minutes to establish an initial baseline.
- Appending the detected alert by using log anomaly to resolution microservice. This module appends the alert with short explanation, category, and subcategory. It also provides the top three resolution links to help SRE resolve the anomaly.
- Display the appended alert on the alert UI as well as Slack or MS teams.
High-Level Design:
Limitations:
- The 'ibm_messageId', 'type' and 'loglevel' are expected to be present in top level of the raw log with the exact same key names.
- If the identified significant message code is not present in the database, then the Domain-specific Statistical baseline anomaly information is not displayed in the UI.
- For WebSphere, High Performance Extensible Logging needs to be enabled. More details here.
Verification on a cluster with offline data and kafka connection:
Creating a connection:
1. Log in to the CPD AIOps UI.
2. Navigate to
Data and tool connections page from the hamburger menu.
3. Click on
Add Connection. Navigate to Kafka block and click on Add connection. Click on connect.
4. Fill in the details as shown in the screenshot and click on
Next.
5. Select
logs and custom for the connection. Note down the Topic name. Edit the Mapping and change it to either WebpShere or MQ mapping. Click on Next.Mapping for WebSphere:
{
"timestamp_settings": {
"pattern": "yyyy-MM-dd'T'HH:mm:ss.SSSZZZZZ",
"timestamp_field": "ibm_datetime"
},
"codec": "custom",
"message_field": "message",
"log_entity_types": "host,ibm_userDir,ibm_serverName,module",
"instance_id_field": "ibm_serverName",
"rolling_time": 10,
"custom_regex": []
}
Mapping for MQ:
{
"timestamp_settings": {
"pattern": "yyyy-MM-dd'T'HH:mm:ss.SSSX",
"timestamp_field": "ibm_datetime"
},
"codec": "custom",
"message_field": "message",
"log_entity_types": "host,ibm_userDir,ibm_serverName,module",
"instance_id_field": "ibm_serverName",
"rolling_time": 10,
"custom_regex": []
}
6. Enable the
Data flow and click on Done.
Data Ingestion:
Use kcat or any other kafka utility to ingest the data file onto the kafka topic noted on step 5.
- In a terminal, oc login to the cluster.
- Export
sasl_username and broker. Extract kafka certificate.export sasl_password=$(oc get secret cp4waiops-cartridge-kafka-auth-0 --template={{.data.password}} | base64 --decode); export BROKER=$(oc get routes iaf-system-kafka-bootstrap -o=jsonpath='{.status.ingress[0].host}{"\n"}'):443 oc extract secret/iaf-system-cluster-ca-cert --keys=ca.crt --to=-> ca.crt - Ingest file to topic. Replace with the actual topic from above. Replace with the input file name.
kcat -X security.protocol=SASL_SSL -X ssl.ca.location=ca.crt -X sasl.mechanisms=SCRAM-SHA-512 -X sasl.username=cp4waiops-cartridge-kafka-auth-0 -X sasl.password=$sasl_password -b $BROKER -P -t <topic-name> -l <file-name>
Data Schemas:
1. Sample output of A in the design diagram for WebSphere IBM product.
{
"type": "was_message",
"host": "ABC",
"ibm_cellName": "ndcell",
"ibm_nodeName": "node2",
"ibm_serverName": "XYZ",
"ibm_sequence": "1616168930974_0000000013393",
"message": "SRVE0014E: Uncaught service() exception root cause TradeAppServlet: javax.servlet.ServletException: TradeServletAction.doLogin(...)Exception logging in user uid:36259with passwordxxx\n\tat com.ibm.websphere.samples.daytrader.web.TradeServletAction.doLogin(TradeServletAction.java:344)\n\tat com.ibm.websphere.samples.daytrader.web.TradeAppServlet.performTask(TradeAppServlet.java:130)\n\tat com.ibm.websphere.samples.daytrader.web.TradeAppServlet.doPost(TradeAppServlet.java:98)\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:707)\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:790)\n\tat com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1235)\n\tat com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:779)\n\tat com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:478)\n\tat com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:179)\n\tat com.ibm.ws.webcontainer.filter.WebAppFilterChain.invokeTarget(WebAppFilterChain.java:143)\n\tat com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:96)\n\tat com.ibm.websphere.samples.daytrader.web.OrdersAlertFilter.doFilter(OrdersAlertFilter.java:96)\n\tat com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:197)\n\tat com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:90)\n\tat com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:979)\n\tat com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1119)\n\tat com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:82)\n\tat com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:963)\n\tat com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1817)\n\tat com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:382)\n\tat com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:465)\n\tat com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:532)\n\tat com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:318)\n\tat com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:289)\n\tat com.ibm.ws.ssl.channel.impl.SSLConnectionLink.determineNextChannel(SSLConnectionLink.java:1187)\n\tat com.ibm.ws.ssl.channel.impl.SSLConnectionLink.readyInboundPostHandshake(SSLConnectionLink.java:768)\n\tat com.ibm.ws.ssl.channel.impl.SSLConnectionLink$MyHandshakeCompletedCallback.complete(SSLConnectionLink.java:464)\n\tat com.ibm.ws.ssl.channel.impl.SSLUtils.handleHandshake(SSLUtils.java:1137)\n\tat com.ibm.ws.ssl.channel.impl.SSLHandshakeIOCallback.complete(SSLHandshakeIOCallback.java:87)\n\tat com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175)\n\tat com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)\n\tat com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)\n\tat com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)\n\tat com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)\n\tat com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:816)\n\tat com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)\n\tat com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1909)\nCaused by: javax.ejb.EJBException: See nested exception; nested exception is: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.6.8.WAS-v20200807-b093148): org.eclipse.persistence.exceptions.DatabaseException\nInternal Exception: com.ibm.websphere.ce.cm.StaleConnectionException: [jcc][t4][2043][11550][4.27.25] Exception java.net.ConnectException: Error opening socket to server ABC/X.X.X.X on port 50,000 with message: Connection refused (Connection refused). ERRORCODE=-4499, SQLSTATE=08001 DSRA0010E: SQL State = 08001, Error Code = -4,499\nError Code: -4499\nCall: SELECT USERID, ADDRESS, CREDITCARD, EMAIL, FULLNAME, PASSWD FROM accountprofileejb WHERE (USERID = ?)\n\tbind => [1 parameter bound]\nQuery: ReadObjectQuery(name=\"readAccountProfileDataBean\" referenceClass=AccountProfileDataBean sql=\"SELECT USERID, ADDRESS, CREDITCARD, EMAIL, FULLNAME, PASSWD FROM accountprofileejb WHERE (USERID = ?)\")\n\tat com.ibm.ejs.container.util.ExceptionUtil.EJBException(ExceptionUtil.java:503)\n\tat com.ibm.ejs.container.util.ExceptionUtil.EJBException(ExceptionUtil.java:379)\n\tat com.ibm.ejs.container.BusinessExceptionMappingStrategy.mapException(BusinessExceptionMappingStrategy.java:376)\n\tat com.ibm.ejs.container.BusinessExceptionMappingStrategy.setUncheckedException(BusinessExceptionMappingStrategy.java:549)\n\tat com.ibm.ejs.container.EJSDeployedSupport.setUncheckedLocalException(EJSDeployedSupport.java:477)\n\tat com.ibm.websphere.samples.daytrader.ejb3.EJSLocal0SLTradeSLSBBean_78e9c356.login(EJSLocal0SLTradeSLSBBean_78e9c356.java)\n\tat com.ibm.websphere.samples.daytrader.TradeAction.login(TradeAction.java:593)\n\tat com.ibm.websphere.samples.daytrader.web.TradeServletAction.doLogin(TradeServletAction.java:316)\n\t... 36 more\nCaused by: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.6.8.WAS-v20200807-b093148): org.eclipse.persistence.exceptions.DatabaseException\nInternal Exception: com.ibm.websphere.ce.cm.StaleConnectionException: [jcc][t4][2043][11550][4.27.25] Exception java.net.ConnectException: Error opening socket to server ABC/X.X.X.X on port 50,000 with message: Connection refused (Connection refused). ERRORCODE=-4499, SQLSTATE=08001 DSRA0010E: SQL State = 08001, Error Code = -4,499\nError Code: -4499\nCall: SELECT USERID, ADDRESS, CREDITCARD, EMAIL, FULLNAME, PASSWD FROM accountprofileejb WHERE (USERID = ?)\n\tbind => [1 parameter bound]\nQuery: ReadObjectQuery(name=\"readAccountProfileDataBean\" referenceClass=AccountProfileDataBean sql=\"SELECT USERID, ADDRESS, CREDITCARD, EMAIL, FULLNAME, PASSWD FROM accountprofileejb WHERE (USERID = ?)\")\n\tat org.eclipse.persistence.exceptions.DatabaseException.sqlException(DatabaseException.java:316)\n\tat org.eclipse.persistence.sessions.JNDIConnector.connect(JNDIConnector.java:147)\n\tat org.eclipse.persistence.sessions.DatasourceLogin.connectToDatasource(DatasourceLogin.java:162)\n\tat org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.connectInternal(DatasourceAccessor.java:346)\n\tat org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.connectInternal(DatabaseAccessor.java:309)\n\tat org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.reconnect(DatasourceAccessor.java:583)\n\tat org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.reconnect(DatabaseAccessor.java:1634)\n\tat org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.incrementCallCount(DatasourceAccessor.java:321)\n\tat org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:615)\n\tat org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeCall(DatabaseAccessor.java:560)\n\tat org.eclipse.persistence.internal.sessions.AbstractSession.basicExecuteCall(AbstractSession.java:2064)\n\tat org.eclipse.persistence.sessions.server.ServerSession.executeCall(ServerSession.java:586)\n\tat org.eclipse.persistence.sessions.server.ClientSession.executeCall(ClientSession.java:262)\n\tat org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:262)\n\tat org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:248)\n\tat org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.selectOneRow(DatasourceCallQueryMechanism.java:751)\n\tat org.eclipse.persistence.internal.queries.ExpressionQueryMechanism.selectOneRowFromTable(ExpressionQueryMechanism.java:2811)\n\tat org.eclipse.persistence.internal.queries.ExpressionQueryMechanism.selectOneRow(ExpressionQueryMechanism.java:2764)\n\tat org.eclipse.persistence.queries.ReadObjectQuery.executeObjectLevelReadQuery(ReadObjectQuery.java:555)\n\tat org.eclipse.persistence.queries.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1183)\n\tat org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:908)\n\tat org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1142)\n\tat org.eclipse.persistence.queries.ReadObjectQuery.execute(ReadObjectQuery.java:441)\n\tat org.eclipse.persistence.queries.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:1230)\n\tat org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2900)\n\tat org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1866)\n\tat org.eclipse.persistence.internal.sessions.AbstractSession.retryQuery(AbstractSession.java:1936)\n\tat org.eclipse.persistence.sessions.server.ClientSession.retryQuery(ClientSession.java:698)\n\tat org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.retryQuery(UnitOfWorkImpl.java:5540)\n\tat org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1902)\n\tat org.eclipse.persistence.internal.sessions.AbstractSession.retryQuery(AbstractSession.java:1936)\n\tat org.eclipse.persistence.sessions.server.ClientSession.retryQuery(ClientSession.java:698)\n\tat org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.retryQuery(UnitOfWorkImpl.java:5540)\n\tat org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1902)\n\tat org.eclipse.persistence.internal.sessions.AbstractSession.retryQuery(AbstractSession.java:1936)\n\tat org.eclipse.persistence.sessions.server.ClientSession.retryQuery(ClientSession.java:698)\n\tat org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.retryQuery(UnitOfWorkImpl.java:5540)\n\tat org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1902)\n\tat org.eclipse.persistence.internal.sessions.AbstractSession.retryQuery(AbstractSession.java:1936)\n\tat org.eclipse.persistence.sessions.server.ClientSession.retryQuery(ClientSession.java:698)\n\tat org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.retryQuery(UnitOfWorkImpl.java:5540)\n\tat org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1902)\n\tat org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1848)\n\tat org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1799)\n\tat org.eclipse.persistence.internal.jpa.EntityManagerImpl.executeQuery(EntityManagerImpl.java:915)\n\tat org.eclipse.persistence.internal.jpa.EntityManagerImpl.findInternal(EntityManagerImpl.java:858)\n\tat org.eclipse.persistence.internal.jpa.EntityManagerImpl.find(EntityManagerImpl.java:734)\n\tat org.eclipse.persistence.internal.jpa.EntityManagerImpl.find(EntityManagerImpl.java:603)\n\tat com.ibm.ws.jpa.management.JPATxEmInvocation.find(JPATxEmInvocation.java:231)\n\tat com.ibm.ws.jpa.management.JPAEntityManager.find(JPAEntityManager.java:230)\n\tat com.ibm.websphere.samples.daytrader.ejb3.TradeSLSBBean.login(TradeSLSBBean.java:533)\n\tat sun.reflect.GeneratedMethodAccessor271.invoke(Unknown Source)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)\n\tat java.lang.reflect.Method.invoke(Method.java:508)\n\tat com.ibm.ejs.container.EJSContainer.invokeProceed(EJSContainer.java:5347)\n\tat com.ibm.ejs.container.interceptors.InvocationContextImpl.proceed(InvocationContextImpl.java:652)\n\tat com.ibm.ws.cdi.ejb.impl.InterceptorChain.proceed(InterceptorChain.java:120)\n\tat com.ibm.ws.cdi.ejb.impl.EJBCDIInterceptorWrapper.invokeInterceptors(EJBCDIInterceptorWrapper.java:140)\n\tat com.ibm.ws.cdi.ejb.impl.EJBCDIInterceptorWrapper.aroundInvoke(EJBCDIInterceptorWrapper.java:56)\n\tat sun.reflect.GeneratedMethodAccessor109.invoke(Unknown Source)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)\n\tat java.lang.reflect.Method.invoke(Method.java:508)\n\tat com.ibm.ejs.container.interceptors.InterceptorProxy.invokeInterceptor(InterceptorProxy.java:201)\n\tat com.ibm.ejs.container.interceptors.InvocationContextImpl.proceed(InvocationContextImpl.java:632)\n\tat org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:64)\n\tat com.ibm.ws.cdi.ejb.impl.WeldSessionBeanInterceptorWrapper.aroundInvoke(WeldSessionBeanInterceptorWrapper.java:58)\n\tat sun.reflect.GeneratedMethodAccessor105.invoke(Unknown Source)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)\n\tat java.lang.reflect.Method.invoke(Method.java:508)\n\tat com.ibm.ejs.container.interceptors.InterceptorProxy.invokeInterceptor(InterceptorProxy.java:201)\n\tat com.ibm.ejs.container.interceptors.InvocationContextImpl.proceed(InvocationContextImpl.java:632)\n\tat com.ibm.ejs.container.interceptors.InvocationContextImpl.doAroundInterceptor(InvocationContextImpl.java:306)\n\tat com.ibm.ejs.container.interceptors.InvocationContextImpl.doAroundInvoke(InvocationContextImpl.java:273)\n\tat com.ibm.ejs.container.EJSContainer.invoke(EJSContainer.java:5239)\n\t... 39 more\nCaused by: com.ibm.websphere.ce.cm.StaleConnectionException: [jcc][t4][2043][11550][4.27.25] Exception java.net.ConnectException: Error opening socket to server ABC/X.X.X.X on port 50,000 with message: Connection refused (Connection refused). ERRORCODE=-4499, SQLSTATE=08001 DSRA0010E: SQL State = 08001, Error Code = -4,499\n\tat sun.reflect.GeneratedConstructorAccessor134.newInstance(Unknown Source)\n\tat sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:57)\n\tat java.lang.reflect.Constructor.newInstance(Constructor.java:437)\n\tat com.ibm.websphere.rsadapter.GenericDataStoreHelper.mapExceptionHelper(GenericDataStoreHelper.java:630)\n\tat com.ibm.websphere.rsadapter.GenericDataStoreHelper.mapException(GenericDataStoreHelper.java:689)\n\tat com.ibm.ws.rsadapter.AdapterUtil.mapException(AdapterUtil.java:2289)\n\tat com.ibm.ws.rsadapter.spi.WSRdbDataSource.getPooledConnection(WSRdbDataSource.java:2158)\n\tat com.ibm.ws.rsadapter.spi.WSManagedConnectionFactoryImpl.getConnection(WSManagedConnectionFactoryImpl.java:1830)\n\tat com.ibm.ws.rsadapter.spi.WSManagedConnectionFactoryImpl.createManagedConnection(WSManagedConnectionFactoryImpl.java:1587)\n\tat com.ibm.ws.rsadapter.spi.WSManagedConnectionFactoryImpl.createManagedConnection(WSManagedConnectionFactoryImpl.java:1137)\n\tat com.ibm.ejs.j2c.FreePool.createManagedConnectionWithMCWrapper(FreePool.java:2178)\n\tat com.ibm.ejs.j2c.FreePool.createOrWaitForConnection(FreePool.java:1848)\n\tat com.ibm.ejs.j2c.PoolManager.reserve(PoolManager.java:3880)\n\tat com.ibm.ejs.j2c.PoolManager.reserve(PoolManager.java:3128)\n\tat com.ibm.ejs.j2c.ConnectionManager.allocateMCWrapper(ConnectionManager.java:1556)\n\tat com.ibm.ejs.j2c.ConnectionManager.allocateConnection(ConnectionManager.java:1035)\n\tat com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java:646)\n\tat com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java:613)\n\tat org.eclipse.persistence.sessions.JNDIConnector.connect(JNDIConnector.java:135)\n\t... 111 more\n",
"ibm_datetime": "2021-03-19T11:48:50.974-0400",
"ibm_messageId": "SRVE0014E",
"ibm_methodName": "service",
"ibm_className": "com.ibm.ws.webcontainer.servlet.ServletWrapper",
"ibm_threadId": "000019d6",
"module": "com.ibm.ws.webcontainer.servlet.ServletWrapper",
"loglevel": "SEVERE",
"ext_appName": "PQR",
"unix_ts": 1616168930974,
"start": 1616168930000,
"end": 1616168940000
}
2. Sample output of B in the design diagram.
{
"timestamp": 1616169360000,
"application_id": "1000",
"instance_id": "XYZ",
"application_group_id": "1000",
"type": "was_message",
"source_query_url": null,
"model_version": "na",
"start": 1616169350000,
"end": 1616169360000,
"elastic_index_id": 1106234340,
"count": [
2454,
279
],
"sequence": [],
"timestamps": [1616169351089,1616169351155,1616169351156,1616169351156,1616169351156,1616169351156,1616169351157,1616169351157,1616169351158,1616169351158,1616169351158,1616169351158,1616169351159,1616169351159,1616169351159,1616169351159,1616169351165,1616169351167,1616169351319,1616169351607,1616169351607,1616169351607,1616169351607,1616169351607,1616169351608,1616169351608,1616169351609,1616169351609,1616169351609,1616169351609,1616169351610,1616169351610,1616169351610,1616169351610,1616169351615,1616169351621,1616169351694,1616169351695,1616169351727,1616169351727,1616169351821,1616169351821,1616169351821,1616169351821,1616169351822,1616169351822,1616169351822,1616169351824,1616169351824,1616169351824,1616169351824,1616169351824,1616169351825,1616169351825,1616169351825,1616169351829,1616169351831,1616169352018,1616169352021,1616169352021,1616169352021,1616169352021,1616169352021,1616169352021,1616169352021,1616169352021,1616169352022,1616169352022,1616169352022,1616169352023,1616169352023,1616169352024,1616169352024,1616169352024,1616169352024,1616169352025,1616169352030,1616169352031,1616169352032,1616169352032,1616169352034,1616169352034,1616169352034,1616169352034,1616169352034,1616169352035,1616169352035,1616169352035,1616169352035,1616169352041,1616169352042,1616169352044,1616169352047,1616169352047,1616169352047,1616169352047,1616169352048,1616169352048,1616169352048,1616169352049,1616169352049,1616169352050,1616169352050,1616169352050,1616169352050,1616169352070,1616169352082,1616169352101,1616169352111,1616169352124,1616169352276,1616169352277,1616169352277,1616169352277,1616169352277,1616169352278,1616169352278,1616169352279,1616169352279,1616169352280,1616169352280,1616169352280,1616169352280,1616169352281,1616169352286,1616169352288,1616169352404,1616169352470,1616169352770,1616169352900,1616169352987,1616169353115,1616169353118,1616169353304,1616169353465,1616169353575,1616169353578,1616169353603,1616169353605,1616169353736,1616169353739,1616169353739,1616169353739,1616169353739,1616169353739,1616169353740,1616169353740,1616169353740,1616169353742,1616169353742,1616169353742,1616169353742,1616169353742,1616169353743,1616169353743,1616169353743,1616169353743,1616169353747,1616169353749,1616169353794,1616169354236,1616169354448,1616169354448,1616169354449,1616169354449,1616169354449,1616169354449,1616169354450,1616169354451,1616169354451,1616169354451,1616169354451,1616169354451,1616169354452,1616169354452,1616169354456,1616169354457,1616169354541,1616169354556,1616169354802,1616169354819,1616169354883,1616169355098,1616169355099,1616169355148,1616169355151,1616169356093,1616169356093,1616169356093,1616169356095,1616169356096,1616169356128,1616169356130,1616169356170,1616169356324,1616169356417,1616169356421,1616169356626,1616169356697,1616169356697,1616169356697,1616169356697,1616169356700,1616169356700,1616169356700,1616169356701,1616169356730,1616169356731,1616169356808,1616169356812,1616169356833,1616169356863,1616169356864,1616169357021,1616169357043,1616169357047,1616169357076,1616169357078,1616169357084,1616169357103,1616169357115,1616169357116,1616169357116,1616169357120,1616169357121,1616169357126,1616169357191,1616169357193,1616169357252,1616169357254,1616169357291,1616169357407,1616169357472,1616169357473,1616169357473,1616169357474,1616169357475,1616169357485,1616169357487,1616169357772,1616169357900,1616169357902,1616169357902,1616169357991,1616169358121,1616169358307,1616169358307,1616169358307,1616169358468,1616169358468,1616169358468,1616169358581,1616169358607,1616169358741,1616169358751,1616169358796,1616169358797,1616169358797,1616169359239,1616169359239,1616169359239,1616169359460,1616169359544,1616169359544,1616169359544,1616169359559,1616169359559,1616169359559,1616169359741,1616169359744,1616169359805,1616169359805,1616169359805,1616169359822,1616169359822,1616169359822,1616169359887,1616169359887,1616169359888],
"mapping": [
"Unknown_normal",
"Unknown_error"
],
"entities": {
"host": [
{
"name": "ABC",
"counts": 279
}
],
"ibm_serverName": [
{
"name": "XYZ",
"counts": 279
}
],
"module": [
{
"name": "eclipselink",
"counts": 22
},
{
"name": "com.ibm.ejs.container.BusinessExceptionMappingStrategy",
"counts": 23
},
{
"name": "SystemOut",
"counts": 87
},
{
"name": "eclipselink.query",
"counts": 119
},
{
"name": "daytrader",
"counts": 18
},
{
"name": "com.ibm.ws.webcontainer.servlet.ServletWrapper",
"counts": 5
},
{
"name": "com.ibm.ws.webcontainer.webapp",
"counts": 5
}
]
},
"embeddings": [
10.199202492452969,
-2.3097720070726228,
-2.0666546228871618,
-6.48876801140618,
-4.674866059275863,
2.7002522338477976,
3.125268016991029,
-3.7652392072917575,
5.819926845777925,
2.917124688362576,
3.412858931575811,
-1.1336996232982426,
-4.908359206606317,
-2.809773695604874,
1.0613694393863158,
-0.309138282518697,
1.5258082348526756,
-2.263572302988586,
4.400773835910493,
0.45330414129311966
],
"named_entities": {
"exception": [
{
"name": "org.eclipse.persistence.exceptions.DatabaseException",
"counts": 255
},
{
"name": "com.ibm.websphere.ce.cm.StaleConnectionException",
"counts": 279
},
{
"name": "java.net.ConnectException",
"counts": 279
},
{
"name": "com.ibm.websphere.rsadapter.GenericDataStoreHelper.mapException",
"counts": 81
},
{
"name": "javax.ejb.EJBException",
"counts": 44
},
{
"name": "javax.persistence.PersistenceException",
"counts": 66
},
{
"name": "org.eclipse.persistence.internal.jpa.QueryImpl.getDetailedException",
"counts": 32
},
{
"name": "org.eclipse.persistence.exceptions.DatabaseException.sqlException",
"counts": 57
},
{
"name": "DatabaseException",
"counts": 57
},
{
"name": "com.ibm.ws.rsadapter.AdapterUtil.mapException",
"counts": 56
},
{
"name": "java.sql.SQLNonTransientException",
"counts": 23
},
{
"name": "javax.servlet.ServletException",
"counts": 5
},
{
"name": "com.ibm.ejs.container.util.ExceptionUtil.EJBException",
"counts": 10
},
{
"name": "com.ibm.ejs.container.BusinessExceptionMappingStrategy.mapException",
"counts": 10
},
{
"name": "BusinessException",
"counts": 10
},
{
"name": "com.ibm.ejs.container.BusinessExceptionMappingStrategy.setUncheckedException",
"counts": 10
},
{
"name": "com.ibm.ejs.container.EJSDeployedSupport.setUncheckedLocalException",
"counts": 10
}
],
"error_log": [
{
"name": "error_log",
"counts": 279
}
],
"normal": [
{
"name": "normal",
"counts": 2336
}
]
},
"oob_entities": {
"log_level": [
{
"name": "WARNING",
"counts": 22
},
{
"name": "SystemOut",
"counts": 2541
},
{
"name": "SEVERE",
"counts": 51
},
{
"name": "INFO",
"counts": 119
}
],
"message_id": [
{
"name": "CNTR0020E",
"counts": 23
},
{
"name": "SRVE0014E",
"counts": 5
},
{
"name": "SRVE0293E",
"counts": 5
}
]
}
}
3. Sample output of C in the design diagram.
{
"id": "bc9bcd6c-7fd3-3e44-918e-e54b5ed97cf6",
"occurrenceTime": "2021-03-19T15:53:30.000Z",
"sender": {
"name": "Log Anomaly"
},
"event_link": "",
"summary": "Logs containing errors for component: XYZ with a primary message code of SRVE0014E. Evidence includes: domain-specific entities",
"description": "SEVERE: 65\nCNTR0020E: 29\nSRVE0014E: 12\nSRVE0293E: 12",
"severity": 5,
"type": {
"eventType": "problem",
"classification": "Statistical baseline anomaly",
"condition": "websphere-SRVE0014E-sev-5"
},
"resource": {
"name": "XYZ",
"type": "application",
"application": "XYZ",
"entity": "XYZ"
},
"expirySeconds": 315600000,
"details": {
"log_anomaly_detector": "{\"start_timestamp\": 1616169210000, \"end_timestamp\": 1616169220000, \"original_group_id\": \"websphere-SRVE0014E-sev-5\", \"causality\": {\"service\": [\"XYZ\"]}, \"detected_at\": 1679654183381.0, \"source_application_id\": \"XYZ\", \"log_anomaly_confidence\": 0.95, \"log_anomaly_model\": [\"RSM_oob_entities\"], \"error_templates\": [2, 4, 5, 6], \"text_dict\": {\"template_ids\": [\"oob_entities_SEVERE\", \"oob_entities_CNTR0020E\", \"oob_entities_SRVE0014E\", \"oob_entities_SRVE0293E\", \"oob_entities_SystemOut\"], \"template_list\": [\"SEVERE\", \"CNTR0020E\", \"SRVE0014E\", \"SRVE0293E\", \"SystemOut\"], \"count_vector\": [65, 29, 12, 12, 1753], \"expected_count_vector\": [0, 0, 0, 0, 0], \"expected_count_lower_bound\": [0, 0, 0, 0, 0], \"expected_count_upper_bound\": [0, 0, 0, 0, 0]}, \"application_group_id\": \"1000\", \"application_id\": \"1000\", \"model_version\": \"na\", \"severity_from_model\": 0, \"description\": \"SEVERE: 65\\nCNTR0020E: 29\\nSRVE0014E: 12\\nSRVE0293E: 12\"}"
},
"insights": [
{
"id": "insight-lad",
"type": "aiops.ibm.com/insight-type/lad/templates",
"details": {
"error_template_list": [
"CNTR0020E",
"SRVE0014E",
"SRVE0293E"
],
"error_count_vector": [
29,
12,
12
],
"significant_message_code": "SRVE0014E",
"type_window": "websphere"
}
}
]
}
4. Sample output of D in the design diagram.
{
"tenantid": "cfd95b7e-3bc7-4006-a4a8-a73a79c71255",
"requestid": "2ae86952-be37-4291-932c-059737431f59",
"requestTime": "2023-03-24T10:36:33.200276311Z",
"type": "create",
"entityType": "alert",
"entity": {
"summary": "Logs containing errors for component: XYZ with a primary message code of SRVE0014E. Evidence includes: domain-specific entities",
"severity": 5,
"signature": "{application=XYZ, entity=XYZ, name=XYZ, type=application}-Statistical baseline anomaly-websphere-SRVE0014E-sev-5",
"resource": {
"name": "XYZ",
"type": "application",
"application": "XYZ",
"entity": "XYZ"
},
"insights": [
{
"id": "insight-lad",
"type": "aiops.ibm.com/insight-type/lad/templates",
"details": {
"error_template_list": [
"CNTR0020E",
"SRVE0014E",
"SRVE0293E"
],
"error_count_vector": [
25,
7,
7
],
"significant_message_code": "SRVE0014E",
"type_window": "websphere"
}
},
{
"id": "cbd73f00-872e-11ed-958a-4fb5dc05c3c7#aiops.ibm.com/action/internal/alerts/derive",
"type": "aiops.ibm.com/insight-type/created-by",
"source": "",
"details": {
"policyId": "cbd73f00-872e-11ed-958a-4fb5dc05c3c7",
"actionId": "aiops.ibm.com/action/internal/alerts/derive",
"partitionKey": "{application=XYZ, entity=XYZ, name=XYZ, type=application}-Statistical baseline anomaly-websphere-SRVE0014E-sev-5"
}
}
],
"deduplicationKey": "{application=XYZ, entity=XYZ, name=XYZ, type=application}-Statistical baseline anomaly-websphere-SRVE0014E-sev-5",
"description": "SEVERE: 55\nCNTR0020E: 25\nSRVE0014E: 7\nSRVE0293E: 7",
"eventCount": 1,
"type": {
"eventType": "problem",
"classification": "Statistical baseline anomaly",
"condition": "websphere-SRVE0014E-sev-5"
},
"firstOccurrenceTime": "2021-03-19T15:52:00.000Z",
"sender": {
"name": "Log Anomaly"
},
"event_link": "",
"details": {
"log_anomaly_detector": "{\"start_timestamp\": 1616169120000, \"end_timestamp\": 1616169130000, \"original_group_id\": \"websphere-SRVE0014E-sev-5\", \"causality\": {\"service\": [\"XYZ\"]}, \"detected_at\": 1679654176818.0, \"source_application_id\": \"XYZ\", \"log_anomaly_confidence\": 0.95, \"log_anomaly_model\": [\"RSM_oob_entities\"], \"error_templates\": [2, 4, 5, 6], \"text_dict\": {\"template_ids\": [\"oob_entities_SEVERE\", \"oob_entities_CNTR0020E\", \"oob_entities_SRVE0014E\", \"oob_entities_SRVE0293E\", \"oob_entities_SystemOut\"], \"template_list\": [\"SEVERE\", \"CNTR0020E\", \"SRVE0014E\", \"SRVE0293E\", \"SystemOut\"], \"count_vector\": [55, 25, 7, 7, 2441], \"expected_count_vector\": [0, 0, 0, 0, 0], \"expected_count_lower_bound\": [0, 0, 0, 0, 0], \"expected_count_upper_bound\": [0, 0, 0, 0, 0]}, \"application_group_id\": \"1000\", \"application_id\": \"1000\", \"model_version\": \"na\", \"severity_from_model\": 0, \"description\": \"SEVERE: 55\\nCNTR0020E: 25\\nSRVE0014E: 7\\nSRVE0293E: 7\"}"
},
"id": "214efbad-2b02-4cb1-95a6-07f814623ce0",
"state": "open",
"lastOccurrenceTime": "2021-03-19T15:52:00.000Z",
"expirySeconds": 315600000
}
}
Alert verification on UI:
1. Wait for 5 minutes. Navigate to Stories and alerts page from Hamburger menu from UI.
2. Click on Alerts tab, locate the log-anomaly WebSphere alert. Verify Explainability information is added.
For more information about domain-specific log anomaly detection, see this blog
[{"Type":"MASTER","Line of Business":{"code":"LOB45","label":"Automation"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSRS1M","label":"IBM Cloud Pak for Watson AIOps"},"ARM Category":[{"code":"a8m3p000000PC33AAG","label":"Watson AIOps-\u003EAI Manager-\u003EAI Modelling\/Algorithms-\u003ELog Anomaly"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"3.6.2"}]
Was this topic helpful?
Document Information
Modified date:
01 April 2023
UID
ibm16964180