JR64158: BAI EVENT LOSS WHEN RUNNING ODM IN SPARK

APAR status

• Closed as program error.

Error description

• The BAI plugin is emitting processed decision events to BAI
  running within a OCP cluster.
  What we observe is that the total number of events received in
  BAI for any Spark job is always less than the number of decision
  responses received by the Spark driver.
  
  It appears the odm-bai client threads cannot always emit their
  events to Kafka before the executor JVMs are being shutdown by
  Spark.
  

Local fix

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * Users working on SPARK.                                      *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * The BAI plugin is emitting processed decision events to BAI  *
  * running within a OCP cluster.                                *
  * What we observe is that the total number of events received  *
  * in                                                           *
  * BAI for any Spark job is always less than the number of      *
  * decision                                                     *
  * responses received by the Spark driver.                      *
  *                                                              *
  *                                                              *
  ****************************************************************
  * RECOMMENDATION:                                              *
  ****************************************************************
  

Problem conclusion

• It appears the odm-bai client threads cannot always emit their
  events to Kafka before the executor JVMs are being shutdown by
  Spark.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  CLOUD PAK FOR A

• Fixed component ID

  5737I2300

Applicable component levels