• “temp_working_space”= <Fixed Size>
  Use the specified amount in <Fixed Size>, e.g. 500 GB, as initial size of the transient storage pool.
• “temp_working_space”=”automatic”
  This is the default setting used with 7.5.11 or later if temp_working_space is not explicitly set in the JSON configuration file. With this setting the initial size of the temporary tablespace in the data pool was the smaller of the following two values (80% of the LPAR memory, 50% of the free space of the data pool), but the accelerator might have reduced the size over time in case the data pool free space was low. To derive the initial size of the transient storage pool from this setting take one of the following options:
  • Use 80% of the configured LPAR memory as the initial size per accelerator LPAR. The amount of configured LPAR memory can be determined from the LPAR configuration, e.g. in the HMC.
  • Open a support case with IBM support to determine the current size of the configured temporary tablespace and use this as the minimum initial size of the transient storage pool.
• “temp_working_space”=”unlimited”
  This is the default setting for Accelerator maintenance levels lower than 7.5.11 if temp_working_space is not explicitly set in the JSON configuration file. If your accelerator installation uses the “unlimited” setting (either as default setting or explicitly set), take one of the following options:
  • Use 80% of the configured LPAR memory as the initial size per accelerator LPAR.
  • Open a support case with IBM support to monitor the temporary storage requirements of your workload and determine the initial size of the transient storage pool based on the results.

3.2 Configuring a transient storage pool on external storage

• FCP- or FICON-attached storage is required.
• The use of all-flash storage is strongly recommended.
• For FICON-attached storage devices the use of HyperPAV and zHPF is mandatory.
• For FCP-attached storage devices a connection to a Fibre Channel SAN using a switched fabric is required.

"transient_devices": {
      "type": "dasd",
      "devices": [
        "0.0.9b12"
      ]
    }

3.3 Operational considerations when using a transient storage pool

3.3.1 Monitoring transient storage pool usage

• The SMF counter Q8STTSA provides the current amount of disk space used by all paired Db2 subsystems for transient data. Note, that the provided value is not a high water mark, but a current value. Thus, a regular monitoring of this counter would be required to get a good understanding of the transient storage usage over time.
• With Accelerator maintenance level 7.5.12, the accelerator-internal monitoring has been improved to track the temp space usage high watermark in the accelerator trace file. Thus, IBM Support can help to determine the current high water mark from the trace file.

• Increase the transient storage pool size
  Only additional storage can be used to increase the transient storage pool. You cannot move storage from the data pool to the transient storage pool
• Set the Db2 Warehouse WLM threshold MAXTEMPSPACECONSUME
  This threshold is used cancel queries automatically that require more transient storage than the configured threshold value. Open a support case with IBM support to set this threshold on the accelerator.

3.3.2 Considerations for setting or unsetting WLM threshold MAXTEMPSPACECONSUME

Back to top

4 Configuring and using local NVMe storage (Option B)

• Estimating the size of local NVMe storage
• Configuring local NVMe storage
• Operational considerations when using local NVMe storage

4.1 Estimating the size of local NVMe storage

For using local NVMe storage, the sizing guidelines are straightforward as outlined in the following. The guidelines apply to multi-node accelerator as well as for single-node accelerator installations.  The same guidelines also apply to new or existing accelerator installations.

As a general sizing guideline use two NVMe cards (Samsung PM1733A) of 15 TB each per accelerator LPAR.

Each NVMe card requires a carrier built into the LinuxONE machine, thus in addition to the NVMe cards two carriers are required per accelerator LPAR.  For example, if your multi-node accelerator consists of 4 LPARs (1 head node and 3 data nodes), then according to the sizing guideline this results in 8 carriers and 8 NVMe cards (each 15 TB).
If local NVMe storage should be used for a single-node accelerator installation and the LPAR memory is < 4 TB, you should consider to use just one 15 TB NVMe card and one carrier.

If for any reason different NVMe card sizes (e.g. smaller ones) are under consideration, open a support case with IBM support to monitor the transient storage requirements of your workload and determine the required size based on the results. But note, that any other NVMe card with different size or from different vendor cannot be acquired from IBM.  Only the mentioned 15 TB card from Samsung can be acquired from IBM.

4.2 Configuring local NVMe storage

For configuring local NVMe storage for transient data, the following requirements exists:

• One of the following IBM LinuxONE systems:
  • IBM LinuxONE^TM Emperor 4 (LA1 - 3931)
  • IBM LinuxONE^TM Rockhopper 4 (LA2 and AGL - 3932)
  • IBM LinuxONE^TM III LT1 (8561)
  • IBM LinuxONE^TM III LT2  (8562)
• IBM Adapter for NVMe 1.1 with FC0448
  • IBM part number: P/N 02WN273
  • The number of adapters must be equal to the number of carriers calculated in the sizing process.
• NVMe card(s) “Samsung PM1733A MZWLR15THBLA-00A07 (15.36TB)”
  • IBM part number: P/N 01CM547
  • Contact your IBM zHW representative to acquire the cards from IBM
  • The number of cards must be equal to the number of carriers calculated during sizing process.
  • IBM provides support for this NVMe card and adapter FC0448 when used for an accelerator deployment. If a different NVMe card is used for your accelerator deployment, IBM only provides support for the adapter, but not for the NVMe card.
  • For NVMe card specification see this Link
• Minimum MCL level S84a for z15 based LinuxONE
• Minimum MCL level S27 for z16 based LinuxONE

For more information on the installation of the adapter and NVMe card into a LinuxONE system, refer to one of the following documentations:

• z15 based LinuxONE: https://www.ibm.com/support/pages/8562-installation-manual-physical-planning
• z16 based LinuxONE: https://www.ibm.com/support/pages/3931-installation-manual-physical-planning

After the NVMe cards have been correctly installed into the LinuxONE system and are visible to the accelerator SSC LPAR(s) after reactivation of the SSC LPAR(s), specify the “transient_storage”:”NVMe” keyword in the “runtime_environment” section. In a multi-node accelerator environment, define it for all data node LPARs. Define it also for the head node LPAR if NVMe cards have also be installed into the system to be used by the head node LPAR.

For example:

For example: 
"runtime_environments": [
        {
            "cpc_name": "Z16_4",
            "lpar_name": "LPAR1",
            "transient_storage": "NVMe",
            "network_interfaces": [
                      ...

See the following chapters in the Accelerator documentation for more information on the “transient_storage” configuration parameter:
https://www.ibm.com/docs/en/daafz/7.5?topic=z-installing-starting-appliance-single-node-setup
https://www.ibm.com/docs/en/daafz/7.5?topic=z-installing-starting-appliance-multi-node-setup

Remove any custom setting of the "temp_working_space" parameter because this parameter applies to transient data in the data pool only. As soon as transient data is processed on local NVMe storage, the "temp_working_space" parameter becomes ineffective.

If “transient_devices” and “transient_storage” are defined in the JSON configuration file, the setting of “transient_storage” takes precedence and “transient_devices” is ignored.

After you completed the changes in the JSON configuration file, upload it to the accelerator as described in this chapter: https://www.ibm.com/docs/en/daafz/7.5?topic=z-updating-existing-configuration

After the JSON configuration file has been successfully uploaded and applied, your accelerator uses all available local NVMe storage devices for writing transient data as part of your workload execution.

4.3 Operational considerations when using local NVMe storage

This chapter describes how to monitor the expected lifetime of an NVMe card and what to do if a NVMe card fails unexpectedly.

In addition, this chapter provides considerations for setting or unsetting the WLM threshold MAXTEMPSPACECONSUME in combination with using local NVMe storage.

4.3.1 Monitoring the lifetime of a NVMe card

A NVMe card has an expected lifetime of multiple years depending on the size of the card and the I/O activity on it. The accelerator estimates the remaining lifetime and issues a warning message in the Admin UI starting when 50% of the expected lifetime has been reached. This message can be used to plan replacement of the cards early before 90%-100% are reached or the NVMe card fails.

The following screenshot shows an example of this message:

The expected lifetime or endurance of a NVMe card is outlined in the specification of a NVMe card and is expressed using the metric ‘Drive writes per day’ (DWPD). For the “Samsung PM1733A” NVMe card with 15 TB the DWPD is 1 for 5 years. This means that the endurance is 5 years assuming that the card is completely overwritten once a day with 15 TB of data. If for example only 7.5 TB transient data is written each day, the expected lifetime of the card is 10 years.

4.3.2 NVMe card failures

If a NVMe card fails for any reason, for example because the end of lifetime has been reached and the card hasn’t been exchanged before, the accelerator gets into a ‘hang’ situation and manual recovery steps are required.

If a NVMe card failure occurs during startup of an accelerator the symptoms are the following:

• The accelerator will hang in the system state "starting". After several hours after a timeout is reached the system state becomes "failed".
• In the Admin UI storage panel no storage information is displayed for the LPAR with the failed NVMe card or the storage panel is not shown at all. For other LPARs (in case of a multi-node deployment), the storage and NVMe card information is displayed correctly.
• In the Admin UI logs panel no log messages are shown for the LPAR with failed NVMe card after messages related to network connection. For other LPARs (in case of a multi-node deployment), further log messages are written.

If a NVMe card failure occurs during operation of an accelerator the symptoms are the following:

• Executing queries that write transient data are hanging, but they do not fail. Internally they are waiting (forever) for the transient storage to become available, therefore they do not fail.
• In the Admin UI no storage information is displayed for the LPAR with the failed NVMe card. The display hangs. For other LPARs (in case of a multi-node deployment) the storage and NVMe card information is displayed correctly.

To recover from this situation

• Change the JSON configuration file: remove the “transient_storage”:”NVMe” keyword for the LPAR with the failed card.
• Upload the JSON configuration file to the accelerator
• Reactivate the accelerator SSC LPARs

Any transient data on this LPAR will then be written to the data pool. This might impact performance.

As a final resolution replace the failing NVMe card and reactivate the accelerator SSC LPARs afterwards. Don’t forget to revert any temporary changes in the JSON configuration file and upload it to use the NVMe local storage again for all LPARs.

4.3.3 Considerations for setting or unsetting WLM threshold MAXTEMPSPACECONSUME

The Db2 Warehouse WLM threshold MAXTEMPSPACECONSUME can be used to cancel queries that require more transient storage than the configured threshold value.

Has the threshold MAXTEMPSPACECONSUME been set in your current accelerator installation?

If the answer is Yes, it is suggested to remove the MAXTEMPSPACECONSUME threshold when moving to a transient storage pool. Thereby queries that are currently canceled due to the threshold will finish successfully.

Keep in mind: if you are facing so-called run-away queries consuming a lot of resources in general, not just transient storage and impacting the overall performance of the accelerator, then consider keeping the threshold or reestablishing it again later. For removing or recreating this threshold, open a support case with IBM support.

Back to top