What's new and changed?
IBM® Spectrum Conductor 2.5.1 includes various new features, and enhancements to, existing features.
![(2.5.1 Fix)](../251fix.gif)
System configurations and integrations
This version of IBM Spectrum Conductor has been extended to support additional or upgraded system configurations.- Supported system configurations
-
- IBM Spectrum Conductor 2.5.1 supports and includes a resource orchestrator (enterprise grid orchestrator or EGO 4.0).
- Miniconda is no longer bundled with IBM Spectrum Conductor 2.5.1. IBM Spectrum Conductor continues to support Anaconda or Miniconda for management. For details on the supported Miniconda or Anaconda distributions for IBM Spectrum Conductor 2.5.1, see Supported Spark, Miniconda, Jupyter, and Dask versions for details.
- If you are installing or upgrading with other IBM Spectrum Computing
family products, IBM Spectrum
Conductor
2.5.1 supports a multiple
product environment with these IBM Spectrum Computing
family products:
- IBM Spectrum Symphony 7.3.2
- IBM Spectrum LSF® Standard Edition for Symphony®
10.1.0.2.
With IBM Spectrum LSF version 10.1.0.2, you use IBM Spectrum Conductor or IBM Spectrum Symphony as the base scheduler, and then add on IBM Spectrum LSF (refer to Installing or upgrading IBM Spectrum Conductor with other IBM products). You can also use LSF 10.1.0.6 or later as the base scheduler, and then add on IBM Spectrum Conductor 2.5.1 (refer to IBM Spectrum LSF integration for details).
- You can perform a rolling upgrade of IBM Spectrum Conductor from version 2.4.0, 2.4.1, or 2.5.0 to IBM Spectrum Conductor 2.5.1. To upgrade any other version of IBM Spectrum Conductor to 2.5.1, perform a parallel upgrade instead.
Inclusive terminology
IBM Spectrum Conductor to 2.5.1 user interface and messages have changed to remove the use of any non-inclusive terminology and replaced those terms with more inclusive ones. Specifically, we now use the terms primary host and primary candidate host instead of master host and master candidate host. The documentation has changed to reflect the product changes for the new terminology.While IBM values the use of inclusive language, terms that are outside of IBM's direct influence are sometimes required for the sake of maintaining user understanding. As other industry leaders join IBM in embracing the use of inclusive language, IBM will continue to update the documentation to reflect those changes.
Installing, upgrading and configuring
- Verify the integrity and validate the authenticity of your downloaded installation packages from IBM Passport Advantage®
- Now, for added security, after you have downloaded your IBM Spectrum Conductor installation packages from IBM Passport Advantage, you can verify the integrity and validate the authenticity of these files. Simply run a single jarsigner utility. For details on using jarsigner, see Downloading entitled packages.
- Set the built-in
Admin
user password during fresh installations - For added security, you can now set the password for the built-in
Admin
user, rather than use a preset password. Complete this password setting during a fresh installation of IBM Spectrum Conductor on your primary host using the new egoconfig setpassword command. When you run the new egoconfig join command after installation and before setting product entitlement, the system reminds you to run the new egoconfig setpassword command. - New parameter to handle authentication requests in separate threads in VEMKD process
- With the new EGO_AUTH_THREAD_NUMBER parameter, you can specify the number of threads that are needed to handle authentication requests in the VEMKD process. For more information, see ego.conf reference.
Instance groups and notebooks
- Support high availability for the Dask manager
- To enable high availability for the Dask manager, specify the High availability recovery directory, where the Dask manager writes recovery data that is accessible by all Dask manager candidates. For more information, see the Configuring Dask settings for an instance group topic.
- New Jupyter notebook environment variables
- This version of IBM Spectrum
Conductor
includes several new Jupyter environment variables:
Environment variable name Purpose JUPYTER_SERVICE_ACTIVITY_TIMEOUT Use the new JUPYTER_SERVICE_ACTIVITY_TIMEOUT Jupyter environment variable to specify the number of seconds to pass before the server shuts down when the server is idle. NB_EXTRA_CONF_PARAM_parameterName Use the new NB_EXTRA_CONF_PARAM_parameterName and JEG_EXTRA_CONF_PARAM_parameterName environment variables to set extra notebook and Jupyter enterprise gateway configuration parameters. REMOVE_CONFIG_DIR_ON_START Use the new REMOVE_CONFIG_DIR_ON_START Jupyter environment variable to specify how the Jupyter start script handles notebook configuration directory each time a Jupyter notebook starts. For more information about these new Jupyter environment variables, see Jupyter notebook environment variables.
- New data connectors for instance groups to use data on remote file systems
- Configure the IBM Cloud Object Storage and Amazon S3 data connectors for instance groups to switch between data sources on demand. IBM Spectrum Conductor can now be configured to access Amazon S3 file systems by using Amazon S3 data connectors. For more information, see Data connectors.
- New operations to streamline the process of moving data connectors between clusters
- You can remove, export, and import data connectors to move from one cluster to the other:
- For more information on how to remove existing data connectors that are no longer needed from the cluster, see Removing data connectors.
- For more information on how to export an existing data connector from the cluster management console to the file system, see Exporting data connectors.
- For more information on how to import an external data connector package that was exported from the cluster, see Importing data connectors.
- New operations to manage instance level configurations for Anaconda or Miniconda distributions
-
- Fetch configurations for Anaconda or Miniconda distribution instances. For more information, see Retrieving Anaconda configurations.
- Add new configuration parameter values, remove, or update existing configuration parameter values for Anaconda or Miniconda distribution instances. For more information, see Managing Anaconda configurations.
- Override configuration settings for SparkCleanup service
- Use registration or modification services to override the settings set globally during SparkCleanup profile setup. For more information, see Configuring cleanup for instance groups during registration or modification.
- New environment variable for the SparkCleanup service
-
Use the new
CONDUCTOR_CLEANUP_DEPLOY_PACKAGES
environment variable to specify whether the SparkCleanup service cleans up SOAM deployment packages. For more information, see Configuring cleanup for instance groups during profile setup topic.When you set this
CONDUCTOR_CLEANUP_DEPLOY_PACKAGES
environment variable tofalse
, you can configure theegodeploy
service to use the existing SOAM deployment files as opposed to downloading the files again. For more information, see Disabling deployment package cleanup by redepolying existing packages topic.
Spark applications
- New parameter to automatically restart driver during disruptions
- With the new spark.ego.driver.blockhost.supervise parameter, you can automatically restart the driver if the driver fails to start with a specific range of driver exit codes. For more information, see Spark on EGO Spark application parameters.
Cluster and resource management
- Extended RESTful APIs for multidimensional resource scheduling
- In addition to the existing GET method to create a new multidimensional resource plan, this
release of IBM Spectrum
Conductor includes
eight new RESTful APIs for managing multidimensional resource scheduling. See the
MultidimensionalScheduling section within the curent version of the REST API for
/platform/rest/ego/v1 for details; the following table highlights each
API:
Table 1. RESTful APIs for multidimensional resource scheduling Method Description and URL New or existing GET Get a list of all multidimensional resource plans in the cluster https://host_name:port/platform/rest/ego/v1/mdsplans
Existing GET Get a multidimensional resource plan by name https://host_name:port/platform/rest/ego/v1/mdsplans/{mdsPlanName}
New POST Create a new multidimensional resource plan https://host_name:port/platform/rest/ego/v1/mdsplans
New PUT Modify an existing multidimensional resource plan https://host_name:port/platform/rest/ego/v1/mdsplans/{mdsPlanName}
New DELETE Delete a multidimensional resource plan https://host_name:port/platform/rest/ego/v1/mdsplans/{mdsPlanName}
New GET Export the XML of all existing multidimensional resource plans in the cluster https://host_name:port/platform/rest/ego/v1/mdsplans/xml
New GET Export the XML for a single existing multidimensional resource plan in the cluster https://host_name:port/platform/rest/ego/v1/mdsplans/xml/{mdsPlanName}
New PUT Import the XML of a multidimensional resource plan to update an existing multidimensional resource plan https://host_name:port/platform/rest/ego/v1/mdsplans/{mdsPlanName}
New POST Import the XML of a multidimensional resource plan to create a new multidimensional resource plan https://host_name:port/platform/rest/ego/v1/mdsplans/xml
New - Set overcommit ratios for resource metrics at the multidimensional resource group level
- In addition to setting resource overcommit ratios at the multidimensional resource plan level, you can now set this for each resource group within your plan. Leverage this enhancement following the steps in Setting resource overcommit ratios in your multidimensional resource plan.
- New parameter to set how LIM gets the IP address from a hostname
- Configure the new EGO_GET_HOST_IP_ADDRESS_WITHOUT_LIM_DNS_CACHE parameter in the ego.conf file to Y, to specify for LIM to ignore the host's DNS cache and to use the EGO host file or system API instead. This configuration is useful if unavailable hosts cannot be removed automatically.
Performance and stability
- Configure the number of connections that can remain in the LIM and VEMKD backlogs
- For improved connection stability, you can now configure the number of connections that can remain in the LIM and VEMKD backlogs, by configuring the new EGO_PLATCOMMDRV_BACKLOG_SIZE_ON_LINUX setting in the ego.conf configuration file.
Security
- Terminology update: TLS (Transport Layer Security) protocol for security
- Starting in IBM Spectrum
Conductor
2.5.1, the documentation has
been updated to refer SSL as TLS for the IBM Spectrum
Conductor security protocol;
functionality of IBM Spectrum
Conductor
security has not changed. TLS uses encryption to protect the transfer of data and information, in
the same way as SSL.
This terminology change is simply to keep up with the evolving industry terminology change. IBM Spectrum Conductor documentation now refers to TLS as the supported protocol, as much as possible. Some parameter names, environment variables, attribute values, messages, and so on, continue to SSL in their names, but can be treated as TLS protocol.
- Secure built-in
Admin
andGuest
user credentials - For added security, the
Admin
andGuest
user credential values and usage have changed:Admin
- You can now set the password for the built-in
Admin
user, rather than use a preset password by following Set the built-in Admin user password during fresh installations.
Guest
- You no longer need to use
Guest
as the username and password. For anywhereGuest
is used (such as a sample or an example), treat as for explanatory purposes only; replaceGuest
with your real username and password for your cluster.
- TLS support for Dockerized service instances
- When you use Docker to run IBM Spectrum Conductor service instances, by default, Docker service instances use the UDS transport protocol. You can optionally enable service instances in a TLS-enabled Docker environment. To set TLS protocol in your instance groups, see Setting Docker or cgroup container definitions for an instance group.
Host factory framework for cloud bursting
- Supporting IBM Cloud Virtual Servers for VPC Gen 2 as a cloud service provider
- Host factory for IBM Spectrum
Conductor adds IBM
Cloud Gen 2 as an
additional cloud service provider. For more information, see Configuring your cluster for IBM Cloud Gen 2,
ibmcloudgen2prov_config.json
, andibmcloudgen2prov_templates.json
topics. - Manage requestor plug-ins and instances, or provider plug-ins and instances
- You can view, create, update, and delete a requestor plug-in, requestor instance, provider plug-in, or provider instance from the cluster management console:
- View and update configurations
- View and update service-level host factory configuration. For more information, see Viewing configurations.
- Enhancements for the AWS provider plug-in
-
- New parameters for timeout and retries
- Introducing three new parameters that you can use to get status of instances from the AWS provider. Configure AWS_INSTANCE_PENDING_TIMEOUT_SEC for specifying the timeout before a pending instance is terminated, AWS_DESCRIBE_REQUEST_RETRY_ATTEMPTS for the number of retries, and AWS_DESCRIBE_REQUEST_INTERVAL for delay between retries. For more launch information about how to configure these new parameters, see awsprov_config.json reference.
- New parameters for AWS EC2 instances
- Introducing six new
parameters for AWS EC2 instances:
- Configure volumeType and iops for EC2 On-Demand and Spot instances.
- Configure percentOnDemand, vmTypesOnDemand, vmTypesPriority, or allocationStrategyOnDemand for EC2 On-Demand instances.
For more details on how to configure these new parameters, see awsprov_templates.json reference.
- New option for allocationStrategy
- Introducing a new option for spot instance allocationStrategy parameter. Specify the new capacityOptimizedPrioritized option if you want to give certain Spot instances types a higher chance of launching first. For more information about the new capacityOptimizedPrioritized option for the allocationStrategy parameter, see awsprov_templates.json reference.
- Removed
ncores
as a value for theattributes
parameter in all host template provisioning files - The
attributes
parameter has been changed for the host template provisioning files: thencores
value is no longer supported; use thencpus
value instead. All host template files (provider_nameprov_templates.json) under Reference for host factory and template examples have been updated to no longer showncores
in the templates.For backwards compatibility, the HostFactory service and plug-ins continue to work if
ncores
was specified previously. IBM Spectrum Conductor uses the default value of 1 (that is, if you configuredncores
to another value, it will not take effect and the value of 1 will be used instead).