Using extensions in IBM Cloud Private Cloud Foundry

In IBM Cloud Private Cloud Foundry, an extension is a package that contains a deployment process and its required scripts and files. You can use the extension framework to integrate your deployments with your Cloud Foundry deployment or run them independently.

You can phase the deployment process in multiple steps. The configuration manager runs the steps, or states, sequentially according to the order that you set in the manifest file, or that you set in the dependencies between states. In the manifest file, you provide step names, the name of a script to run at each step, and any required parameters and their values. When you run a deployment, the steps run following a topology sort order until a script execution fails or all steps complete. If the execution of the script that is associated with the step fails, that step is marked as FAILED, and all successful steps are marked with SUCCEEDED.

Creating an extension
Running the extension

Creating an extension

You create extensions by writing scripts for each deployment step, creating a manifest file that outlines the step execution order and any parameters, and compressing the manifest and step files.

Create the scripts to run your deployment steps. You must be able to run these scripts on a computer that uses Ubuntu.
Create the manifest file. The manifest file must be located in the root directory of your extension as extension-manifest.yml and it must contain a states section that provides the step sequence and the name of the script for each step. You can find an example of the extension-manifest.yml in the <your_data_directory>/extensions/embedded/cfp-bosh-templates. The states section will be extracted during the registration of the extension and stored as states-file.yml in the extension directory. If you need to provide other parameters, you can create more sections to contain them.

Note: You have to specify the next_states attribute for each state and then a topology sort is executed to calculate the execution sequence. Or, if you don't specify the next_states attribute, then the execution order follows the sequence of the state listed in the states section of the extension-manifest.yml.

The states section takes the following form:
```
   states:
   - name: task1
     log_path: /tmp/task.log
     status: READY
     start_time:
     end_time:
     reason:
     script: scripts/success.sh task1
     script_timeout: 10
     next_states: [ "task2", "task3" ]
   ...
```
See the State parameters table.

The extension-manifest.yml contains global parameters.
- The global parameter states_update_mode describes what must be done when an extension is re-registered. See the registration section. It can take the values merge, replace, or new. merge is the default value.
- The global parameter validation_config_url allows you to call a web service to validate your configuration. It contains the URL of that web service. The configuration is posted to that web service. The web service must return a response for each attribute of the configuration. The response includes a value attribute that is set to the current value of the attribute and a message_type which includes the value warning or error.
  
  Response example:
```
uiconfig:
bluemix_apps_domain:
  value: local.mybluemix.net
...
cluster_name:
  message: 'ESX: 10.1.1.10 is not accessible using port 443. dial tcp 10.1.1.10:443:
i/o timeout'
  message_type: warning
  value: MY_CLUSTER
...
```
- The global parameter generate_config_url allows you to call a web service to transform the saved configuration if needed.
  
  Optionally, you can add a call_state section that inserts a step in the parent states file from where the extension must be called. If you add a call_state, you cannot create a state.yml file to use to add the calling step to the parent states file.
  
  The call_states section takes the following form:
```
call_state:
 phase: AtEachRun
 previous_states: [ <previous_states> ]
 next_states: [ <next states> ]
```
  The call_states section uses the same default values as the states section. Specifying the previous_states and next_states will ease the extension insertion process. You will not have to provide information in the ./cm states insert -i <extension_name> -n <insertion_state_position_reference> -b command or when you register the extension by using the user interface. ./cm states insert -i <extension_name> will suffice to insert the extension at the correct place in the states file.
  
  The ui_metadata section takes the following form:
```
ui_metadata:
 production:
   label: "Production environment"
   groups:
   - name: "network"
     title: "Network"
     properties:
     - name: "console_ip"
       label: "Console IP"
       description: "The IP address of the console"
       type: "text"
       validation_regex: "^(?:(?:25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9]?[0-9])\\.){3}(?:25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9]?[0-9])$"
       validation_error_message: "The field must be an IP address"
       mandatory: true
       hidden: false
       sample_value: "E.g. 10.10.1.12"
       default: "10.10.1.12"
   - name: "security"
     title: "Security"
     properties:
     - name: "encryption"
       label: "Encryption"
       properties:
       - name: "encryption_type"
         label: "Encryption type"
         type: "text"
       - name: "method"
         label: "Method"
         type: "text"

     ...
 development:
   label: "Development environment"
   groups:
   - name: "Security"
   ...
```
  The ui_metadata section is composed of a list of configurations. In the previous example, the configuration names are production and development. The UI includes a drop-down box to select the configuration that you want to use. The label of the configuration is used to populate the drop-down box. If you omit the label, the configuration name is used. Each configuration contains a list of groups defined under groups. Each group is represented by a tab in the UI and has a name, a title, and a list of properties. Each property has a number of attributes including the properties attribute, which allows you to define a hierarchy of properties.
  
  See the UI metdata parameters table.
  
  The following code is property examples.
  - dropdown:
```
 - name: "vmware_disk_type"
   label: "VMware disk type"
   description: "VMware default disk type (thin/preallocated)"
   type: "dropdown"
   mandatory: true
   default: thin
   items:
   - id: thin
     label: thin
   - id: preallocated
     label: preallocated
```
  - checkbox:
```
 - name: enabled
   label: "Enable backup"
   description: "If checked the backup will run on the provided schedule"
   type: checkbox
   default: true
   sample_value: 'true'
```
  - hierarchy:
```
 - name: "bbr_backup"
   label: "BBR backup setup"
   description: "BBR backup setup for director/deployment"
   properties:
   - name: customer_nfs_host
     label: "Customer NFS host"
     type: "text"
     default: ''
     sample_value: 'mynsf.mycompany.com'
   - name: customer_nfs_path
     label: "Customer NFS path"
     type: "text"
     default: ''
     sample_value: '/cfbackup'
```
Create a folder for your extension and copy the scripts and manifest file to the folder. Place the manifest file in the root of the directory. You can place the script files in any sub-folder, but ensure that the script parameter values contain the correct relative file paths. The folder structure resembles the following example:
```
extension-manifest.yml
scripts/
scripts/success.sh
scripts/README.md
```
Compress the extension folder to a .zip file. The archive file must be compatible with the ZIP64 format. See .ZIP File Format Specification. The zip command on Ubuntu and iOS is compatible with ZIP64.

State parameters

Table 1. State parameters
Parameter	Description	Value	Default value	Required
`name`	The name of the state.	Alphanumeric string	n/a	True
`label`	The label of the state.	Alphanumeric string	n/a	False
`script`	The script to run during the state. You can provide the relative path from the root of the extension .zip file. If you provide an absolute path, it is a path in the inception container.	File path	n/a	True
`phase`	When the script runs. If you set this parameter value to `AtEachRun`, the script runs even if the state status is `SUCCEEDED`. If `phase` is blank or omitted then the states will run only if the status is `READY` or `FAILED`.	`AtEachRun` Blank	Blank	False
`log_path`	The path of the log for that state. When the script for a state runs, the previous execution log is backed up.	File path	/data/logs/extensions/custom/extension_name.log	False
`time_out`	The maximum time that the script can run. If the script does not complete in the allotted time, it stops and the state status changes to `FAILED`.	Time in minutes	60	False
`status`	The state status. The state status changes after its script runs.	READY SKIP RUNNING SUCCEEDED FAILED	n/a	True
`start_time`	The start time stamp of the script in Coordinated Universal Time. This value is changed by the configuration manager.	n/a	n/a	False
`end_time`	The end time stamp of the script in Coordinated Universal Time. This value is changed by the configuration manager.	n/a	n/a	False
`reason`	Why the state failed. This value is changed by the configuration manager during the deployment.	n/a	n/a	True
`protected`	If the state is protected, the end-user cannot remove the state.	n/a	False	False
`deleted`	If this parameter is set to true then that state is removed from the `states-file.yml` after merge.	n/a	false	False
`prerequisite_states`	If the current states are READY/FAILED then the listed states are set to READY too. Each listed state must be before the current state.	array of string	none	False
`rerun_on_run_of_states`	If one state on the list is READY/FAILED then the current state is set to READY.	array of string	none	False
`states_to_rerun`	If the current state is READY/FAILED then the listed states are set to READY too. Each listed state must be after the current state.	array of string	none	False
`previous_states`	This is a calculated field.	n/a	none	False
`next_states`	Array of states names that must run after this state. Use this parameter to to define the order in which you want to execute the states. The deployment engine does a topological sort based on this array. If you set it for one state, you must set it for all states. If you do not specify this parameter, the next state in the states file is considered as the next state to execute.	n/a	none	False

UI metadata parameters

Table 2. ui_metadata parameters
Attributes	Description	Value	Default value	Required
`name`	The name of the attribute. This name is used in the configuration file. The name must follow the same format of a YAML tag (YAML tag) and must not contain a dot(.) or an exclamation mark(!).	Alphanumeric string	n/a	True
`label`	The label to display in the UI.	n/a	The value defined by name	False
`description`	The description to display in the UI.	n/a	n/a	False
`type`	The type can be `text`, `textarea`,`number`, `checkbox`, `dropdown` and `array`.	n/a	text	False
`items`	Used when type is equal to `dropdown` and contains an array of objects. Each object being id and label.	n/a	text	False
`validation_regex`	The regular expression to validate the entry.	n/a	n/a	False
`validation_error_message`	The message to display if the entry does not match the regular expession.	n/a	n/a	False
`mandatory`	Defines if the field is mandatory.	true/false	true	False
`hidden`	If true the field will be not shown in the UI.	true/false	false	False
`sample_value`	The value to display as a hint in the UI	n/a	n/a	False
`default`	The default value for the field.	n/a	n/a	False

Notes:

The validation_regex and mandatory attributes are not enforced and do not prevent you from saving the configuration. The extension designer should implement extra validation in the scripts associated with the extension states.
A value is double-quoted if quoted or double-quoted in the UI. This is useful when you want to set numbers as a string.

Type array is only supported for arrays of text fields and arrays of objects as illustrated in the following examples.

Array of text fields

- name: "default_security_groups"
  label: "Default security groups"
  description: "Security group used for each AMI to control network traffic"
  type: "array"

Array of objects

- name: deployments
  label: "Deployments"
  description: "Deployments to backup"
  type: array
  properties:
  - name: name
    label: "Deployment name"
    description: "The deployment name to backup"
    type: text
    sample_value: "My deployment"

Running the extension

When you run an extension, the following rules are applied:

The first script with status READY or FAILED runs.
The extension execution stops when a script execution fails.
Scripts with the status SKIP or SUCCEEDED do not run.
If you set the phase parameter value to AtEachRun, the script for that step runs each time that you start the engine, even if its status is SUCCEEDED.
While a script runs, its status is set to RUNNING.

Register the extension on the inception container. Registering an extension uploads the contents of the extension archive file to the /data/extensions/custom/<extension_name> folder on the inception container and extracts the states section to create a states-file.yml file. If the extension was already registered, the states-file.yml already exists. Depending on the states_update_mode (merge|replace|new), the current states-file from a previous registration will be either merged, replaced or the new states-file.yml will be named states-file-new.yml.

Note: You do not have to register extensions that are provided by IBM . They are already registered. Run the following command to register extensions. <extension_name> is the name of the extension and <archive_path> is the path and name of the compressed archive folder.
```
 ./cm extension -e <extension_name> register -p <archive_path>
```
Confirm that the extension is registered. Run the following command to verify that the extension is registered and that the extension name is listed in the output:
```
 ./cm extensions
```
If your scripts contain variables, you can create a configuration file to define the variables and provide them to the extension.
1. Create the configuration file. The file must be in YAML format and start with a uiconfig section. For example, to set the value of a servicebroker_port parameter, the contents of the configuration file resemble the following text:
```
uiconfig:
 servicebroker_port: 8080
```
2. To provide the configuration file to the inception container, run the following command. <extension_name> is the name of the extension you registered, and <config_file> is the name of your configuration file. The configuration file is stored as the uiconfig.yml file in the /data/extensions/custom/<extension_name> folder on the inception container. The scripts can use the configuration file to retrieve each parameter that they need at run time.
```
./cm extension -e <extension_name> save -c <config_file>
```
Deploy the extension. You can integrate the extension with the main IBM Cloud Private Cloud Foundry deployment, add the extension to the deployment of another extension, or deploy the extension by itself.

Important: The configuration manager tool, ./cm, is automatically installed when you deploy IBM Cloud Private Cloud Foundry, so you must run the cm commands from the installation directory.
- To insert the extension into the IBM Cloud Private Cloud Foundry deployment, complete the following steps:
  1. View the IBM Cloud Private Cloud Foundry states.
```
./cm states
```
  2. If the extension manifest does not contain a call_step section, then create a state_cf.yml file for the IBM Cloud Private Cloud Foundry deployment. This file is the template file that is inserted in the IBM Cloud Private Cloud Foundry deployment as a new step to run. The script parameter is automatically set to call the extension. Parameters are set by default as described in the previous table. The file contains the following text:
```
name: <extension_name>
```
    Add the extension to the IBM Cloud Private Cloud Foundry state file. <state_name> is the state to run the extension after. If you need to run the extension before a state, replace the -n option with the -b option.
```
./cm states insert -s state_cf.yml -n <state_name>
```
  3. If the extension manifest contains a call_step section, you can add the extension to the IBM Cloud Private Cloud Foundry state file as illustrated in the following command:
```
./cm states insert -i <extension_name> -n <state_name>
```
    The <extension_name> is the name of the extension you want to run and <state_name> is the state to run the extension after. If you need to run the extension before a state, replace the -n option with the -b option. The <state_name> is not requested if the previous_states and next_states are defined in the call_state of the extension.
  4. To verify the updated state order, rerun the ./cm states -e extension_name command.
  5. Run the launch_deployment.sh command to deploy IBM Cloud Private Cloud Foundry and your extension.
- To insert the new extension in an existing extension, complete the following steps. <extension_name> is the existing extension.
  1. View the states of the existing extension.
```
./cm states -e extension_name
```
  2. Create a state_<extension_name>.yml file for the extension, where <extension_name> is the name of the existing extension. This file is the template file that is inserted in the extension deployment as a new step to run. The script parameter is automatically set to call the extension. Parameters are set by default as described in the previous table. The file contains the following text:
```
name: <extension_name>
```
  3. Add the new extension to the existing extension state file:
```
./cm states -e <extension_name> insert -s state_<extension_name>.yml -n <state_name>
```
    <extension_name> is the existing extension, and <state_name> is the state to run the extension after. If you need to run the extension before a state, replace the -n option with the -b option.
  4. To verify the updated state order, run the ./cm states -e extension_name command again.
  5. Deploy the existing extension:
```
./cm extension -e <extension_name> deploy
```
    The <extension_name> parameter is the name of the existing extension.
- To deploy the extension by itself, run the following command:
```
  ./cm extension -e <extension_name> deploy
```
  The <extension_name> parameter is the name of the extension. This command runs in the background, but you can add a -w flag to the command to wait for the deployment to finish.
  
  During the deployment, a states-file.yml file is created in the extension directory. The configuration manager engine updates the file. Each entry is updated during the deployment with default values, if needed, a status, and start and stop time stamps.
  - The script time stamps in Coordinated Universal Time. The time stamp is set at the beginning and the end of the script execution.
  - The statuses are automatically set based on the exitCode of the scripts. If the script does not finish execution (exit code not equal zero), the script status is marked as FAILED.
  - The StdOut and StdErr output from the script are recorded in the log file.
  - The process stops and is marked as FAILED if timeout value is exceeded.
Monitor the extension: You can check the log files of an extension by running the ./cm extension -e extension_name logs command. You can add the -f flag to follow the logs. The different log files of the deployment are displayed successively.
Optional: Unregister the extension. Unregistering the extension removes the extension data from the inception engine, but does not remove the deployment from your cluster. After you deploy the extension, you cannot remove it unless you create an extension that contains the steps to remove the installed extension. To unregister the extension, run the following command:
```
./cm extension -e extension_name unregister
```
<extension_name> is the name of the extension.
Optional: If you added the extension to another extension but do not want to deploy the new extension the next time that you deploy the existing extension, you can remove it. Run the following command:
```
./cm states -e extension_name delete -n state_name
```
<extension_name> is the existing extension, and <state_name> is the state that you run the extension after.