Upgrading and deleting deployed nodes
ATTENTION!! IBM Blockchain Platform Software Edition is being replaced by IBM Support for Hyperledger Fabric!! IBM Blockchain Platform Software Edition will no longer be supported after April 30, 2023. Customers have been directed to migrate their networks by April 30, 2023. After this date, IBM Blockchain Platform software networks that are not migrated to IBM Support for Hyperledger Fabric will be at risk for potential security vulnerabilities. Migration scripts are provided, and the disruption to your network is minimal. See Migrating to IBM Support for Hyperledger Fabric for details.
After creating CAs, peers, and ordering nodes, you need to monitor the resources used by the nodes and potentially reallocate resources.
Target audience: This topic is designed for network operators who are responsible for creating, monitoring, managing, and upgrading their components in the blockchain network.
After a node has been created, there are three main ways to update it.
- Update its configuration. Just as it is possible to override configuration parameters when deploying a node, it is possible to edit many parameters and redeploy the node. For more information, see our topic on Advanced deployment options.
- Increase the resources allocated to it. If you notice that the performance of a node is beginning to lag or that its storage is beginning to run out, you can choose whether to deploy a larger node and join it to channels or whether to increase the resources allocated to your existing node.
- Upgrade its Fabric version. Every node is deployed using a release version of Hyperledger Fabric. When new versions of Fabric become available on IBM Blockchain Platform, Upgrade available is visible on the tile representing the node. While upgrading to a new version of Fabric is always recommended, unless is rendered necessary by the capabilities of channels you want to join, it is typically optional. For more information, check out Upgrading to a new version of Fabric.
In this topic, we'll cover the process for reallocating resources, upgrading to a new version of Fabric, and deleting nodes.
Considerations when reallocating resources
Resizing a node requires the containers to be rebuilt, which can cause a delay in the functioning of the node.
Tools such as IBM Cloud Monitoring can be used to help monitor the usage in your cluster. If you determine that a worker node is running out of resources, you can add a new larger worker node to your cluster and then delete the existing working node.
While it takes less effort to deploy enough resources to your Kubernetes cluster from the start and therefore be able deploy and expand resources without having to increase the resources in your cluster, the bigger the deployment, the more it will cost. Users need to consider their options carefully and recognize the tradeoffs that they are making regardless of the option that they choose.
You can scale your cluster by monitoring your nodes and following the instructions available from your cloud provider to add more nodes, larger nodes, or increasing the size of the nodes, depending on the options available in your cloud provider. The method you will use to increase storage will depend on the storage class you chose for your cluster. Note that if you are about to exhaust the storage on your peer or ordering node, you might need to deploy a new peer or ordering node with more storage and let it sync via your other components on the same channels.
Note that you do not need to adjust the CPU, memory, or storage for your smart contract pods. These pods will automatically use as many resources as they need to function efficiently. In cases where your smart contracts are struggling because of insufficient resources, you will have to address this at the cluster level.
Upgrading to a new version of Fabric
Support for Hyperledger Fabric v1.4 is now deprecated, and support for Fabric v1.4 will be removed from IBM Blockchain Platform on March 31, 2023. Users should therefore upgrade to Fabric v2.2 as soon as possible. Your applications may require changes as a result of upgrading to v2.2, so please plan for appropriate testing. Note that Fabric v1.4 has not been supported by the Hyperledger community since April of 2021. In addition, Fabric v1.4 uses Golang v1.14, which is no longer receiving security updates from the Golang community.
While some new versions of Fabric only require updating the Fabric version on nodes, some include new channel capabilities that must also be updated.
In these cases, the process of "updating to the latest" release is, at a high level, a two step process:
- Upgrade the Fabric version on all nodes.
- Update the channels to the new capability levels. For information about how to update channels, see Capabilities.
You must upgrade nodes before you update the channels. If a node attempts to read a configuration block containing a capability level it does not understand (which is true in cases where the capability is a higher level than the node version), the node will crash on all channels. The node must then be upgraded to the appropriate Fabric version before it can be used again.
The process of upgrading a node involves two main steps:
- Backing up the persistent volumes associated with the node. These backups ensure that in the case of an upgrade failure in which the peer pod is corrupted that the node can be re-deployed using the ledger. For more information, see Backup considerations for each node type.
- Upgrading the Fabric versions of the nodes one at a time (also known as a "rolling upgrade").
It may also be necessary to update SDKs and smart contracts before you can take advantage of the latest Fabric features. For more information, check out Step three: Update SDKs and smart contracts.
Upgrading nodes from Fabric v1.4 to v2.2
Use the following recommended procedure to upgrade your Fabric v1.4 peer and orderer nodes to v2.2, regardless of whether you are also migrating your v1.4 chaincode to run on Fabric v2.2. Once on v2.2, you can migrate your nodes to v2.4.
Attention: To update to Fabric v2.2, the recommended process (below) adds a new peer, rather than updating an existing peer in place. Adding a new peer avoids the extended time to rebuild CouchDB (if applicable) and downtime when an existing peer is updating. The final step then deletes the replaced peer.
Update your CAs
Update your Certificate Authority (CA) nodes before you upgrade your peer and orderer nodes, as follows:
Attention: Most application solutions do not interact with CAs continually, in which case CAs can be upgraded with no impact to application usage. However, any application flows that use fabric-ca functionality may be unavailable for 1-2 minutes during the update.
- Using your console, navigate to your Certificate Authority and select the first CA node. Click on upgrade available and select 1.5.5-2 or later.
- The CA node will restart with the upgrade installed.
- Repeat the steps for each CA node.
Update your orderers
After updating your Certificate Authority (CA) nodes, update your orderer nodes, as follows:
- Using your console, navigate to the ordering service and select the first ordering node. Click on upgrade available and select 2.2.10-1 or later. Click through the confirmations dialogs.
- The node will restart using the 2.2.10-1 (or selected) image. Transaction processing will continue during the node restart, because the remaining four orderer nodes make a quorum. The selected orderer node will be unavailable for 1-2 minutes during the update. ATTENTION!!: If the updating orderer node does not restart and come back online successfully, please contact IBM Support. DO NOT attempt to update any remaining orderer nodes before contacting IBM Support.
- Repeat the procedure for each remaining orderer node.
Update your peers
After updating both your Certificate Authority (CA) and orderer nodes, update your peer nodes, as follows:
ATTENTION!!: When creating new v2.2 peers using the steps below, selecting 2.2.10-1 is highly recommended - other selectable versions may have chaincode implications.
-
Before upgrading any peer with CouchDB installed, you must rebuild the CouchDB state database from CouchDB v2.x to v3.x. The recommended process is to add a new peer, and then let the database synchronize. This removes the downtime that would occur when upgrading in place. Then continue as follows, for peers both with and without CouchDB:
-
Add a new peer using your console. Do NOT install chaincode on the new peer.
-
Remove the new peer from both peer gossip and service discovery, as follows:
a) Get the Custom Resource Definition (CRD) of the new peer, by running:
kubectl get ibppeer -n [NAMESPACE]
b) Back up the CRD:
kubectl get ibppeer [IBPPEER_NAME] -n [NAMESPACE] -o yaml > ibppeer_crd_backup.yaml
c) Edit the CRD:
kubectl edit ibppeer [IBPPEER_NAME] -n [NAMESPACE]
d) Change
spec.peerExternalEndpoint:
todo-not-set
as follows:spec.peerExternalEndpoint: do-not-set
e) Delete the peer deployment:
kubectl get deployment -n [NAMESPACE]
and then:kubectl delete deployment [PEER DEPLOYMENT NAME] -n [NAMESPACE]
f) The peer will restart - then it should not be discoverable by application service discovery. Do NOT update the client connection profiles at this time.
-
Add the new peer to the channel(s) that existing peers are participating in. Do NOT install channel chaincode on the new peer.
-
Let the new peer synchronize its blocks, to the block height of the existing peers.
-
Reenable PEER_GOSSIP on the new peer, as follows:
a) Get the Custom Resource Definition (CRD) of the new peer, by running:
kubectl get ibppeer -n [NAMESPACE]
b) Back up the CRD:
kubectl get ibppeer [IBPPEER_NAME] -n [NAMESPACE] -o yaml > ibppeer_crd_backup.yaml
c) Edit the CRD:
kubectl edit ibppeer [IBPPEER_NAME] -n [NAMESPACE]
d) Change
spec.peerExternalEndpoint
to a blank string, using empty quotation marks, as follows:spec.peerExternalEndpoint: ""
e) Delete the peer deployment:
kubectl get deployment -n [NAMESPACE]
and then:kubectl delete deployment [PEER DEPLOYMENT NAME] -n [NAMESPACE]
f) The new peer will restart - then it should begin participating in service discovery and getting new blocks. Application targeting will depend on the connection profile and application logic.
-
Install chaincode on the new peer.
-
Test the new environment by running applications with all peers (original and new) enabled.
-
(Recommended) Test applications by turning off the peer that is being replaced, as follows:
a) Run:
kubectl patch [ibppeer-name] -n [namespace] -p=
[{"op": "replace", "path":"/spec/replicas", "value":0}]- type=json
b) Ensure application continuity with the original peer is NOT running.
-
Delete the original peer.
-
Repeat the prior steps for each peer you are updating.
Upgrading nodes from Fabric v1.4 to v2.4
Upgrading IBM Blockchain Platform nodes directly from Hyperledger Fabric 1.4.x to the latest Fabric version is possible, but deploying a new Fabric 2.2.x peer instead of upgrading is recommended. Fabric installation is done by IBM Blockchain Platform, but can take hours or days depending on the size of the database to be built. See the Fabric documentation on upgrading for more information.
If your Fabric v1.4 nodes use Node.js chaincode, use the following sequence to upgrade these nodes from Fabric v1.4 to v2.4:
- Deploy new peers with Fabric v2.2
- Update the Node.js chaincode on these peers to 2.4 shim
- Install and instantiate the new chaincode
- Upgrade the peers from Fabric v2.2 to v2.4.
Step one: Back up your ledger (optional)
It is recommended to take regular backups of the persistent volumes of your nodes as part of the normal process of network administration. For example, in our topic on backing up and restoring components and networks, an example schedule is provided which recommends that backups be taken of the ordering nodes and the peers (including the state database of the peer, if using CouchDB) each night.
However, if you are not taking regular backups, it is recommended that you minimally take a backup before attempting to upgrade a node, as it allows for a node to be restored to an earlier running state in cases where the upgrade fails. For information about how to backup your nodes by taking a snapshot of the relevant persistent volumes, check out Taking snapshots.
Step two: Upgrade your nodes one at a time
If you are upgrading both peer and ordering node binaries, it is a best practice to upgrade the ordering nodes first, as ensuring that the ordering nodes (and by extension, the ordering service) is functioning correctly is more important to the health of your network as a whole than the functioning of any particular peer.
The process for upgrading a node is relatively straightforward. First, make sure you are using the console where the node was created. You cannot use the console to update imported nodes. When a node upgrade is available, Upgrade available is visible on the node tile. If Upgrade available does not appear on the tile when it should be there, make sure you have upgraded to the latest version of the console.
You can then update the node:
- Click on the tile representing the node.
- The upgrade panel can be accessed either by clicking the Info and usage tab, next to the Upgrade available notice, or the Fabric version box. Clicking either one opens the Upgrade Fabric version side panel.
- On the Upgrade Fabric version side panel, review the version of your node and the version you are upgrading to. If this is right, click Next.
- On the next panel, confirm the information and enter the name of the node being upgraded.
The node will be unavailable during the upgrade. The status was turn grey and will read Status unknown or Unavailable. When the upgrade has completed, the status will turn green and be Ready. If the upgrade fails and the node lapses into an unrecoverable state, follow the instructions for Restoring nodes from a snapshot.
It is a best practice to only upgrade one node of each type at a time. In other words, if you need to upgrade both peers and ordering nodes, you can start a single peer upgrade and a single ordering node upgrade at the same time. However, do not attempt to upgrade multiple peers or multiple ordering nodes at the same time, as this threatens the availability of your components.
It is not possible to upgrade an ordering node if any node in your ordering service is down for any reason. Coordinate with all of the administrators of your ordering service before attempting to upgrade.
Step three: Update SDKs and smart contracts
It is a best practice to upgrade your SDK to the latest version as part of a general upgrade of your network. While the SDK will always be compatible with equivalent releases of Fabric and lower, it might be necessary to upgrade to the latest SDK to leverage the latest Fabric features. Also, after upgrading, it's possible your client application may experience errors. Consult the your Fabric SDK documentation for information about how to upgrade.
Although support for Fabric 2.0 networks was added to the platform, you can still run your existing smart contracts on your peers that run a v1.4 image on a channel with an application capability level of 1.4 or lower. Should you later decide
to upgrade your peer to a v2.x image and update your channel application capability level to 2.0, you may need to update your existing smart contract. However, after you upgrade your peer image to v2.x and channel application
capability v2.x, there is no longer a way to update the original smart contract. Instead, when an update is required, you need to repackage the smart contract in the new .tar.gz
or .tgz
format using v2 of the VS
Code extension and then propose the definition to the channel using the new smart contract lifecycle process.
Review the following considerations:
Node
If your smart contract was written in Node, then you might need to update it. By default, a Fabric v1.4 peer will create a Node v8 runtime, and a Fabric v2.x peer creates a Node v12 runtime. In order for a smart contract to work with Node
v12 runtime, the fabric-contract-api
and fabric-shim
node modules must be at v1.4.5 or greater. If you are using a smart contract that was originally written to work with Fabric 1.4, update the Node modules by running
the following command before deploying the smart contract on a Fabric v2.x peer. See Support and compatibility for fabric-chaincode-node for more
information.
npm install --save fabric-contract-api@latest-1.4 fabric-shim@latest-1.4
Go
Because Fabric v2.x peers do not have a "shim" (the external dependencies that allowed smart contracts to run on earlier versions of Fabric), you need to vendor the shim and then repackage any smart contracts written in Golang (Go) that use the Go SDK. "Vendoring the shim" effectively means that you are copying the dependencies into your project. Without this vendoring and repackaging, the Go smart contract cannot run on a peer using a Fabric 2.x image. If you are using the IBM Developer Tools to develop and package your smart contract, the tooling performs the vendoring for you. This process is not required for smart contracts that are written in Java or Node.js, nor for Go smart contracts that use the Go contract-api.
Java
The build.gradle
file for the smart contract must be updated:
- If the smart contract uses the
shadowjar
2.x plugin, then it should be updated to version 5 by using the following code:plugins { id 'com.github.johnrengelman.shadow' version '5.1.0' id 'java' }
- The
repositories
section of the file must contain themaven URL
forjitpack
, for example:repositories { ... maven { url 'https://jitpack.io' } }
Init functions
If the smart contract was written using the low-level APIs provided by the Fabric Chaincode Shim API, your smart contract needs to contain an Init
function that is used to initialize the chaincode. This function
is required by the smart contract interface, but does not necessarily need to be invoked by your applications. Because you cannot use the console to deploy a smart contract that contains an Init
function, you need to move that
initialization logic into the smart contract itself and call it separately. For example, the smart contract can use a reserved key to check whether the smart contract has already been initialized or not. If not, then call the initialization
logic, otherwise proceed as usual. If your smart contract needs to include the Init
function, the only way to deploy it is by using the Fabric peer lifecycle chaincode install command or the IBM Blockchain Platform collection for Ansible. You
can also refer to the Fabric documentation for
more details on how to use an Init
function with the Fabric chaincode lifecycle.
Repackage smart contract
Attention! The IBM Blockchain Platform Extension for VS Code (VS Code extension) referenced throughout the documentation is an open-source project which is no longer active, and therefore not officially supported by IBM. Refer to the alternatives to the VS Code extension.
After you have updated your smart contract, use v2 of the VS Code extension to repackage your smart contract.
For a look at how the new lifecycle is administered in the console, check out Deploy a smart contract using Fabric v2.x. For a look at the possibilities the new lifecycle opens up, check out Writing powerful smart contracts.
Step four: Update capabilities
Once your nodes, SDKs, and smart contracts have been upgraded to use the latest Fabric version, you can update your channel configuration to use the latest capabilities. Note that the Fabric version of your nodes must be at least at the corresponding capability level of the channel the node is joined to.
For more information about capabilities and how to update a channel configuration to enable them, check out Capabilities.
Deleting components
The best practice for deleting components is to delete them using the console. This will also delete all of the artifacts associated with a node including your ledger data in persistent storage and the keys that are stored as secrets. Deleting a peer will not, however, delete any smart contract pods associated with it. These must be deleted separately. Deleting a component is usually achieved by logging onto the console where a component was created or installed, clicking on the component and finding the related trash can icon. You will typically be prompted to type the name of the component and to confirm your decision. You can also delete nodes by using the IBM Blockchain Platform APIs.
However, there are cases in which this type of deletion will not be successful. For example, occasionally when a node fails to deploy it will not be possible to delete it using the console. The same can be true if the console loses connection with the cluster for some reason.
In these cases, it will be necessary to delete the node or relevant pods manually. Your Kubernetes cluster on IBM Cloud manager of choice might have a UI that allows you to delete pods. Check the documentation for your cluster for instructions.
Because smart contracts installed on a 2.x peer are deployed into their own pods and not directly into the peer container, they will not be deleted when a peer is deleted. They will have to be deleted either using the UI of your cluster or by issuing kubectl commands. Smart contracts installed on a v1.4.x peer will be deleted when the peer is deleted.
If you are using OpenShift, you have the option to use either the kubectl CLI (which is native to Kubernetes), or the OpenShift cluster (oc) CLI. The commands should be largely the same, except that OpenShift uses "projects" instead of "namespaces". If you are running any cluster type other than OpenShift, you will have to use the kubectl CLI. In this topic, we'll use both CLIs.
If you want to delete all of your smart contract pods, you can issue this command:
kubectl get po -n <PROJECT_NAME> | grep chaincode-execution | cut -d" " -f1 | xargs -I {} kubectl delete po {} -n <PROJECT_NAME>
Where <PROJECT_NAME>
is the name of your OpenShift project.
If you want to delete a single smart contract pod, you will first have to figure out the name of your smart contract pod.
First, get a list of all of the smart contract pods running in your cluster:
kubectl get po -n <PROJECT_NAME> | grep chaincode-execution | cut -d" " -f1 | xargs -I {} kubectl get po {} -n <PROJECT_NAME> --show-labels
You should see results similar to:
NAME READY STATUS RESTARTS AGE LABELS
chaincode-execution-0a8fb504-78e2-4d50-a614-e95fb7e7c8f4 1/1 Running 0 14s chaincode-id=javacc-1.1,peer-id=org1peer1
NAME READY STATUS RESTARTS AGE LABELS
chaincode-execution-f3cc736f-94ef-454d-8da3-362a50c653d9 1/1 Running 0 4m chaincode-id=nodecc-1.1,peer-id=org1peer1
Your smart contract name and version is visible next to the chaincode-id.
To delete a single pod, issue this command, substituting the <POD_NAME>
for the name of your pod, for example the smart contract pod chaincode-execution-0a8fb504-78e2-4d50-a614-e95fb7e7c8f4
, as well as your <PROJECT_NAME>
:
oc delete pod <POD_NAME> -n <PROJECT_NAME>
If you cannot use your console or the APIs to remove your nodes, you can manually remove all of the nodes from your cluster by using the OpenShift CLI. Navigate to your OpenShift Project:
oc project <PROJECT_NAME>
Then run the following commands to delete all of your blockchain nodes:
kubectl delete ibpca --all
kubectl delete ibppeer --all
kubectl delete ibporderer --all
You may also choose to only delete all of a single type of node within a namespace, for example, by only issuing kubectl delete ibppeer --all
.
Note that if you delete your entire project, your smart contract pods will also be deleted.