Changing the service type during a helm upgrade
You may encounter a potential issue during a helm release upgrade or rollback that
involves a change of service type from NodePort
to ClusterIP
. This
known error will prevent you from performing the upgrade or rollback.
Known Issue when performing a helm upgrade
When performing a helm release upgrade, the error message below is shown after a failed helm upgrade due to a service conversion failure:
Error: UPGRADE FAILED: Service "tutorial-ibm-netcool-probe-syslogd" is invalid:
spec.ports[0].nodePort: Forbidden: may not be used when `type` is 'ClusterIP'
The error is due to the additional parameter nodePort being included in the
service type NodePort
template, the Helm release upgrade cannot be completed.
Service type ClusterIP
"ports": [
{
"name": "http",
"port": 8888,
"protocol": "TCP",
"targetPort": "http"
}
Service type NodePort
"ports": [
{
"name": "http",
"nodePort": 30240,
"port": 8888,
"protocol": "TCP",
"targetPort": "http"
}
ClusterIP
to NodePort
. Attempts to roll back to
the previous revision with a service type ClusterIP
will cause the status of
rollback process to be placed in pending.Known error message during a failed rollback
When performing a helm release rollback, the following error message is shown after a failed helm rollback due to service conversion failure:
Error: Service "tutorial-ibm-netcool-probe-syslogd" is invalid: spec.ports[0].nodePort:
Forbidden: may not be used when `type` is 'ClusterIP'
Workarounds To The Issue
There are three methods you can use to avoid the service conversion issue above. You will only need to perform one of these methods:
- Method 1: Installing the new version of the helm chart with a different release name and update all clients to point to the new probe service endpoint if required. Then delete the old release. This is the recommended method but requires a re-configuration on the client side.
- Method 2: Manually changing the service type using
kubectl edit svc
. This method requires more manual steps but preserves the current service name and previous revisions of the helm chart. After performing this workaround, users should be able to perform a helm upgrade. - Method 3: Deleting and purging the existing helm release, and then install the new version of helm chart with the same release name.
Method 1: Install a new helm release with a different name
Follow the Helm chart README section to perform a new release install and then delete the old release. Remember to re-configure existing clients to update the probe endpoint.
Method 2: Edit the existing Service of the release
The following workaround requires more manual steps.
Manually edit the service using kubectl
This workaround is intended as an alternative for users who must not delete the existing release due to valid reasons, for example if the Cluster Admin wants to keep track of the changes by keeping records of past revision of an existing release.
A user with Cluster Admin
or Admin
role is required to change a
service type. User Admin
will have to edit the service of the probe chart by using
kubectl edit svc
. Before performing the workaround, user have to identify the
release name of the probe chart. It is recommended that user only perform the workaround during
scheduled maintenance. As an example, the following steps explain how to apply the changes to the
IBM Netcool Probe
chart that includes Logstash and Prometheus probes.
- To determine the release name of the helm chart, run
helm ls
and append with--tls
for TLS enabled clusters. - Based on the release name, identify the service that is servicing the probe by running
kubectl get svc
to get the name of the service. - Remove line 25 with the nodePort parameter and replace the service type on
line 34 with
ClusterIP
by usingkubectl edit svc "service name"
. Append the service name of a particular release from Step 2 into the command.Precautionary step: Note that, you will not be able to perform a rollback as the workaround is not considered as a helm upgrade. Before editing the service template, you should save the existing service template as a backup by running the following command:
kubectl get svc "servicename" -o yaml > service_backup.yaml
.1 # Please edit the object below. Lines beginning with a '#' will be ignored, 2 # and an empty file will abort the edit. If an error occurs while saving this file will be 3 # reopened with the relevant failures. 4 # 5 apiVersion: v1 6 kind: Service 7 metadata: 8 creationTimestamp: 2018-07-31T04:20:00Z 9 labels: 10 app: ibm-netcool-probe 11 chart: ibm-netcool-probe 12 component: logstashprobe 13 heritage: Tiller 14 release: onetutorial 15 name: onetutorial-ibm-netcool-probe-logstashprobe 16 namespace: default 17 resourceVersion: "4273993" 18 selfLink: /api/v1/namespaces/default/services/onetutorial-ibm-netcool-probe-logstashprobe 19 uid: fdc46a83-9478-11e8-983d-005056a0a011 20 spec: 21 clusterIP: 10.0.0.230 22 externalTrafficPolicy: Cluster 23 ports: 24 - name: webhook-http 25 nodePort: 32025 26 port: 80 27 protocol: TCP 28 targetPort: 80 29 selector: 30 app: ibm-netcool-probe 31 component: logstashprobe 32 release: onetutorial 33 sessionAffinity: None 34 type: NodePort 35 status: 36 loadBalancer: {}
- Save and exit to apply the changes to the service template. The new service template for
onetutorial-ibm-netcool-probe-logstashprobe
will look like this.1 # Please edit the object below. Lines beginning with a '#' will be ignored, 2 # and an empty file will abort the edit. If an error occurs while saving this file will be 3 # reopened with the relevant failures. 4 # 5 apiVersion: v1 6 kind: Service 7 metadata: 8 creationTimestamp: 2018-07-31T04:20:00Z 9 labels: 10 app: ibm-netcool-probe 11 chart: ibm-netcool-probe 12 component: logstashprobe 13 heritage: Tiller 14 release: onetutorial 15 name: onetutorial-ibm-netcool-probe-logstashprobe 16 namespace: default 17 resourceVersion: "4274540" 18 selfLink: /api/v1/namespaces/default/services/onetutorial-ibm-netcool-probe-logstashprobe 19 uid: fdc46a83-9478-11e8-983d-005056a0a011 20 spec: 21 clusterIP: 10.0.0.230 22 ports: 23 - name: webhook-http 24 port: 80 25 protocol: TCP 26 targetPort: 80 27 selector: 28 app: ibm-netcool-probe 29 component: logstashprobe 30 release: onetutorial 31 sessionAffinity: None 32 type: ClusterIP 33 status: 34 loadBalancer: {}
- When the changes has been applied the following response message is shown:
service "onetutorial-ibm-netcool-probe-logstashprobe" edited
- Repeat the same steps on
service onetutorial-ibm-netcool-probe-prometheusprobe
. - When the changes have been applied, the following response message is shown:
service "onetutorial-ibm-netcool-probe-prometheusprobe" edited
- Verify the status of the service by using
kubectl get svcand
verify that the service is able to reach the probe by sending curl test event to the probe.
Method 3: Replace the Helm existing release
The following workaround requires less manual steps.
Delete the existing Helm release and install the new release with the same name
Caution: By deleting the existing release, user will lose the configuration from the revision history of the release.
Note: To determine the release name of the helm chart, run helm ls
and append
--tls
for TLS enabled clusters.
- Click on Workloads > Helm Releases.
- Under Action, select Delete.
To delete the existing helm release using command line, use following command:
helm delete "release name" --purge
- Install the new release with the same release name.