This article will explain how to separate multiple tenants consuming IBM public cloud managed services.

It has also taken into account that each tenant can be responsible for more than one component of a value chain.

Each IBM Cloud service will be discussed separately, including different alternatives and their pros and cons. The description on how to handle multiple tenants in each service is based on the documentation officially available from IBM, as well as experience gained from several IBM Cloud projects performed for customers. At the end of each section, a recommendation is given.

The following services are covered in this post:

1. IBM Cloud Kubernetes Service
2. IBM Cloud Databases
3. IBM Cloud Object Storage
4. Virtual Machines (Classic Infrastructure)
5. IBM Container Registry
6. IBM Key Protect
7. IBM Certificate Manager
8. Activity Tracker with LogDNA/Log Analysis with LogDNA

Infrastructure team

The more tenants that operate in the same IBM Cloud account, the more important it is to establish a centralized infrastructure team. There needs to be a single instance that has a broad overview over the implemented infrastructure and how the tenants are being isolated from each other. Adding to that, some managed services do not allow multitenancy or contain information that should not be available to all tenants.

In previous projects, this infrastructure team consisted of people, independent from any tenant or component. It needs to work closely together with the IBM Cloud account owner since the team requires extended privileges to create the necessary infrastructure.

IBM Identity and Access Management (IAM)

Each of the discussed managed services is integrated with IBM Identity and Access Management (IAM). The access to each service is divided into two categories, with each having several sub-categories. The two main categories are “platform” and “service” roles.

While “platform” roles relate to actions on the infrastructure level (e.g., creating the service, adding worker nodes, increasing CPU/memory, etc.), “service” roles relate to the privileges within the infrastructure (e.g., creating new namespaces, administrating the database, etc.). However, not all managed services can be configured on a “service” role level. Privileges can be granted per-user or connected to an access group. Users and service IDs (functional IDs, for example, to be used by automation tools like Jenkins) can then be assigned to such access groups.

Exceptions to this concept are services based on IBM Cloud classic infrastructure (formerly Softlayer) like virtual machines, bare metal devices, and gateway appliances. The network management is also done via Classic Infrastructure. There is a separate section for granting those privileges in a lot more detail compared to the managed services.

For more information see the IBM IAM overview.

1. IBM Cloud Kubernetes Service

Due to the integration with IAM, the different access privileges do not need to be set via Kubernetes-integrated RBAC (cluster-)roles and (cluster-)role bindings—instead, they can be defined on an account-management level. It is still possible to set additional roles and privileges via the Kubernetes RBAC system, which can be required for service accounts included in certain deployments. Those additional service accounts and roles are not tracked in the IBM Cloud UI. Therefore, it needs to be considered whether or not the additional accounts and roles are mandatory. If the Kubernetes RBAC system is used on a more granular level than given by IBM’s IAM, the privileges have to be tracked in both systems.

In general, the permissions can be divided into three clusterroles: “admin,” “edit,” and “view.” Those roles can be either applied cluster-wide (clusterrolebindings) or on a namespace-level (rolebindings). Both bindings are divided into three categories: “ibm-operate,” “ibm-edit,” and “ibm-view.” A description of these roles and their according rolebindings can be found at IBM Cloud IAM service roles.

Example outputs

Before explaining the different isolation options, it is necessary to understand the “manager” and “writer” IAM roles for the IBM Cloud Kubernetes Service:

• Manager: This applies the “admin” clusterrole if no namespace is defined (therefore, indicates all namespaces). If a namespace is defined, the “admin” role is granted for the specified namespace. Details on the actions a manager can perform are listed at “Kubernetes resource permissions per RBAC role.”
• Writer: Depending on if a namespace was defined, this role grants the “edit” cluster role to either the defined namespace or the whole cluster. Details on the actions a writer can perform are listed at “Kubernetes resource permissions per RBAC role.”

There are several options for handling multitenancy within IBM Cloud Kubernetes Service.

Create one IBM Cloud Kubernetes Service cluster per tenant

While this is the most isolating option, it is also the most expensive and inflexible one. Inflexible mostly because worker nodes and according resources can only be ordered in certain sizes and will be reserved, even though they might not be used.

Each tenant gets its own Kubernetes cluster. The tenants identify at least one (although, better if more) cluster administrator(s) while the rest of the team has “writer” privileges on the cluster. If there are several teams within the tenant that are responsible for different components, it is also possible to define the privileges more granularly. This, however, also significantly increases the management overhead for the infrastructure team who handles those privileges, especially, if there are a lot of tenants. Another difficulty appears if one or more components are moving out of the tenant’s responsibility and are, for example, taken over by another tenant. Then, these components would have to be extracted from the cluster and integrated into another cluster.

Create one IBM Cloud Kubernetes Service cluster per component

With this option, each component gets its own Kubernetes cluster, granting a lot of flexibility for the component itself. In case the component’s responsible entity changes, this can be performed on the IBM Cloud account level (IAM) without requiring the cluster to be redeployed. This option also provides a high level of isolation. However, it can also become very expensive, depending on how many components are included in the value chain. Furthermore, if some components do not require much resources, a dedicated cluster will be overdimensioned. In such a scenario, one tenant could be responsible for more than one cluster if the tenant administrates several components. This creates additional, unnecessary infrastructure management overhead.

Create one IBM Cloud Kubernetes Service cluster with tenant separation on the namespace level

Since the IBM Cloud IAM allows you to restrict access on the namespace level, it is possible to have several tenants working in one cluster while still being separated from each other. This allows very efficient usage of resources. Each tenant has its own namespace within the cluster. The tenant then identifies a group of people who get “Manager” privileges on that namespace to administrate it. The rest are granted “Writer” privileges to work within the namespace.

There is one pitfall with this concept, however, namely that all components belonging to one tenant run in the same namespace—this opens the gates for several issues. If there is a problem with one component affecting the namespace (e.g., a malfunctioning Ingress definition), all other components will be affected as well. Furthermore, if one component is moving out of the responsibility of one tenant and transferred to another, the component would have to be redeployed from scratch into another namespace.

Create one IBM Cloud Kubernetes Service cluster with component separation on the namespace level

This scenario is similar to the previous one (“Create one IBM Cloud Kubernetes Service cluster with tenant separation on the namespace level”) with the difference being that in this case, each component is deployed to its own namespace. This still gives the advantage of optimizing the resource consumption as well as isolating the tenants from each other. It even gives more granularity for the assignment of access privileges. Each tenant can be responsible for more than one namespace, which allows to assign different team members to their associated component. The tenant should, however, identify a small group of people who get “Manager” privileges on all their component namespaces. The other employees can then be divided according to their associated component and get “Writer” privileges.

All responsibilities can be handled on the IBM Cloud account level without requiring any changes on the infrastructure level. This means that if the component moves out of the responsibility of one tenant to another, the only changes necessary are on the IAM level. The privileges of the previous tenant are removed, while the other tenant is granted the required privileges.

Recommendation

Previous experience shows that one IBM Cloud Kubernetes Service cluster with component separation on the namespace level provides the most flexibility while optimizing the costs and hardware utilization. It also separates the cluster into failure domains, based on namespaces.

2. IBM Container Registry

IBM offers its own container registry (CR) for uploading, building, and storing container images. The IBM CR also comes with a vulnerability advisor, which checks the container images for deprecated versions and malware. The CR is split into namespaces where the images are placed. The namespace names have to be unique.

Images are usually created separately by each tenant, while each component can consist of several container images. It is common to have tenants manage their own CR namespace(s) so that efficient workflows can be guaranteed. Isolation, therefore, takes place on the CR namespace level, with each component having its own namespace, analogous to the IBM Cloud Kubernetes Service setup. This is made possible by applying the according IAM privileges.

While Kubernetes clusters are often separated into stages, the CR should contain one component namespace for all stages. Versioning within the CR allows you to reflect the different stages of a component’s lifecycle so that certain version can be deployed to the according Kubernetes clusters.

The infrastructure team creates the necessary CR namespaces and assigns “Viewer” platform access as well as “Reader” and “Writer” service access. The different privileges associated with the roles are described at managing access for Container Registry.

3. IBM Cloud Databases

IBM Cloud Databases (ICD) includes several databases which can be ordered separately. Since they are all integrated with IBM’s IAM, this section’s content can be applied to all of the databases. The ICD family includes the following:

• Databases for PostgreSQL
• Databases for Redis
• Databases for Elasticsearch
• Databases for etcd
• Databases for MongoDB
• Messages for RabbitMQ

While all database services can be handled separately in the IAM, it is not possible to assign “service” roles; only “platform” access can be granted, which allows management on the infrastructure level. The “platform” access is divided into four categories: “Viewer,” “Editor,” “Operator,” and “Administrator.” Every role besides the Viewer role should only be granted to the infrastructure management team. The according privileges are listed at Identity and Access Management Integration.

Each ICD instance comes with one database administrator user. This user has all privileges on the database level and can, for example, create additional users, databases, tables, and so on. They also handle the database-internal access rights and privileges. Since there is only one database administrator (at least in the beginning), it does not make sense to have multiple tenants operate their contents in the same ICD instance. However, in a loosely coupled scenario, where one component is connected with another through an ICD instance, it is common to grant readable access for other services and tenants to the ICD’s content. This leaves only two options.

Create one ICD instance per tenant

Each tenant has one instance with administrative access. They are then self-responsible for creating databases, users, privileges, and database configuration. The tenant is also responsible for granting external access to the database instance (e.g., other services or tenants). The internal structure—for example, one database per component—is up to each tenant. It should be noted that issues concerning one component can also affect the other components running in the same ICD instance.

Create one ICD instance per component

Each tenant gets access to several ICD instances—one for each component. The responsibilities are equal to the previous option (create one ICD instance per tenant). In this scenario, however, issues concerning one component will most likely not affect any other components.

On the one hand, this option gives more granularity; on the other hand, however, it also requires more management effort than the previous option. Adding to that, ICD backups and restores are only possible on the instance level. If a tenant has databases belonging to different components running in the same instance, it would not be possible to only backup or recover one dedicated database. Those restrictions need to be considered based on each tenant and component.

Recommendation

Both are valid options and each project needs to decide separately on which option is preferred. In production scenarios, it makes sense to accept the additional management effort of the second option (create one ICD instance per component), especially due to the limitations of backup and restore in the first option. Additionally, the second option increases availability and granularity, which further shrinks the failure domains.

Each ICD family member has its own database roles, and more information can be found here:

• PostgreSQL user management
• Redis: Only one user per instance possible (admin)
• Elasticsearch connection strings: New users need to be created by infrastructure team
• Etcd connection strings: New users need to be created by infrastructure team
• MongoDB user management
• RabbitMQ user management

4. IBM Cloud Object Storage

Access to IBM’s Cloud Object Storage (COS) can only be managed through IBM’s IAM. Each COS instance consists of one or more buckets where data can be stored, and they are isolated from each other. The access can be restricted from the instance- down to the bucket-level. The buckets are individually configured so that they can have their own encryption key and separately appear in the IBM Activity Tracker (if enabled).

It is possible to create service credentials directly within the COS instance. By default, the service credential’s access rights are applied to the complete instance and not separately on a single bucket. It is also possible to restrict the service credential’s access to one or more specified buckets afterwards, through IBM’s IAM.

One instance for all tenants

While it would be possible to isolate the tenants from each other, it would be a huge amount of administrative effort to set up such a scenario. Each tenant would require one or more buckets, depending on the desired architecture. Correlated to that is the required amount of service credentials that would need to be created and, accordingly, restricted afterward since, by default, they allow access to all buckets within the instance.

One instance per tenant

In such a scenario, all tenants are isolated on the instance level, which is the highest possible isolation level. Each tenant can use a desired number of buckets to differentiate between data and components. Each tenant then has one service credential with which they can access all buckets.

One instance per component

Overall, with this option, the number of instances will most likely be higher compared to the second scenario (one instance per tenant). The advantage of this option is that it allows you to manage the different components on a more granular level.

If, for example, a component is moving out of the project or into another tenant’s responsibility, this transition can easily be handled. Whereas, if the isolation takes place with one instance per tenant, such transitions become more complicated (e.g., moving one component out of the responsibility but keeping the others). The management effort on infrastructure side for this option is a bit higher since privileges need to be set for each COS instance.

Recommendation

The second and third option are pretty similar, and both are valid options. The third option (one instance per component) further shrinks the failure domains without increasing the costs. It also aligns with the recommended concept for IBM Cloud Kubernetes Service, IBM Container Registry, and ICD. The decision needs to be made depending on the individual project.

General remarks

Previous experience shows that it is easier and more efficient for the infrastructure team (and the tenants) to give the tenants privileges to create and manage buckets on their own. On the one hand, this puts more responsibility on the tenants; on the other hand, it accelerates workflows since the tenants do not need to contact the infrastructure team for creating and editing buckets. To realize such a scenario, there need to be some common agreements on responsibilities, naming conventions, encryption keys, housekeeping rules, etc.

The associated privileges are “Viewer” for “Platform access” and “Reader” together with “Writer” for “Service access.” The according service credentials would then be generated by the infrastructure team and passed to the tenant so that the tenant can operate the COS instance on their own.

5. Virtual machines/bare metal (classic infrastructure)

As initially described, the access to IBM Cloud classic infrastructure resources can only be assigned on a user-basis. The permissions can be set within four categories: Account, devices, network, and services, with each category having server sub-permissions. An administrator can choose to give single sub-permissions or use one of three predefined “permission sets” provided by IBM:

• View only: This grants very restricted access to only view bare metal servers and virtual machines. The user can view the credentials to access the systems.
• Basic user: In addition to “view only” privileges, the user can edit other users’ privileges, manage firewalls and load balancers, and add IP addresses.
• Super user: This user has every possible sub-permission for the four categories. In addition to a “basic user,” the super user can add new servers and manage gateway devices and the network down to the VLAN-level.

Recommendation

If virtual machines (VMs) are required, there needs to be a common agreement on the connected responsibilities. It is not recommended to give everyone the privileges to create new virtual machines or delete them since this is directly correlated to creating costs or triggering production outages. If possible, the virtual machines should also be set up by the infrastructure team, while the tenants identify the group of people who need access to those machines. Also, the tenants need to define, on a granular basis, which privileges are necessary.

Furthermore, each project should decide individually if VMs based on IBM classic infrastructure are necessary or if the project can make use of IBM Cloud VPC. VPC brings extended capabilities like Bring-Your-Own-IP (BYOIP) and load balancers as a service, while being significantly faster than the IBM Cloud classic infrastructure.

Unified managed services

There are some managed services that provide shared functionality. They contain information and resources that are used by other service instances across the IBM Cloud account. This can even involve confidential data that should not be available to any tenant. Therefore, those services need to be managed by the centralized infrastructure team. The following services fit into that category. Usually there is only one instance of each service within the same account.

6. IBM Key Protect

The IBM Key Protect (KP) service is responsible for storing encryption keys and providing them to the different KP-enabled services (e.g., IBM Cloud Kubernetes Service, COS, ICD). Those keys can either be created through the KP instance or can be securely imported by the customer (Bring-Your-Own-Key).

Since the KP instance contains keys used by all services and by each tenant, the keys should be handled carefully. It is not recommended to give tenants access to the account’s Key Protect instance. Instead, the key management should be handled by the infrastructure team. Each tenant requests their required keys from the infrastructure team so that they can provide the keys through the KP instance.

7. IBM Cloud Certificate Manager

Similar to the Key Protect service, IBM Cloud Certificate Manager is the central source for all certificates to be used, for example, by different Ingresses of the IBM Cloud Kubernetes Service. The Certificate Manager should be managed by the infrastructure team, which creates new certificate signing requests, imports certificates, and distributes them to the different managed service instances.

8. IBM Cloud Activity Tracker/IBM Log Analysis with LogDNA

Since both offerings are based on LogDNA, they can be treated equally when it comes to multitenancy. The offerings are integrated with IBM’s IAM so that access can be given based on the platform and service level.

The service access allows you to differentiate between read-only users, standard users (who can modify some data) and administrators. However, what this access does not allow is any differentiation based on the observed managed services and activities. As a result, even a read-only user can view any data or logs collected by the Activity Tracker and Log Analysis services. Those data and logs also contain account-related activities like creating new users, loading user tokens, or creating new root keys. Data which should not be available to everyone.

Therefore, it is currently recommended to restrict the access to both services to the infrastructure team. This team is then able to provide requested information to the tenants or export defined datasets to external solutions that are accessible for the tenants.

Resource Groups

IBM public cloud allows you to isolate and separate tenants through resource groups. Each account comes with a default resource group and additional resource groups can be created afterward. A resource group can contain IBM public cloud service instances but does not contain any users (tenants). Instead, users are assigned to specific resource groups and, optionally, service instances within that resource group. This allows you to isolate the tenants on the infrastructure level, even if they are assigned administrator privileges within the resource group, so that they can create new service instances.

Conclusion

Often, projects need to decide individually which kind of separation and isolation best fits the security and operations requirements. As a general recommendation, projects should make use of the capabilities enabled through virtualization to split resources as much as possible, without generating additional costs. This gives them the possibility to size failure domains just right to meet the functional and non-functional requirements.

In any case, during the planning phase, every project should consider creating a concept on how to handle multiple tenants. IBM public cloud offers a lot of capabilities for managing multiple tenants within one cloud account

The concept of resource groups is usually applied if tenants create and manage the required infrastructure (IBM Cloud Kubernetes Service clusters, IBM Cloud Databases instances, etc.) on their own. In the case of a centralized infrastructure and IAM management team, additional resource groups are not specifically necessary since tenants are usually not given any administrative privileges within the default resource group, so that they could create new service instances.

Additional information to resource groups and their range of application can be found in the best practices for organizing resources in a resource group article.