How to run a serverless batch job on your private VM in a VPC with IBM Cloud Code Engine.
IBM Cloud Code Engine has a cool batch feature that allows you to run batch workloads in a serverless fashion. You just have to pay per use, with no cost for the infrastructure.
Still, customers asked whether it would be possible to have the ease of use of Code Engine combined with isolated VMs in the IBM Cloud as execution targets for batch workloads.
What is IBM Cloud Code Engine?
IBM Cloud Code Engine is a fully managed, serverless platform that runs your containerized workloads, including web apps, microservices, event-driven functions and batch jobs with run-to-completion characteristics. The Code Engine experience is designed so that you can focus on writing code and not on the infrastructure that is needed to host it.
What is VPC?
IBM Cloud Virtual Private Cloud (VPC) is a virtual network that is tied to your customer account in IBM Cloud. VPCs offer a “best-of-both-worlds” approach to cloud computing. IBM Cloud VPC gives you the security of a private cloud with the availability, cost-effectiveness and scalability of the public cloud.
Batch workloads on VMs
We published a new variant of running batch workloads in Code Engine. It allows the execution to take place on VSIs (VMs) located in the IBM Cloud Virtual Private Cloud Service (VPC) driven by the Code Engine serverless model.
This allows you to do the following:
- Process a large amount of data (e.g., videos, machine learning models, backup files, pay-checks, etc.) that is secured in your private network or on IBM Cloud Object Storage
- Have complete flexibility over the machine types (e.g., x86, System/Z, GPU)
- Attach additional storage (e.g., to uncompress .zip files before processing)
- Run your compute tasks over several hours or even days
- Run from a single to many thousands of VMs in parallel
- Connect to any other service running in your VPC or located in your data center
On top of all these benefits, you can have an easy and developer-friendly programming model that abstracts away any details of infrastructure elements and reduce the total cost of ownership for your infrastructure resources by only paying for exactly what you use, without idling resources.
Curious to get hands-on and try it out?
You can run batch workload in your VPC using a VSI proxy — please find the sample in this repository. The readme file has all the details to get started.
Contact IBM Cloud Code Engine Support if you want to learn more or have any feedback.
Was this article helpful?
YesNo