As applications and infrastructures grow more complex and dynamic, how could DevOps teams automate and streamline critical log collection, monitoring and analysis in cloud, on-premises and hybrid environments?

Synergy helped weave the LogDNA logging platform into the IBM global ecosystem, binding the two companies together as both partners and consults-clients to deliver innovative, managed Kubernetes services.

In 2013, while working on their third startup company, Christopher Ngyuen and Lee Liu had an epiphany: no log management tool was fully capable of handling the intense needs of a modern, cloud-native development stack. Traditional log analysis methodology simply wasn’t responsive or scalable enough to manage the level of data sprawl that happens in dynamic cloud environments.

If it frustrated them, they knew other DevOps teams were chasing data activity, too. So, they pivoted to start crafting a logging platform they could use, building on top of the popular Elasticsearch tool.

Logging has always provided critical information for developers on the lookout for code errors. When it involved a single server, logging was relatively straightforward, if often slow. But when virtualization moved into the data center, the process got significantly messier.

Centralized logging evolved to corral log files, but it was still up to developers, IT or infrastructure teams to organize the logging for themselves — a time-consuming, expensive use of resources. Worse, the growing clamor for zero downtime increased the speed at which developers needed real-time visibility into issues and changes. Today, containers that make up an application are proliferating as a new foundation for modernization.

“Anytime your application grows, you have more logs to deal with. Anytime you have new applications that come into your environment, or you’re launching new products, you have more logs,” says Norman Hsieh, VP of Business Development at LogDNA. “And effectively, somebody needs to spend more time managing and scaling that.”

After putting their integrated logging version to the test on Hacker News, Hsieh says, “We immediately saw a genuine demand and need for new perspective on logging.”

He continues: “At some point, you literally don’t have to touch any boxes, but you could have thousands of machines out there running your applications. Now multiply that by 10 or 100 in terms of the number of containers you’re running. You’re looking at a huge sprawl of data in these architectural frameworks, and it’s all automated but you don’t really know what’s going on. The only way to get that information is to save a log. And what you need now is a tool modernized enough to provide DevOps intelligence, know the frameworks and organize all that data automatically for you.”

And then the co-founders saw the writing on the wall: Kubernetes. “We thought it was an awesome technology to back our own architecture,” Hsieh adds. A groundswell of developers was fast adopting the lightweight, open-source platform for managing containerized workloads.

The LogDNA team seized on the advanced orchestration capabilities of Kubernetes in cloud environments, and looked for a leader in this space.

The work LogDNA was doing didn’t go unnoticed by IBM Cloud and Cognitive Software Chief Technology Officer (CTO) Jason McGee and his team, which itself was shifting focus to cloud, Kubernetes services and DevOps. Both companies were also working in the open source community, applying the momentum in the developer space.

“I give a lot of credit to IBM giving us a shot and saying, ‘These guys are doing something interesting with the technology they have today,’” says Hsieh. “We saw the same synergy between what Jason’s team was doing and where we were headed. When they talked about leading with Kubernetes on the IBM Cloud side, we knew this is exactly the space we wanted to push in.”

Like DNA strands winding together, LogDNA and IBM formed a tight relationship that transformed into both an integrated partnership and, happily, a client opportunity for LogDNA.

“We have had the chance to work extremely closely with IBM and not just from the business development side. We’re also working hand-in-hand with the internal teams that have built services within IBM,” says Hsieh. “We’re often asked by people in the Bay Area community how we get to do that. I think a lot of it is our willingness to say, ‘Hey, we are going to grow and expand with IBM.’”

One of the opportunities that added value to LogDNA and demonstrates the partnership was a new bare metal offering from IBM.

“You know our business is logging, but when you dig deep, we are also a storage and a big data company,” Hsieh explains. “So, when we initially looked at the IBM Cloud Kubernetes Service, we worried that current offerings were not going to meet our demands. We met with some of IBM’s distinguished engineers and they said, ‘Why don’t you take a look at the bare metal offering and see if there is something more flexible to get you to the requirements you need.’ We didn’t even know that existed.”

IBM had fast-tracked an offering to launch immediately, but this surprise was a game changer for LogDNA.

“We had taken our cloud-native architecture and provided a multichannel SaaS product using Kubernetes,” says Hsieh. “It enabled us to offer a product that can be deployed in all different types of managed Kubernetes environments but the bare metal option allowed us to get the IOPS [input/output operations per second] we needed to read and write quickly out of storage, and at a less-expensive price than network-based storage.”

With recommendations for some specification changes, LogDNA pressed IBM to get it done faster. It wasn’t the only time IBM leaned in.

“They have been extremely helpful. Early on, we were both learning how fast we were growing and so we weren’t even sure how many servers we needed. But the process of getting new servers and getting the new IBM Cloud Kubernetes Service up has just been getting better and better.”

Imagine you’re a disruptive ride-sharing application company, with millions of customers using your app around the clock—and driving a prodigious amount of log files in a massive infrastructure. Every car request, every ride taken, and the level of customer engagement leapfrogs the logging demands of yesterday.

“We think there is a higher market for that level of scalability versus what the incumbents are offering today. Thanks to our partnership with IBM, we can address that now,” says Hsieh.

“What is beautiful about LogDNA is that clients can start when their journey starts, and then stay with it in a consistent form regardless of where apps are running, or whether they are in early stages, modernization, or native application area,” says Sue Hahn, IBM Cloud Western Region Business Partner and Channel Sales and IBM Global Markets – Cloud Sales. “It can log and provide visibility across that whole spectrum—public, on-premises, hybrid, any cloud, seamlessly across various environments.”

The goal is to provide a log management tool that optimizes a developer’s data.

Hsieh says: “That means we focus heavily on things like automatic parsing. Any data that comes into LogDNA, we automatically take care of it for you, since we can recognize exactly what kind of logs they are. We bundle our services for simplicity and ease of use, so you don’t have to worry about logs.”

LogDNA attributes the ability to operate consistently around the world, no matter into which data center they push, to the global footprint that IBM offers. As IBM’s preferred logging service, the company’s product will be available in the IBM Cloud Service Catalog in all IBM service regions.

“We’ve expanded into Dallas, Frankfurt, with EU-Managed operations, Tokyo and London,” Hsieh says. “We’ll have Sydney soon. None of this would be possible without the consistency IBM brings—the exact, same deployment.”

Hahn adds: “That’s huge, to be able to support customers in a very short period of time because of the consistency of the offering. Wherever LogDNA deploys, it’s the same environment so we don’t have to change any operational processes. Customers can select and order our services, identify they want logging, and that pulls LogDNA directly into the order.”

In addition, IBM is using LogDNA as a log analysis tool for its internal systems.

The bulk of the partnership centers around LogDNA’s use of the IBM Cloud Kubernetes Service, but the company also engages with IBM Cloud Object Storage for testing, because the product offers integration directly into Cloud Object Storage. In the future, Hsieh says, the expectation is to apply other IBM services, such as IBM Watson® technology, to augment its machine learning capabilities, or a multicloud manager.

“Right now, our product is heavily focused on the DevOps space. Our strong suit is in providing better insights, better observability into development stacks. We’re building tools that enable developers to, essentially, not think about logs at all. We’re providing the convenience of a very robust log management tool without the inconvenience of having to manage or configure anything,” Hsieh says. “There are hidden costs to building out logging solutions, in resources and time. At the end of the day, we are letting our customers focus on the things they need to focus on.”

Based in Mountain View, California, LogDNA (link resides outside of ibm.com) is a software developer and service provider that offers log management and other solutions to companies such as IBM, Lime and Instacart. The firm was established in 2015 and employs approximately 70 people.