Search audit events in LogDNA using Python.

Earlier this year, IBM and LogDNA announced an integrated offering—Activity Tracker with LogDNA. It allows you to manage and search activity events in LogDNA instances on IBM Cloud. There are IAM, account management, and all kinds of service instance events that can be tracked. Viewing the events is typically done in the LogDNA UI. I, however, want to perform searches on the command line and integrate it with IBM Cloud Functions. 

In this article, I discuss the small tool that I wrote to search the activity logs and export them.

LogDNA interfaces 

After provisioning the LogDNA instance as part of the IBM Cloud observability portal, you can launch its UI. There, it is possible to define views, perform searches, and manage instance settings. An important setting is the configuration of log archiving (i.e., the automatic backup of activity data to cloud object storage—compliance and more).

LogDNA allows you to ingest and export events through APIs. The required API keys can be obtained via UI in the settings for organization.

Search and export via API 

The export API requires to and from parameters to limit the time range and supports event filters (e.g., by hosts, event level, and, most importantly, by query string). Authentication is performed by API service key.

With that information, I had everything I needed to write a small Python script—logdna-search—that serves as a foundation for a more capable command line tool or package for some serverless activity analytics. The script requires a configuration file with the instance region and the service key. The number of hours for the time range and a query string are optional. The following searches for log events from the past 24 hours, where the initiator of that security-related event has a name starting with “hloeser“:

searchLogDNA.py logConfigEU.json 24 'initiator.name:hloeser'

The above returns events as JSON lines; each event is a JSON record on its own.

More fun with jq

If you are familiar with jq, a command line JSON processor, you appreciate its filtering and formatting capabilities. Because of the nature of the activity events and LogDNA logs, the JSON-based events are embedded into JSON records with additional metadata, making it harder to read. With the help of jq, it is easy to filter down:

searchLogDNA.py logConfigEU.json 24 'initiator.name:hloeser' | jq -r '._line'

To extract just the initiator part of the events, apply another filter:

searchLogDNA.py logConfigEU.json 24 'initiator.name:hloeser' | jq -r '._line' | jq -r '.initiator'

Try it out

Want to test it on your own? The Python script is available on GitHub in the repo logdna-search. Information on Activity Tracker with LogDNA on IBM Cloud is in the documentation.

If you have feedback, suggestions, or questions about this post, please reach out to me on Twitter (@data_henrik) or LinkedIn.