Now serving: Event-Driven Ansible with a dash of durability

In the realm of automation, the ability to respond to events in real-time is a game-changer. At Red Hat, we've been pioneering in this space with Event-Driven Ansible, which can consume messages from various sources like AWS Simple Queue Service (SQS), Azure Service Bus, and Kafka to trigger automated actions. Today, we're excited to delve into a powerful integration pattern involving AWS Lambda, AWS SQS, and Event-Driven Ansible.

Imagine this: A SaaS application sends a webhook POST request. This request triggers a Lambda function, which validates an API key or other payload data, filters the payload, and sends a message to SQS. Event-Driven Ansible subscribes to the queue, consumes the message and triggers an automated action. Let's explore this workflow in detail.

The Workflow

Here's a visual representation of the workflow with AWS Lambda and AWS SQS:

Ideally, in this model, webhook POSTs should selectively  be sent to the SQS queue. Rulebooks within Event-Driven Ansible have the ability to validate that a key within the header contains the specified value – but that means the message is already on my queue. I want to stop that from happening. In this case, my Lambda function should be able to validate that the payload is recognized before sending it to SQS.

Once a message is validated and sent to SQS, SQS sends back the unique message ID to confirm receipt.

Event-Driven Ansible uses a source plugin for SQS and acts as a message consumer. Once the message is on the queue, my rulebook is configured to act on these messages – so once a message is validated and in the queue, it is ready for a rulebook to act on it. As part of this producer/consumer relationship, messages are removed from the queue as soon as Event-Driven Ansible consumes them. This operation is all managed by the SQS Python client library that is implemented in the Event-Driven Ansible SQS source plugin. It enables messages to persist on the queue until they are handled by Event-Driven Ansible.

Validating and Filtering with AWS Lambda

The Lambda function acts as a secure gateway between the SaaS application, the message queue, and Event-Driven Ansible. For example, it can validate the API key of the webhook POST sender, so that only authorized requests are processed. Additionally, it can filter the payload based on the value of certain keys, allowing for more fine-grained control over the data that gets passed to the messaging layer and, ultimately, to Event-Driven Ansible. You could also trade the Lambda function for Eventbridge, but that seemed a little heavy for my initial use case.

Here's a simplified example of how you can implement a webhook receiver using Python as a Lambda function:

This is just my first example, so please improve upon it as you think about similar implementations. If you’d like to use my example to get started, it’s on GitHub. The script is quite simple so you should be able to pull apart this recipe and identify the ingredients that enable validating the header content of the payload before forwarding the payload as a message on the SQS queue and sending a response to the webhook sender. And of course, feel free to add your comments and enhancements to this GitHub repository.  

Consuming Messages with Event-Driven Ansible

On the other end, Event-Driven Ansible is subscribed to the messaging queue on AWS, ready to consume messages and trigger automated actions based on these messages. With its ability to consume messages from various sources like Amazon SQS, Azure Service Bus, and Kafka, Event-Driven Ansible provides a powerful way to react to events in real-time.

Here’s a rulebook example along with SQS to respond to failures logged by a third-party SaaS application:

Using this integration pattern, we benefit from the ease and ubiquity of webhook events and the durability of event messages as they are posted to a messaging service supported by Event-Driven Ansible. The durability provided by the messaging layer provides that messages are received and read by Event-Driven Ansible. This means that, if for some unknown reason the rulebook activation restarts, messages that were sent while the rulebook activation was unavailable are handled appropriately as soon as it becomes available again.

Wrapping Up

By combining the power of AWS Lambda + SQS, and Event-Driven Ansible, we can create a powerful, responsive automation system. This system can react to events from SaaS applications and trigger automated actions, helping to streamline processes and improve efficiency.

On the Ansible team, we're excited about the possibilities that this integration opens up. We can't wait to see what you'll automate next!

Where to go next

• Learn more about Event-Driven Ansible.
• Get hands-on with on-demand Ansible Automation Platform self-paced exercises, including Event-Driven Ansible labs. 
• Trial subscription - Are you ready to install on-premises? Get your own trial subscription for unlimited access to all the components of Ansible Automation Platform.
• Subscribe to the Red Hat Ansible Automation Platform YouTube channel
• Follow Red Hat Ansible on Twitter - Do you have questions or an automation project you want to show off? Tweet at us!
• Join the Ansible Community Forum to discuss this blog post