Measuring the success and impact of an automation community of practice

An automation community of practice can be a cornerstone of your organizational strategy to adopt automation at scale. Yet there’s a challenge: how do we know if the community is healthy and driving you towards your strategic goals around automation?

The value of a community of practice

As we’ve established over this blog series, the values of a community of practice are transformative. From upleveling the skills, sharing knowledge and practices, or just establishing a network of other professionals across the organization for support and troubleshooting, the value of community can be a differentiator, something that sets your organization apart. But as we develop our community of practice, we need to make sure that it is actually providing these benefits, so we need to have a way to determine if our community is successful.

Revisit the formation

When you launch a community of practice, you will  define a purpose. This may be a mission statement, a targeted outcome or goals from a start-at-the-end exercise (hopefully your purpose is related to your automation strategy). 

There are two things to do with this. First, simply identifying that the purpose is still valid—revisiting this stated purpose on an ongoing basis is important, as priorities may change. Second, you need to identify what is measurable that aligns with your purpose.

Stepping back, what could your purpose look like?

• Education and upleveling our associates with automation skills
• Defining processes for better automation deployments
• Driving collaboration across multiple departments to create more automation content

Using our own internal Red Hat automation community of practice as an example, our purpose is  twofold:

1. To provide a forum for educating and upleveling our associates to present automation topics and projects
2. To define a community to manage and curate resources for the larger Ansible community, such as the Automation Good Practices library and the Ansible Content Collections for configuration as code

Identify what’s measurable

Taking a purpose and turning it into a measurable goal is a challenge. But a measurable goal is an important part of being able to test the impact of our management of the community. 

Building off of your purpose, you can begin identifying the metrics you will use to judge your community’s health. Again, using our own internal community of practice as an example, there are a number of metrics we’ve investigated. These include:

• The number of attendees at our bi-weekly meetings
• The number of pull requests made to our code base
• The number of unique contributors to our code base
• The number of resource documents created for internal knowledge sharing
• The number of viewers on our resource website

As the purpose of your community of practice may be different, identifying an easily measurable and trackable metric is vital to understanding the evolution of your community's impact over time.

State of the union

As important as these measurable statistics are, remember that this is still a community of people with opinions and feelings. Surveying your community is another valuable tool to gain insights and understand the sentiment of your community members. We did this with our own community of practice, and here are some questions from our survey that you can use as inspiration:

• Why do you attend the community of practice meetings?
• How would you rate your involvement? Do you want to be more involved? What prevents you from being more involved?
• What do you want to see more or less of?
• What would increase your happiness and satisfaction with the community of practice?

These insights are valuable for us in many ways. Identifying what most attendees are seeking to gain from the meetings has inspired us to inject some new content into our community through efforts like bringing in outside speakers to discuss topics we’re interested in or developing contribution guides to help people feel like they can be more involved and active. Through these surveys, we have driven more measurable success criteria, as our members are successfully contributing more, expanding their network and bringing in new members from their circles.

Share in the success and struggle

One final thought on measuring success comes from a pair of customer communities I’ve worked with.

Take customer A. With this customer, upper management is present driving a level of interaction with the community that is making members want to attend. To see and be seen. In their current remote-heavy workplace environment, this was a primary place to show off their successes in front of leadership. Attending these calls was always a high priority, and the community achieved its purpose, which was to share the efforts of different teams within the organization to foster collaborative efforts and reduce rework. The schedule of presenters was constantly filled and the presentations were engaging. Communication flowed as members would set up follow-on conversations to get more details and learn from each other’s experiences.

Now let us look at customer B. This customer had many of the same factors: upper management was present and they had a similar remote-heavy workplace. Attendance was seen as mandatory for visibility. But unlike customer A, the purpose of this community was established to be a place to get technical help and support from each other and to discuss challenges with implementing automation. On these calls, it was always silent. No one wanted to engage in technical discussions or to ask for assistance because of who else was in the room. Politics dominated this customer and, as such, the community was ultimately unsuccessful and disbanded after a few months.

The purpose of your community of practice matters. Healthy communities thrive on the openness of the community members to share successes and struggles. I’ve seen several communities established in our customer sites and while there is no right or wrong answer to what your community purpose is—the successes or the struggles of a community come down to the community members participating and sharing experiences. Measuring and shaping the success and impact of a community of practice all comes back to identifying the purpose of the community.

Additional resources

• Get expert guidance kicking off your automation community of practice with Red Hat Consulting
• Hear from our top customers about how they approach their automation strategy
• Enable automation learning, productivity and collaboration with Ansible plug-ins for Red Hat Developer Hub