By Jason Dickerson, Adam Miller, and Josh Swanson
As Red Hat pushes further out to the edge, more flexibility is required when building and running automation topologies to facilitate the automation of technology outside the standard hybrid cloud environment.
Recently, we've begun pushing an architecture pattern that involves managing discrete deployments of Ansible Automation Platform that are highly distributed and typically deployed within individual sites. While this approach brings freedom and flexibility to the automation topology, it also introduces complexity, especially when considering how to traverse network boundaries.
This post is a deep dive on Controller of Controllers as an architecture. It explores how you can use Red Hat Interconnect to address complexity and facilitate deployment into edge sites.
Compelling Factors
A few key points drive the requirement to deploy Ansible Automation Platform in a Controller of Controllers configuration:
- Remote sites will not allow inbound connections but will allow outbound connections.
- Automation autonomy should be maintained even if the external connection to the site is lost.
- Blast radiuses are limited to individual instances of Controllers at the sites instead of taking down the organization's entire automation capability.
- Supports true multi-tenancy if deployed across multiple disjointed environments, such as a managed service provider managing numerous customers' environments.
When added up, Controller of Controllers meets these requirements while remaining a highly automated solution. Really, this becomes a bit of automation inception, with Ansible Automation Platform automating against Ansible Automation Platform.
Controller of Controllers
Controller of Controllers is an architectural approach to automation that uses a parent instance of Ansible Controller to instantiate, configure, and manage child instances of Ansible Controller. This is almost always accomplished via the consumption of Ansible Controller's API, which provides the ability to fully configure Controller and run automation.
The parent controller doesn't do direct automation against endpoints; instead, it configures the child controllers to do that automation. Since this is the Ansible ecosystem, typically, the configuration is defined as Ansible variables, then fed into collections such as ansible.controller and infra.controller_configuration.
Once the child controllers are configured, they begin the automation against the endpoints. This reflects the "traditional" automation model used in hybrid cloud deployments: Ansible Controller automating directly against endpoints.
When these two elements come together, they offer a highly scalable solution that can perform automation at a massive scale while retaining local autonomy for individual deployments of Controller and keeping the configuration and management of that automation fabric highly automated.
Red Hat Interconnect
While Controller of Controllers is very powerful as an architecture for highly distributed automation, one of the main stumbling blocks at implementation is having to span disparate network topologies. Typically, this includes low bandwidth, unreliable links, multiple layers of stateful firewalls that do not allow inbound access, constantly updating routing tables, and even traversing the public internet.
This is where Red Hat Interconnect enters the architecture. As a layer 7 interconnect, it can lash together services across these network topologies. It presents these services as true services within a Kubernetes namespace, even if the service runs far away at a remote site on a limited compute device within a highly protected network.
Using Red Hat Interconnect, a Controller of Controllers architecture can be secured and spanned across a full-scale edge deployment.
This demonstration uses a ROSA cluster set up to run a parent controller and a RHEL virtual machine running the child controller in a lab environment behind several firewalls and routers that can reach outbound .
Set Up the Parent Controller
First, I've deployed an instance of Ansible Controller via the Ansible Automation Platform operator on top of OpenShift. For demonstration purposes, I'm using the default aap namespace; however, this is configurable.
I've configured a few items to manage child controllers within the parent controller instance. First, I created an inventory of child controllers and added a host:
Next, I created a project containing the YAML configuration of the child controllers:
And finally, I made a job template to run that configures the child controllers:
Here is the child controller configuration for reference:
controller_execution_environments:
- name: Device Edge Workshops Execution Environment
image: quay.io/device-edge-workshops/provisioner-execution-environment:latest
pull: missing
controller_inventories:
- name: Edge Devices
organization: Default
variables:
site: site1
controller_hosts:
- name: switch1
inventory: Edge Devices
variables:
ansible_host: 10.15.108.3
- name: switch2
inventory: Edge Devices
variables:
ansible_host: 10.15.108.4
- name: firewall1
inventory: Edge Devices
variables:
ansible_host: 10.15.108.1
- name: firewall2
inventory: Edge Devices
variables:
ansible_host: 10.15.108.2
- name: dcn1
inventory: Edge Devices
variables:
ansible_host: 10.15.108.10
- name: dcn2
inventory: Edge Devices
variables:
ansible_host: 10.15.108.11
- name: acp-node1
inventory: Edge Devices
variables:
ansible_host: 10.15.108.100
- name: acp-node1
inventory: Edge Devices
variables:
ansible_host: 10.15.108.100
- name: acp-node2
inventory: Edge Devices
variables:
ansible_host: 10.15.108.101
- name: acp-node3
inventory: Edge Devices
variables:
ansible_host: 10.15.108.102
controller_groups:
- name: switches
inventory: Edge Devices
hosts:
- switch1
- switch2
- name: firewalls
inventory: Edge Devices
hosts:
- firewall1
- firewall2
- name: dcns
inventory: Edge Devices
hosts
- dcn1
- dcn2
- name: acp
inventory: Edge Devices
hosts:
- acp-node1
- acp-node2
- acp-node3
controller_credentials:
- name: Device Credentials
organization: Default
credential_type: Machine
inputs:
username: admin
password: 'R3dh4t123!'
controller_projects:
- name: Controller of Controllers with Skupper
organization: Default
scm_type: git
scm_branch: main
scm_url: https://github.com/jjaswanson4/controller-of-controllers-with-skupper.git
controller_templates:
- name: Example Job
organization: Default
inventory: Edge Devices
project: Controller of Controllers with Skupper
playbook: playbooks/sample-playbook.yml
Establish the Connection between the Parent and Child Controller
At a remote site is an instance of Ansible Controller without any configuration. I've installed Skupper (the upstream of Red Hat Interconnect) CLI.
First, authenticate to the OpenShift cluster with the OC CLI tooling:
[jswanson@site1-controller aap-containerized-installer]$ oc login --token=sha256~my-token-here --server=https://api.rosa-nmvzn.goqx.p1.openshiftapps.com:6443
Logged into "https://api.rosa-nmvzn.goqx.p1.openshiftapps.com:6443" as "cluster-admin" using the token provided.
You have access to 103 projects, the list has been suppressed. You can list all projects with 'oc projects'
Using project "default".
Welcome! See 'oc help' to get started.
Next, switch to the namespace where AAP is deployed and initialize Red Hat Interconnect:
[jswanson@site1-controller aap-containerized-installer]$ skupper init
clusterroles.rbac.authorization.k8s.io "skupper-service-controller" not found
Skupper is now installed in namespace 'aap'. Use 'skupper status' to get more information.
[jswanson@site1-controller aap-containerized-installer]$ skupper status
Skupper is enabled for namespace "aap" in interior mode. Status pending... It has no exposed services.
[jswanson@site1-controller aap-containerized-installer]$ oc get pods | grep -i skupper
skupper-router-c56685cc6-bbjs8 2/2 Running 0 28s
skupper-service-controller-6c59c94fb9-mz79m 1/1 Running 0 26s
Here you can see that Interconnect has been initialized in my namespace, and a few pods are now running.
Now, initialize the gateway on the remote system running Ansible Controller:
[jswanson@site1-controller aap-containerized-installer]$ skupper gateway init --type podman
Skupper gateway: 'site1-controller-jswanson'. Use 'skupper gateway status' to get more information.
[jswanson@site1-controller aap-containerized-installer]$ skupper gateway status
Gateway Definition:
╰─ site1-controller-jswanson type:podman version:2.4.1
Once the gateway is initialized, create a service and expose it within the namespace:
[jswanson@site1-controller aap-containerized-installer]$ skupper service create site1-controller 443
[jswanson@site1-controller aap-containerized-installer]$ skupper gateway bind site1-controller localhost 443
2023/06/28 18:23:21 CREATE io.skupper.router.tcpConnector site1-controller:443 map[address:site1-controller:443 host:localhost name:site1-controller:443 port:443 siteId:2c2edf54-4fcf-49d8-a7c3-9a66600a60c4]
Once complete, the connection between the namespace and the remote system running Controller has been established, and a service has been created within the namespace:
[jswanson@site1-controller aap-containerized-installer]$ oc get svc | grep site1-controller
site1-controller ClusterIP 172.30.63.183 <none> 443/TCP 74s
To confirm, set up a route that allows access to the remote controller instance via the OpenShift controller ingress:
kind: Route
apiVersion: route.openshift.io/v1
metadata:
name: site1-controller
namespace: aap
spec:
host: site1-controller.apps.rosa-nmvzn.goqx.p1.openshiftapps.com
to:
kind: Service
name: site1-controller
port:
targetPort: 443
tls:
termination: passthrough
insecureEdgeTerminationPolicy: None
Once you have configured the route, confirm functionality by opening a web browser, accessing the route, and being greeted with the Controller Web Interface:
Use the Parent Controller to Configure the Child Controller
With connectivity configured and functioning, the API of the child controller is now accessible to the parent controller, and you can begin automating against the child controller.
Launch the job template to configure the child controller. Red Hat Interconnect will transparently handle the traffic between the execution environment running the automation and the child controller's API.
To confirm, visit the child controller's web interface and look for configured items, such as hosts:
Wrap Up
This demonstration deployed the Controller of Controller architecture across a highly disparate network topology using Red Hat Application Interconnect as a connection overlay to enable connectivity between a centralized parent Controller and a child Controller.
In the future, this could be expanded to encompass additional services running in the central location, such as source control or image registries for execution environments, or it could be scaled up to include thousands of child controllers.
Sobre os autores
Josh is Red Hat’s industrial edge architect on the global edge architecture team, focused on the industrial edge. He’s worked on the floor of manufacturing plants and built industrial control systems before moving over into enterprise architecture to handle IT/OT convergence. While at Red Hat, he’s worked with large automotive companies, the oil and gas supermajors, and major manufacturing companies on their approach to next generation compute at the industrial edge.
Mais como este
Navegue por canal
Automação
Últimas novidades em automação de TI para empresas de tecnologia, equipes e ambientes
Inteligência artificial
Descubra as atualizações nas plataformas que proporcionam aos clientes executar suas cargas de trabalho de IA em qualquer ambiente
Nuvem híbrida aberta
Veja como construímos um futuro mais flexível com a nuvem híbrida
Segurança
Veja as últimas novidades sobre como reduzimos riscos em ambientes e tecnologias
Edge computing
Saiba quais são as atualizações nas plataformas que simplificam as operações na borda
Infraestrutura
Saiba o que há de mais recente na plataforma Linux empresarial líder mundial
Aplicações
Conheça nossas soluções desenvolvidas para ajudar você a superar os desafios mais complexos de aplicações
Programas originais
Veja as histórias divertidas de criadores e líderes em tecnologia empresarial
Produtos
- Red Hat Enterprise Linux
- Red Hat OpenShift
- Red Hat Ansible Automation Platform
- Red Hat Cloud Services
- Veja todos os produtos
Ferramentas
- Treinamento e certificação
- Minha conta
- Suporte ao cliente
- Recursos para desenvolvedores
- Encontre um parceiro
- Red Hat Ecosystem Catalog
- Calculadora de valor Red Hat
- Documentação
Experimente, compre, venda
Comunicação
- Contate o setor de vendas
- Fale com o Atendimento ao Cliente
- Contate o setor de treinamento
- Redes sociais
Sobre a Red Hat
A Red Hat é a líder mundial em soluções empresariais open source como Linux, nuvem, containers e Kubernetes. Fornecemos soluções robustas que facilitam o trabalho em diversas plataformas e ambientes, do datacenter principal até a borda da rede.
Selecione um idioma
Red Hat legal and privacy links
- Sobre a Red Hat
- Oportunidades de emprego
- Eventos
- Escritórios
- Fale com a Red Hat
- Blog da Red Hat
- Diversidade, equidade e inclusão
- Cool Stuff Store
- Red Hat Summit