How to onboard edge devices at scale with FDO and Linux

Onboarding edge and Internet of Things (IoT) devices at scale creates significant challenges for sysadmins. To help address them, the FIDO Alliance created the FIDO Device Onboard (FDO) specification as the first widely accepted industry standard aimed at solving edge computing and IoT device onboarding challenges.

In previous articles for Enable Architect, I reviewed some of the challenges when deploying edge computing devices and described the FDO specification. This article brings these concepts to life by showing how you can use FDO with a Red Hat Enterprise Linux (RHEL) for Edge deployment.

How FDO is implemented in Linux

FDO is a specification released by the FIDO Alliance for onboarding IoT and edge computing devices. Linux runs on various CPU architectures, including x86, Arm, RISC, PowerPC, and MIPS. Due to its modularity and resource usage optimizations, Linux can run on small hardware footprint systems, making it suitable for IoT and edge computing devices from smartwatches to satellites floating in space. Linux's open source approach offers a high level of interoperability, extensibility, and innovation, which are important in edge computing.

These are some of the reasons the Linux community is focusing on new features and products for edge computing use cases. This is why you can find RHEL distributed as RPM packages and Libostree images that include system packages to install the RHEL for Edge operating system remotely.

The FDO specification is a good addition to Linux for edge computing and IoT environments. I'll dig into the FDO implementation in RHEL 8.6 (as a technology preview) and RHEL 9.

[ Get started with Automation at the edge. ]

FDO service components

Fedora Linux is the upstream project for RHEL, and you'll find the FDO implementation (written in Rust) in Fedora's GitHub repository. This source-code repository contains the core FDO elements I discussed in Edge device onboarding: What architects need to consider and the protocols used to exchange information between them. The core components are:

• Manufacturing server: Located at the manufacturer's premises, this server signs the device and creates a voucher to set device ownership and bind it to a specific platform or cloud.
• Rendezvous server: This server is located at the owner's premises or the platform where the device management system will be deployed (such as a public cloud). It receives the voucher generated by the Manufacturing server during the first device boot. The server then matches the device UUID with a target platform or cloud and provides information to the device about what owner-server endpoint it should use.
• Owner/onboarding server: This is located at the owner's premises or the platform where the device will be deployed. After device authentication, a secure channel is created between the device and this server. The server then sends the required pieces (such as files and scripts) for the onboarding automation to the device.
• Device client: Installed on the device as the client piece of the solution, it starts queries to the servers where the onboarding automation will be executed. This client could be implemented on top of the device's processor or co-processor firmware, a microcontroller unit, or as a process of an operating system.

In this tutorial, I'll deploy and configure the different FDO services in dedicated systems running RHEL (targeting three separate IP addresses).

FDO is distributed as RPM packages (the fdo-client is included in RHEL for Edge images by default):

# dnf search fdo
...
===================================================================================== Name & Summary Matched: fdo ====================================================================
fdo-admin-cli.x86_64 : FDO admin tools implementation
fdo-client.x86_64 : FDO Client implementation
fdo-manufacturing-server.x86_64 : FDO Manufacturing Server 
fdo-owner-cli.x86_64 : FDO Owner tools implementation
fdo-owner-onboarding-server.x86_64 : FDO Owner Onboarding Server 
fdo-rendezvous-server.x86_64 : FDO Rendezvous Server implementation

An additional package, fdo-aio (FDO all-in-one), installs all components at once and is useful for labs or demos. I will include some details about preparing the server, but check the official Red Hat FDO documentation for more information.

[ Learn how to manage your Linux environment for success. ]

Set up an FDO server in RHEL

You might have multiple people involved in FDO server preparation. Perhaps one person prepares the Manufacturing server, another is in charge of the platform (or a third-party public cloud) and configures the Rendezvous server at the user/owner's location, and someone else configures the boot automation steps on the Owner/onboarding server.

Whether you're doing the entire setup yourself or you're part of a team doing so, you can use the following steps to set up your FDO server.

1. Configure the Manufacturing server

Start by installing the required RPMs:

# dnf install -y fdo-admin-cli fdo-manufacturing-server

During the manufacturing phase, you must "inject" certain certificates and keys into your device, so create them first.

Create a directory to store the certs and keys:

# mkdir /etc/fdo/keys

You can use the fdo-admin-tool to create those secrets. There are four groups: manufacturer, owner, device-ca, and diun (Device Initialize over Untrusted Networks) certs and keys for communication:

# for i in manufacturer owner device-ca diun ; do
fdo-admin-tool generate-key-and-cert \
--destination-dir /etc/fdo/keys $i
done

Check that the certificates and keys exist in the correct directory:

# ls /etc/fdo/keys/
device_ca_cert.pem  device_ca_key.der  diun_cert.pem  diun_key.der  manufacturer_cert.pem  manufacturer_key.der  owner_cert.pem  owner_key.der

The manufacturer service also stores other components, such as the owner vouchers, the manufacturer keys, and information about the manufacturing sessions. Create a dedicated directory for each:

# mkdir -p /etc/fdo/stores/manufacturer_keys
# mkdir -p /etc/fdo/stores/manufacturing_sessions
# mkdir -p /etc/fdo/stores/owner_vouchers

It's time to configure the Manufacturing server. You can find an example configuration in /usr/share/doc/fdo/manufacturing-server.yml. In my case, this is the manufacturing-server.yml configuration file that I created and placed in the /etc/fdo directory:

---
session_store_driver:
  Directory:
    path: /etc/fdo/stores/manufacturing_sessions/
ownership_voucher_store_driver:
  Directory:
    path: /etc/fdo/stores/owner_vouchers
public_key_store_driver:
  Directory:
    path: /etc/fdo/stores/manufacturer_keys
bind: "0.0.0.0:8080"
protocols:
  plain_di: false
  diun:
    mfg_string_type: SerialNumber
    key_type: SECP384R1
    allowed_key_storage_types:
      - Tpm
      - FileSystem
    key_path: /etc/fdo/keys/diun_key.der
    cert_path: /etc/fdo/keys/diun_cert.pem
rendezvous_info:
  - deviceport: 8082
    ip_address: 192.168.122.99
    ownerport: 8082
    protocol: http
manufacturing:
  manufacturer_cert_path: /etc/fdo/keys/manufacturer_cert.pem
  device_cert_ca_private_key: /etc/fdo/keys/device_ca_key.der
  device_cert_ca_chain: /etc/fdo/keys/device_ca_cert.pem
  owner_cert_path: /etc/fdo/keys/owner_cert.pem
  manufacturer_private_key: /etc/fdo/keys/manufacturer_key.der

This configuration contains the paths to the directories, certificates, and keys you created, along with the Rendezvous server IP address and port (the default is 8082). I configured one Rendezvous server, but you can include more.

Once /etc/fdo/manufacturing-server.yml is prepared, you can start the Manufacturing server. First, check that you have the systemd unit file in the server:

# systemctl list-unit-files | grep fdo | grep manufac
fdo-manufacturing-server.service disabled disabled

Enable and start it:

# systemctl enable --now fdo-manufacturing-server.service

Check that the service is listening on port 8080 (default):

# ss -ltn

Remember to open the required ports in your firewall:

# firewall-cmd --add-port=8080/tcp --permanent
# systemctl restart firewalld

2. Configure the Rendezvous server

As you did with the Manufacturing server, install the required RPM packages:

# dnf install -y fdo-rendezvous-server

Create a directory for the required certificates:

# mkdir -p /etc/fdo/keys

The Rendezvous server needs the manufacturer_cert.pem certificate you created in the Manufacturing server, so copy it from there.

[ Download now: Advanced Linux commands cheat sheet. ]

As you did with the Manufacturing server, create the directories needed by the Rendezvous server, including the registered directory (where the registered device owner vouchers will be hosted) and the sessions directory:

# mkdir -p /etc/fdo/stores/rendezvous_registered
# mkdir -p /etc/fdo/stores/rendezvous_sessions

Next, create the configuration file where you include the path to the manufacturer certificate. I created this file (located in /etc/fdo/rendezvous-server.yml) from the example that you can find in /usr/share/doc/fdo/rendezvous-server.yml:

---
storage_driver:
  Directory:
    path: /etc/fdo/stores/rendezvous_registered
session_store_driver:
  Directory:
    path: /etc/fdo/stores/rendezvous_sessions
trusted_manufacturer_keys_path: /etc/fdo/keys/manufacturer_cert.pem
max_wait_seconds: ~
bind: "0.0.0.0:8082"

Check the status of the Rendezvous server service:

# systemctl list-unit-files | grep fdo | grep rende
fdo-rendezvous-server.service disabled disabled

Since it's probably stopped and disabled, enable and start it:

# systemctl enable --now fdo-rendezvous-server.service

Check that the server is listening on the configured port (default 8082):

# ss -ltn

And remember to open the port if you have a firewall configured on this server:

# firewall-cmd --add-port=8082/tcp --permanent
# systemctl restart firewalld

3. Configure the Owner server

Install the required RPMs in this server:

# dnf install -y fdo-owner-cli fdo-owner-onboarding-server

Create the certificate and keys directory:

# mkdir -p /etc/fdo/keys/

Copy the device_ca_cert.pem, owner_key.der, and owner_cert.pem certificates from the Manufacturing server into that new directory.

Create the additional directories needed by the owner service:

# mkdir -p /etc/fdo/stores/owner_vouchers
# mkdir -p /etc/fdo/stores/owner_onboarding_sessions

Prepare a configuration file in /etc/fdo/owner-onboarding-server.yml based on the example found in /usr/share/doc/fdo/owner-onboarding-server.yml:

---
ownership_voucher_store_driver:
  Directory:
    path: /etc/fdo/stores/owner_vouchers
session_store_driver:
  Directory:
    path: /etc/fdo/stores/owner_onboarding_sessions
trusted_device_keys_path: /etc/fdo/keys/device_ca_cert.pem
owner_private_key_path: /etc/fdo/keys/owner_key.der
owner_public_key_path: /etc/fdo/keys/owner_cert.pem
bind: "0.0.0.0:8081"
service_info_api_url: "http://localhost:8083/device_info"
service_info_api_authentication:
  BearerToken:
    token: Kpt5P/5flBkaiNSvDYS3cEdBQXJn2Zv9n1D50431/lo=
owner_addresses:
  - transport: http
    addresses:
      - ip_address: 192.168.122.149
    port: 8081
report_to_rendezvous_endpoint_enabled: true

Include the path to the certificates you already copied and information about where to publish the Owner server service in this file. You will also find references to the Info API Service, such as the URL or the authentication token. In the FDO RHEL implementation, you'll find this Service Info API, which you use to introduce the automation needed for onboarding by including the SSH key, files to be copied or created, commands to be executed, disk to be encrypted, and so forth.

There is an example located in /usr/share/doc/fdo/serviceinfo-api-server.yml you can use as a template to create the configuration file under /etc/fdo/serviceinfo-api-server.yml. Use this to test the service (just copy the /etc/resolv.conf file to /root on the device and run the touch /root/test command):

---
service_info:
  initial_user: 
    username: admin
    sshkeys:
    - "ssh-rsa AAAA...."
  files:
  - path: /root/resolv.conf
    source_path: /etc/resolv.conf
  commands: 
  - command: touch
    args:
    - /root/test
    return_stdout: true
    return_stderr: true
  diskencryption_clevis: 
  - disk_label: /dev/vda4
    binding:
      pin: tpm2
      config: "{}"
    reencrypt: true
  additional_serviceinfo: ~
bind: "0.0.0.0:8083"
device_specific_store_driver:
  Directory:
    path: /etc/fdo/stores/serviceinfo_api_devices
service_info_auth_token: Kpt5P/5flBkaiNSvDYS3cEdBQXJn2Zv9n1D50431/lo=
admin_auth_token: zJNoErq7aa0RusJ1w0tkTjdITdMCWYkndzVv7F0V42Q=

Note: Bear in mind that this configuration will use a Trusted Platform Module (TPM) device to encrypt the disk. Be sure that your device or virtual machine has a TPM, or you will get an error when booting the edge device.

After creating the owner and API service configuration files, check the status of the systemd units:

# systemctl list-unit-files | grep fdo
fdo-owner-onboarding-server.service        disabled        disabled
fdo-serviceinfo-api-server.service         disabled        disabled

Enable and start them if necessary:

# systemctl enable --now fdo-owner-onboarding-server.service
# systemctl enable --now fdo-serviceinfo-api-server.service

Note: Every time you change the configuration files, remember to restart the systemd services.

Check that the server is listening in the configured port (defaults are 8081 for the owner service and 8083 for the API):

# ss -ltn

And remember to open the port if you have a firewall configured in that server:

# firewall-cmd --add-port=8081/tcp --permanent
# firewall-cmd --add-port=8083/tcp --permanent
# systemctl restart firewalld

4. Configure the FDO client

Now it's time to deploy the FDO client in RHEL for Edge. When you create a RHEL for Edge image using RHEL 8.6 or above, the image builder that creates the image automatically includes the fdo-client, so you won't need to do anything special to set up the FDO client.

Prepare the edge devices

Now that all FDO services are running, you can start preparing the edge devices following the FDO workflow (see my Enable Architect article for details).

5. Prepare the device at the manufacturing location

The steps to prepare the device are:

• Install the tools required to identify the device, such as the FDO client and the set of certificates and keys.
• Install the Rendezvous server endpoint (URL).
• Create the ownership voucher.

These steps create a device installed with the certificates that authenticate the system and the FDO client. It is ready to contact the Rendezvous and Owner servers to start the onboarding customization.

Rather than creating the raw image and embedding it into the hardware, RHEL for Edge's approach creates an installable ISO to prepare (preinstall) the device. As mentioned above, this includes the FDO client and the certificates and keys.

To create the ISO for FDO, you first need a RHEL for Edge image published on an HTTP server, which is used to embed it into the ISO.

This article does not cover how to create RHEL for Edge images. You can find that information in the official Red Hat documentation. I also wrote some quick-start scripts to create and publish RHEL for Edge images quickly. Use script 1 to create the image and script 2 to publish it on the HTTP server.

Once you have the RHEL for Edge image on the HTTP server, you need to create a custom ISO using that image as a base. You can do that using script 3 in my repo linked above. If you'd like to do the steps manually, here is the process:

1. Create a new blueprint file. Previously, you needed a blueprint file to create the image; this one is different. Here, you include the FDO Manufacturing server URL rather than the packages to install as part of the image:

   name = "fdo"
   description = "Blueprint for FDO"
   version = "0.0.1"
   packages = []
   modules = []
   groups = []
   distro = ""
   [customizations]
   installation_device = "/dev/vda"
   [customizations.fdo]
   manufacturing_server_url = "http://192.168.122.158:8080"
   diun_pub_key_insecure = "true"

2. Upload the blueprint to the image builder server:

   # composer-cli blueprints push blueprint-fdo.toml

3. Create the ISO using the edge-simplified-installer image type, referencing the RHEL for Edge image on the HTTP server:

   # composer-cli compose start-ostree fdo edge-simplified-installer --ref rhel/9/x86_64/edge --url http://<repo_server_ip>:<repo_server_port>/repo/

4. When the creation process is finished, download the ISO using the UUID created by the previous step:

   # composer-cli compose image 6a0f270c-8c9e-467c-99a4-1afd4d77a8bb

Your RHEL for Edge installation ISO is ready for FDO onboarding. You have two options:

• You can use that ISO at the manufacturing site to preinstall your systems and then distribute them to their final locations (at the owner's site).
• You can directly distribute the ISO to preinstall the devices at the owner's site.

The first option is probably the better one, since you will need access to the Manufacturing server while performing the preinstallation and can skip that step at the owner's site, where you might not have specialized people who, for example, know how to boot from ISO.

You can boot your edge computing device(s) using the ISO at the manufacturing site manually or, if you want to run the deployment at scale, you can host the ISO in an HTTP server and boot the devices using network UEFI HTTP boot. Remember that this ISO is prepared to use UEFI boot (otherwise, you will get a code 0009 error).

Once the device boots from that ISO, it will perform the preinstallation automatically by contacting the Manufacturing server and creating the owner voucher that will be stored in the /etc/fdo/stores/owner_vouchers directory that you created during the Manufacturing server preparation.

After the installation, the system will reboot directly from the hard disk and will try to start the boot-time automation to complete the system installation. This won't happen unless the device has access to the Owner server. Bear in mind that the device may not have access to this server at the manufacturing site.

6. Send the device and the ownership voucher to their destination

The manufacturing step creates two outputs: the device/ISO and the owner voucher. You must physically send the device to its final destination while simultaneously copying the owner voucher into the Owner server (in this example, that's the /etc/fdo/stores/owner_vouchers directory).

In my test environment, I have network access from the Manufacturing server to the Owner server, so I can use secure copy (SCP):

scp /etc/fdo/stores/owner_vouchers/3b643591-30ef-5509-2451-b0a0ff044640 root@192.168.122.149:/etc/fdo/stores/owner_vouchers/

7. Register the device in the Rendezvous server

A few seconds after copying the voucher into the Owner server, the Rendezvous server will also register the device automatically (there is no need to copy anything manually). You can check that the voucher you found in the /etc/fdo/stores/owner_vouchers directory (that's now in both the Manufacturing and Owner servers) is copied into the /etc/fdo/stores/rendezvous_registered directory at the Rendezvous server.

[ Cheat sheet: Get a list of Linux utilities and commands for managing servers and networks. ]

8. Boot the device

Now that everything is configured, it's time to connect, power on, and boot the device.

Right after the first boot, the FDO client identifies itself, creates a secure channel, and does the automation you previously configured in the service API's service configuration file on the Owner server.

You can check the logs to see if the FDO client completed its operations:

# journalctl | grep fdo
Aug 18 03:43:16 localhost.localdomain fdo-client-linuxapp[1055]:  2022-08-18T07:43:16.988Z INFO  fdo_client_linuxapp > Found device credential at FileSystemPath { path: "/boot/device-credentials", deactivation_method: None }
Aug 18 03:43:17 localhost.localdomain fdo-client-linuxapp[1055]:  2022-08-18T07:43:17.147Z INFO  fdo_client_linuxapp > Got TO2 addresses: ["http://192.168.122.149:8081"]
Aug 18 03:43:17 localhost.localdomain fdo-client-linuxapp[1055]:  2022-08-18T07:43:17.147Z INFO  fdo_client_linuxapp > Performing TO2 protocol, URL: "http://192.168.122.149:8081"
Aug 18 03:43:17 localhost.localdomain fdo-client-linuxapp[1055]:  2022-08-18T07:43:17.251Z INFO  fdo_client_linuxapp::serviceinfo > Creating file "/root/resolv.conf" with 55 bytes (mode 384)
Aug 18 03:43:17 localhost.localdomain fdo-client-linuxapp[1055]:  2022-08-18T07:43:17.261Z INFO  fdo_client_linuxapp::serviceinfo > Initiating disk re-encryption, disk-label: /dev/vda4, pin: tpm2, config: {}, reencrypt: true
Aug 18 03:43:18 localhost.localdomain fdo-client-linuxapp[1055]:  2022-08-18T07:43:18.438Z INFO  fdo_client_linuxapp              > Secure Device Onboarding DONE
Aug 18 03:43:18 localhost.localdomain systemd[1]: fdo-client-linuxapp.service: Deactivated successfully.

Your device onboarding is now complete.

This is just an example. Creating a file and running touch does not seem to be a great use case for FDO. I tried to make it simple, but you could edit the service API configuration file to do something like register your RHEL into console.redhat.com with your credentials (note that you don't have to include those credentials in the ISO or the device).

I created a couple of FDO demos, including one that registers the device, that may be useful to start playing with. Please consider contributing to luisarizmendi/rhel-edge-quickstart's development on GitHub.

Wrapping up

The FIDO Device Onboarding (FDO) specification is a welcome approach to overcoming edge and IoT device onboarding challenges. It pays attention to security, offers a separation of responsibilities between the different entities in the distribution chain, and provides solutions to address changes in the chain of ownership during the process.

This article is based on Edge Computing device onboarding — Part III — FDO implementation in Red Hat Enterprise Linux from Medium and is republished with the author's permission.