Adding api.ai voice assistant to Red Hat Mobile Application forms

Red Hat Mobile Application Platform provides a platform to create mobile applications for many use cases and technologies. Workforce management mobile applications are essential in any size corporation and are increasingly popular in many IT organizations. Most workforce management mobile applications include forms that need to be filled out by employees and customers. Red Hat Mobile Application Platform includes many form templates to accelerate the development process of such mobile applications.

Use case

In this blog post, we address the use case where employees need to fill and submit forms as part of their daily workflow and would prefer to use a voice assistant. In this situation, an employee might have his hands busy doing work, or it is simply faster to use voice than actually typing it.

We created a sample project and mobile application for an employee that works for a major racing car company and his role is to test drive cars on different racing tracks and submit a report at the end of each test.

Red Hat Mobile Application Platform forms

Forms and form templates are very well supported as part of Red Hat Mobile Application Platform. You can find many tutorials and guides as part of Red Hat Mobile Application Platform to create form projects that include mobile applications with embedded forms. For this project, we provide the source code for three components:

1. Red Hat Mobile Application Platform CloudApp

2. Red Hat Mobile Application Platform Mobile Client

3. Red Hat Mobile Application Platform Web Client

This code is available for download from github and can be imported into a Red Hat Mobile Application Platform instance.

API.AI

api.ai is a natural language processing service that provides a cloud endpoint for processing language and conversations. In order to use this service, you have to create an account and create a bot for your application. api.ai documentations can be found here. At a minimum, your bot must have an intent and some entities. An intent defines the flow of a conversation, opening and closing statements and questions, variables needed from the conversation and built-in machine learning capabilities.

Creation of the bot can be done manually using the api.ai portal, or it can be executed programmatically using the provided api.ai HTTP/JSON endpoint. For this project, Red Hat Mobile Application Platform CloudApp uses the api.ai endpoint to programmatically create a bot based on the forms associated with the Red Hat Mobile Application Platform project. Detailed information is provided in the coming sections.

Integrating Red Hat Mobile Application Platform forms with API.AI

To integrate Red Hat Mobile Application Platform forms with an api.ai bot, we mapped the Red Hat Mobile Application Platform form fields to fields in the bot’s intent and entity definitions. The first field we mapped is the name of the form mapped to the name of the intent. As seen in the figure below, our form name is “cardrivertestreport” and that maps to an api.ai intent called “cardrivertestreport”.

We also mapped the specific Red Hat Mobile Application Platform form field information to parameter information in api.ai intent and entities. As we can see in the figure below, we mapped the following information:

1. Red Hat Mobile Application Platform form field instruction to api.ai intent field prompt. This prompt is spoken by the api.ai voice assistant when a value is needed for a form field.

2. Red Hat Mobile Application Platform form field code is mapped to the parameter name in api.ai intent.

3. Red Hat Mobile Application Platform form required field was also mapped to the api.ai parameter required field in the intent.

4. Red Hat Mobile Application Platform form field drop down values are mapped to api.ai parameter entities. As we can see in the figure below, Red Hat Mobile Application Platform form field “carmodel” has a drop down menu with multiple values that are mapped to api.ai entity “carmodel” values.

The mapping and creation of api.ai intents and entities are done programmatically by the Red Hat Mobile Application Platform CloudApp. CloudApp gathers all associated forms of a Red Hat Mobile Application Platform project, reads all the fields and does the mapping programmatically using the api.ai HTTP/JSON endpoint.

To start this integration process we created a Red Hat Mobile Application Platform Web Client that provides a push button for initiating this integration. As seen in the figure below, pushing the “Power” button initiates the integration code in CloudApp.

Background image downloaded from pixabay

In the figure below, we can see an overview of the project. In the center, we see the Red Hat Mobile Application Platform project containing the CloudApp, Web Client and Cordova mobile application. To initiate the integration process, we use the Web Client to push the “Power” button that calls the CloudApp. CloudApp reads all forms associated with this project and sets up intents and entities on api.ai. After the setup is complete and api.ai is provisioned, the user can start using the mobile application in the field. In our use case, we have the race car driver filling and submitting forms handsfree while he is driving using his voice only.

Demo and code

All the code used in this project can be found on github:

• Red Hat Mobile Application Platform CloudApp: https://github.com/nakfour/apiai-form-cloudapp

• Red Hat Mobile Application Platform Mobile Client: https://github.com/nakfour/apiai-form-template

• Red Hat Mobile Application Platform Web Client: https://github.com/nakfour/apiai-form-webapp

You can also watch this two-part video demo presentation:

• Part 1: Using Red Hat Mobile Application Platform Web Client to provision intents and entities on api.ai

• Part 2: Race car driver using Red Hat Mobile Application Platform mobile application to fill and submit forms handsfree and using voice assistant

Conclusion

In this blog post, we demonstrated how to add a voice assistant to Red Hat Mobile Application Platform forms. We kept it simple by using one context in an api.ai intent. However, for more detailed and complicated forms and conversations, more than one context is most likely needed in api.ai intents.

Juana Nakfour is a senior Mobile Technical Account manager with more than 15 years advanced R&D experience in the telecommunications industry. Juana’s focus is broad and includes wireless communication networks, embedded/mobile devices and microservices.

A Red Hat Technical Account Manager (TAM) is a specialized product expert who works collaboratively with IT organizations to strategically plan for successful deployments and help realize optimal performance and growth. The TAM is part of Red Hat’s world-class Customer Experience and Engagement organization and provides proactive advice and guidance to help you identify and address potential problems before they occur. Should a problem arise, your TAM will own the issue and engage the best resources to resolve it as quickly as possible with minimal disruption to your business.

Connect with TAMs at a Red Hat Convergence event near you! Red Hat Convergence is a free, invitation-only event offering technical users an opportunity to deepen their Red Hat product knowledge and discover new ways to apply open source technology to meet their business goals. These events travel to cities around the world to provide you with a convenient, local one-day experience to learn and connect with Red Hat experts and industry peers.