3.7. Using the ftrace Utility for Tracing Latencies

3.7. Using the `ftrace` Utility for Tracing Latencies

One of the diagnostic facilities provided with the MRG Realtime kernel is ftrace, which is used to determine where a desired maximum latency is being exceeded. The ftrace utility has a variety of options that allow you to use the utility in a number of different ways. The utility is particularly useful for identifying whether non-deterministic performance results are attributable to the kernel or to user space components. In customer deployments, the tool is most useful to differentiate whether delays are in the kernel or the application.

The ftrace utility is not enabled in the production version of the MRG Realtime kernel as it creates additional overhead. If you wish to use the ftrace utility you will need to download and install the trace variant of the MRG Realtime kernel from the mrg-beta yum repository.

Note

For instructions on how to install kernel variants, or for more information on the mrg-beta yum repository, see the MRG Realtime Installation Guide.

Using the `ftrace` Utility

Once you are using the trace variant of the MRG Realtime kernel, you can set up the ftrace utility. You will need to create a /debug directory and then mount it to use the debugfs file system.
```
# mkdir /debug

# mount -t debugfs nodev /debug
```
To check if the ftrace utility is running, use the cat command to view the /debug/tracing/tracing_enabled file. A value of 1 indicates that ftrace is running, and 0 indicates that it is not running.
```
# cat /debug/tracing/tracing_enabled
1
```
By default, the tracer is enabled. To turn the tracer on or off, echo the appropriate value to the /debug/tracing/tracing_enabled file.
```
# echo 0 > /debug/tracing/tracing_enabled
	
# echo 1 > /debug/tracing/tracing_enabled
```

The ftrace utility has a variety of options that allow you to use the utility in a number of different ways. The available options are:

`events`	Trace specific events.
`wakeup`	Record the time it takes for the highest priority RT task to wake up. If no RT tasks are running, this option will result in no output.
`pre-emptoff`	Record the the longest time for which pre-emption is disabled.
`irqsoff`	Record the the longest time for which interrupts have been disabled.
`pre-emptirqsoff`	Record the the longest time for which either interrupts or pre-emption have been disabled.
`ftrace`	Trace functions using `mcount`.
`sched_switch`	Trace tasks as they are switched by the scheduler.
`none`	Disable all tracing.

To check which options you currently have in the tracer use the cat command on the /debug/tracing/available_tracers file:
```
# cat /debug/tracing/available_tracers
```
To set a single option on the tracer, echo the option name to the /debug/tracing/current_tracer file.
```
# echo ftrace > /debug/tracing/current_tracer
```
Note

If you use a single > with the echo command, it will override any existing value in the file. If you wish to append the value to the file, use >> instead.
The results of any traces are saved to two files. The contents of /debug/tracing/latency_trace contains detailed information:
```
# cat /debug/tracing/latency_trace
```
The /debug/tracing/trace uses a simpler format:
```
# cat /debug/tracing/trace
```

There are a number of options available for changing the format of the output. These options are stored in /debug/tracing/iter_ctrl:

`print-parent`	Show the parent of the functions.
`sym-offset`	Add the offset into the function.
`sym-addr`	Add the address of a symbol.
`verbose`	Increase the verbosity of the tracer output.

Use the cat command to view the current configuration:
```
# cat /debug/tracing/iter_ctrl
```
To set a single option on the tracer output configuration, echo the option name to the /debug/tracing/iter_ctrl file.
```
# echo verbose > /debug/tracing/iter_ctrl
```
To disable a single option on the tracing output configuration, echo the option name with the test no before it to the /debug/tracing/iter_ctrl file.
```
# echo noverbose > /debug/tracing/iter_ctrl
```
Note

If you use a single > with the echo command, it will override any existing value in the file. If you wish to append the value to the file, use >> instead.
The ftrace utility can be filtered by altering the settings in the /debug/tracing/set_ftrace_filter file. If no filters are specified in the file, all processes are traced. Use the cat to view the current filters:
```
# cat /debug/tracing/set_ftrace_filter
```
To change the filters, echo the name of the process to be traced. The filter allows the use of a * wildcard at the beginning or end of a search term. Some examples of filters are:
- Trace only the schedule process:
```
# echo schedule > /debug/tracing/set_ftrace_filter
```
- Trace all processes that end with lock:
```
# echo *lock > /debug/tracing/set_ftrace_filter
```
- Trace all processes that start with spin_:
```
# echo spin_* > /debug/tracing/set_ftrace_filter
```
- Trace all processes with cpu in the name:
```
# echo *cpu* > /debug/tracing/set_ftrace_filter
```
Note

The * wildcard for the tracer filter will only work at the beginning or end of a word. For example: spin_* and *lock will work, but spin_*lock will not.