October 12, 2006

Got a nasty bug you can't replicate in your test environment? Frysk lets you monitor and debug in real time. Find out how to rapidly locate and squash the bugs that plague your open source app.

The Frysk Monitor

by Sami Wagiaalla

Frysk logo

One of the ways Frysk can be used is as an execution analysis tool.

"What is an execution analysis tool," you ask?

An execution analysis tool is a piece of software that enables a user to monitor the execution of a program in anticipation of problems. And when those problems occur, it enables that user to dive in and analyze those problems through the tools provided by a traditional debugger.

Frysk, then, is a non-traditional debugger. It does, of course, provide the functionalities provided by all debuggers, but it can also be used to monitor the debugee before the problems occur. Plus it gives the user a chance to debug while the debugee is still alive and the problem is still present.

No login required. Want to see your comments in print? Send a letter to the editor.

Some of the functionalities of Frysk were addressed in the article "Frysk: Debugging or Monitoring Tool?" by Rick Moseley, published in the previous issue of this magazine. This article will focus on the monitoring side of Frysk.

Before we dive into the details of the Frysk Monitor, let's talk about its grand goal. As mentioned by Rick in his article, one of the requirements given by the Red Hat Customer Advisory Board (CAB) members is that the desired tool should be able to debug large-scale multi-threaded applications, and even problems that may span more than one application.

How does one debug an application with a thousand threads? How does one display this large sum of information in an abstracted way that is still relevant to the user? How does one allow the user to request more information when required? These are the grand goals of the Frysk Monitor.

Sound tough? I agree.

The first window the user sees when he or she runs Frysk is the Startup Manager Dialog.

Frysk Monitoring Window screenshot
Frysk Startup Manager screenshot.

The Startup Manager allows the user to choose the mode they would like to run Frysk in. The user can debug an existing process, which allows the user to select a process and go directly to a source window where he or she can debug it much as one would do with a traditional debugger.

The other two options involve Debug Sessions. A session is a way to narrow down the focus of Frysk Monitor, and Frysk in general, to the process or processes that the user is interested in. In the Startup Manager, the user also has the option of opening a blank session with a terminal. A session with a terminal gives the user a Linux command line terminal, and the session is then populated with the processes that are run from under that terminal. The user can also create a new session or open one that they have previously created.

Clicking New takes the user to the Create Session Assistant. There the user gives a name to the session and chooses the processes that he or she is interested in by moving them from the list on the left, representing all the processes currently running in the system, to the list of the right.

Frysk Create Session Assistant
Frysk Create Session Assistant

The process groups list groups together processes that have a common executable. The reason for this is that the executable path, as opposed to the ID of a process, remains relevant even when Frysk is restarted. A process ID could have been claimed by another process or it could no longer refer to a process at all. Hence, the user only selects the executable path and Frysk is then responsible for populating the session with processes that have that path.

Frysk Process Groups
Process Groups

Clicking Next takes the user to the next page where they can choose the observers they want to add to these processes as well as create custom observers. Observers and Custom Observers will be addressed in detail later in this article. Clicking Finish ends the session creation process and starts the Frysk Monitor.

The Frysk Monitor
The Frysk Monitor

The Frysk Monitor is composed of three main sections: the process view on the left, the status view on the top right, and the log on the bottom right.

The Processes view shows the list of processes on which the current session is focused and displays some information about them. As can be seen here, this session only has one process: the firefox-bin we selected when creating the session. The right-click context menu allows the user to manipulate these process and add observers to them.

The Status view contains event traces for each process in the session and each thread belonging to each one of those processes. The traces basically display a line each time an event occurs. For example, an observer that has been added to that process/thread fires. Hovering over an event line displays more information about the event. Future development plans include enabling the user to click on an event line and go to the relevant area in the source code. In this screen shot we can see that the process called funit-child has just made a clone system call resulting in a new thread being added to the process.

Status window

As Frysk implements more observers, the user will be able to add an observer to watch for virtually any event in the process' code. The resulting traces provide the user with an abstracted view of the process space that hides the gory details--until they are required--while still correctly modeling process behavior.

This abstraction and correct modeling is exactly what users seek when they use print statement debugging. A very simple example of the effectiveness of such an abstraction would be catching an infinite loop in a program. While it is possible to see the infinite loop by stepping though the program, one might easily miss it while too busy examining local states of code and variables. With a correctly-placed print statement, or adding an observer that watches for correct function entry and exit, for example, the pattern would be much easier to see.

The third part of the Frysk Monitor window, the Main Log view, is currently utilized to print statements logged by observers when they fire.

The observer paradigm is one of the fundamental design concepts Frysk. The Frysk core communicates to the GUI only though requests and events. Observers are the clients that listen for those events and notify the interested parties.

The basic observers provided by Frysk can be customized to a certain extent through the Frysk UI. Through the UI the user can create custom observers by adding filters and actions to them. A filter is simply a way of more narrowly focusing on one of the parameters of an event received by the observer. And actions are tasks to be performed once that observer fires. Common examples of actions are spawning a source window showing the current position of execution in the code, and then logging the event.

A simple example of how the primitive Frysk observers can be combined to create more elaborate functionality is the program observer. The job of the program observer is to catch newly-spawned programs precisely at the moment when they start executing their first instruction after the exec system call. A program observer is basically a fork observer that is added to a process. As soon as that process makes a fork system call, the program observer adds itself to the newly-created child as an exec observer. When the child calls exec the program observer catches that event, checks the arguments of the exec and decides whether this is the executable of interest or not. By adding the Frysk program observer to processes in the system that are commonly responsible for creating new processes (such as /bin/bash and gnome-panel) one is able to catch most events of new processes as they are launched. This is fundamental to Frysk sessions, since now they will be able to correctly populate their process list with newly created instances of the executables that the user has chosen, no matter how short-lived those processes are.

Development continues in high flux on all frontiers of Frysk. Development in the Frysk core will result in observers with much higher resolution than is currently available. Frysk Monitor trace lines are being developed to make them more sophisticated, robust, and flexible. As development continues, and as we continue to come closer to understanding the debugging experience, we hope to see the Frysk Monitor revolutionize debugging, and become the tool that we envision it to be.

How to get involved with Frysk

If you want to become involved with Frysk by working on code, submitting ideas for enhancement requests or just monitoring the development, please visit the Frysk website. Here you can subscribe to the Frysk mailing list or find out where the Frysk developers hang out on a public IRC channel.

About the author

Sami is an intern here at Red Hat from the University Of Alberta. During his 16-month tenure he has worked entirely on Frysk making contributions in all areas. Most of his contributions have been to the monitoring side of Frysk and in the system call-tracing. Sami has contributed quite a bit of code to the Java-Gnome project where he was given committer privileges because of his quality contributions for fixing bugs and adding functionality.