What's all this trace stuff anyhow?

Previously in this blog we have talked about the value of code tracing for embedded systems development and how trace forms the entrance to being able to profile your code and perform code coverage. For an embedded application, such types of code analysis typically require an investment in development tools as well as functional target hardware upon which to perform the analysis. If, however, you are simply interested in gaining an insight into what is possible, there are some other ways of experiencing these tools.

Recently the team was invited to the Hochschule für angewandte Wissenschaften München to present on the topic of "Debugging and Profiling for Embedded System Development". Unfortunately, we are not in a position to enable each student to trial our tools but the department head told me that the students had been developing applications using the Beagle Bone Black. Luckily for us, development platforms such as Beagle Bone and Raspberry Pi which run Linux are ideal to practise with trace, profiling and code coverage.

Prior to preparing this presentation I had participated in a webinar from Doulos in which the various tools available in Linux for finding and resolving performance issues were reviewed. In this blog post we will review how to perform trace, profiling and code coverage on most Linux distributions so that you can get a feel for what insights such capabilities offer the developer.

Trace - on an embedded system, when using a development tool package such as winIDEA and an iC5500, trace is used to actually capture all of the assembler instructions executed by the CPU of a microcontroller. These are then spliced together with the C or C++ source code to which it related. Under Linux, the trace tools we show here are working already at high-level language or even operating system layer.

The first tool we will take a look at is "top". Simply by typing top at the command line, you will get to see which processes are executing and how much CPU and memory resources they are currently using.

top displays which processes are running and how much of the CPU and memory they are using.

If an application we have written is consuming masses of processor cycles or is eating up more and more memory, we will quickly see it displayed high up on this list. To get a little more detail, we have to dig a bit deeper.

The command line tool strace gives us more background on where the failure in our code might be. This is achieved by capturing all of the functions your application is calling and time-stamping them. The developers hope would be that, using the results output by this tool, a system call that is taking a long time to execute may be highlighted, or perhaps a system call is being called multiple times unnecessarily. Below we show how the the trace tool can be used to analyse the Xfce desktop application globaltime, a multiclock timeconverter. For comparison, the trace output for an embedded system as it is displayed in winIDEA is also displayed.

Example of using strace under Linux compared to trace on an embedded MCU target with winIDEA.

Profiling - If you would like to see the results from "strace" formatted as statistics, try calling strace -c globaltime instead. In this way it is possible to get a profile of how much time was spent in which portions of your application, and how often individual system calls were called. An alternative tool is sysprof which is GUI based. Through this visual interface we can analyse how much time is being spent in which system calls for any running application. This is analogous to the profiler within winIDEA which is shown below for comparison.

sysprof being used to analyse the functions that globaltime is calling under Linux. A similar profiling example is shown on the right using winIDEA and an embedded MCU.

Code Coverage - finally we take a look at how we go about proving if we have tested all of our application's code. The tools lcov and genhtml together form a great way to generate documentation for your Linux project to prove that you have actually tested all your code. The first performs the coverage analysis whilst the second formats the results as HTML to include in, for example, your project's repository. An example of this is also shown alongside the code coverage view within winIDEA on an embedded target MCU.

lcov and genhtml results for a simple application. On the right, code coverage as displayed within winIDEA.

If you would like to learn more about the tools integrated into Linux for debugging your application, we highly recommend the website from Brendan Gregg.

So that is all for this blog post. We hope that even if you don't have access to a microcontroller based embedded system and appropriate development tools you can still get a feel for how engineers go about ensuring that their code is working correctly using standard Linux tools. It all helps to ensure that it has all be tested and proven to be both reliable and safe.

Until next time, Happy BugHunting!