iSYSTEM has published a new technical article describing the features and benefits of Slow Run mode.
Slow Run allows the capture of the complete program execution and data trace – although the MCU does not provide a debug trace port. Slow Run mode is not a substitute for powerful MCUs with trace ports and according debuggers, but it shows every developer the insights of the complete application’s execution, where without Slow Run mode only teeny-weeny details are visible.
In winIDEA 9.12.7 iSYSTEM introduced the Slow Run mode that allows the acquisition of a complete program trace - also on CPUs without trace port.
Since winIDEA 9.12.14 in addition to program execution, data flow and code coverage analysis Slow Run mode supports also profiling. Profiling analyses when and how long a function is running and provides statisticson many different parameters e.g.
count (how often was the function called)
period (what is the time difference between calls to a function)
net time (time spent in function without subroutines and interrupts)
gross time (time spent in function including subroutines, but without interrupts)
call time (time spent in function including subroutines and interrupts)
Below picture shows a Slow Run trace made over a JTAG interface and the output of winIDEA's profiler and code coverage (CC) analysis. The trace includes the complete program and data flow, although JTAG does not provide a trace port and without Slow Run mode all windows would be empty or - in case of CC - would show not covered.
To help customers with power measurement iSYSTEM provides now a so called Power Probe!
As mentioned in last week's blog post the iC5000 I/O module allows now power measurement. But what do you do if your target's voltage and current exceed the input range of the I/O module?
Take iSYSTEM's Power Probe!
The Power Probe is a small board that is connected between the power supply and the target hardware. On the Power Probe you find two resistor blocks - one divider block for voltage and one shunt resistor block for current. If you need to scale down the voltage or current you just select the appropiate resistor by a jumper.
And now you can connect the I/O module to measure the power consumption!
As mentioned in last week's blog post if you use divider resistors remember to specify in "Hardware/ Options/ IO" dialog a "Multiply" value and the Ohm value for the shunt resistor.
The iC5000 I/O module can now measure the power consumption of your target board! And in winIDEA's profiler you can view the uptake rate (current, power, voltage)!
Basically you will use on the I/O module port AIN0 for voltage measurement and port AIN1 for current measurement. Please select in winIDEA "Hardware/ Options/ IO" dialog the appropiate check box (in picture on the right). But before doing so please consider the input and output range of the I/O module. If your signal is higher than e.g. 5V, use a suitable resistor divider.
In winIDEA "Hardware/ Options/ IO" dialog you specify in "Analog IN" names for the incoming signals. If you use a divider resistor, ensure to set the correct "Multiply" value (in picture on the left).
For current measurement use a shunt resistor to protect the IO module. For correct measurements specify the Ohm value of the resistor (in picture on the right).
The actually measured values of current, power and voltage are shown in winIDEA's profiler over the period of time if you have configured the profiler to profile AUX signals too (profiler configuration dialog, section "Profile", check AUX box) .
For more information and a detailed description of the setup see iC5000 I/O module.
testIDEA can now verify the correctness of test specifications against the debug information. The following items are verified: existence of functions and global variables, hiding of global variables by local variables, duplicate IDs.
Why is that a nice feature? It can happen that typos in the test specification e.g. in the function or variable names, wrong number of function parameters or renamed function or variable names in the source code cause invalid test results or even a target crash.
To discover such problems before test execution, iSYSTEM added the new command "Tools / Verify symbols" to testIDEA. Problems are shown in the Status View. By double clicking the problem line an explanation to the problem is shown in the right (light red) window and the Test Specification Editor view switches to the test specification related to the problem.
iSYSTEM offers organizational measures and documentation as well as a tool setup to validate a dedicated functionality of iSYSTEM's tools according to safety standards like e.g. ISO26262.
Standards require qualification for software tools used in the development and test process before the actual development starts.
iSYSTEM calls it realtime unit test because all tests are executed on the real hardware without any effects on the runtime behavior.
iSYSTEM has enhanced their debug and test tools to allow the generation and execution of test cases from inside the development environment. Realtime unit testing runs test scripts and verifies the application without any influence on its runtime behavior. Realtime unit testing does not require any code instrumentation or any test driver. Without compiling/ linking/ download the test is executed on the real target hardware.
Line execution profiling can help to detect performance bottlenecks in a function's code.
To setup this feature select in the "Profiler Configuration" dialog the function name you like to profile and check the box "Include function lines" below the function list.
With that setting the profiler shows timeline measurements for every line of that function in the views "Timeline" and also in "Statistics".
The statistics criteria available are:
Count
Net Time time spent in the function only (without interrupts, without subroutines)
Gross Time time spent between function entry and exit, including subroutine calls, but without interrupts
Call Time time spent between function entry and exit including interrupts and subroutine calls
Since winIDEA 9.11. build 47 iSYSTEM's new analyzer supports now negative wildcards in profiler's Statistics and Timeline view. Negative wildcards are supported using the - prefix.
Press the filter button in the profiler menu to show the profiler filter dialog and specify a negative filter, e.g. " -OS* -Adc".
This will tell the profiler not to show functions that names start with "OS" and "Adc".
What is tail call merging (tail-call optimization or tail-call elimination)? Tail call merging is a compiler optimization used by almost all modern compilers. But many compilers provide options to suppress tail call merging. Without tail-call merge: When calling a subroutine the caller pushes its return address on the stack and hands control to the subroutine. The very last thing the subroutine does is a pop from stack to read the caller's address and return control to this address. With tail-call merge: When a subroutine call is the very last command to execute (in the caller's function), it´s save to jump to the start of subroutine directly without adding a stack frame and reusing the caller's stack. When the subroutine is done, it will return it's results directly to the caller. For more information see Wikipedia/Tail-call
What are the benefits of tail-call merging? The main benefits of tail-call merging are saving stack space which is very important on recursive function calls and saving execution time which is important for embedded applications where every microsecond might count.
How does tail-call merging affect a profiler? For the profiler tail-call merging looks like moving the subroutine's A code into the function body of the caller's function B. In Range Mode execution in function B would be attributed to function B, instead of the optimized function A. That would lead to incorrect time measurements in the profiler.
iSYSTEM's winIDEA has now a new option. If in Profiler/Configuration the option "Tail-merge analysis" is enabled, the profiler performs analysis of tail-call merge optimizations on the fly. This analysis ensures correct measurements in the profiler, but requires a higher level of debug information quality and relies on object code analysis. If the analysis algorithm fails, profiler session aborts. In such case the tail-call merge analysis can be disabled to revert to regular range mode. Another solution is to tell the compiler not to use tail-call merging.
Realtime unit testing runs test scripts and verifies your application without any influence on its runtime behavior.
Why is that important? Because most unit test tools on the market instrument the application's code and install a test driver through which resource utilization and task execution is changed. Additionally the instrumented code must be compiled, linked and downloaded to the target system for every test case.
The much better approach is realtime unit testing: Realtime unit testing does not require code instrumentation or any test driver. iSYSTEM has enhanced their debug and test tools to allow the generation and execution of test cases from inside the development environment. No test driver is necessary. Without compiling/linking/download the test is executed on the real target hardware.
iSYSTEM calls that realtime unit test because the test is executed on the real hardware without any effects on the runtime behavior.
How does realtime unit test work?
First you need a working hardware that is connected by an iSYSTEM debugger (Blue Box e.g. iC5000, iC3000) the PC. Second the software must basically work to initialize the hardware and run until main().
At that point testIDEA takes over. Based on a test specification testIDEA puts the parameters for that function test on the stack – that’s the only additional resource needed by testIDEA - , executes the function in realtime, compares the return values to the expected ones specified in the test specification and returns to main().
testIDEA is a free and open programming interface (API)that is completely integrated in winIDEA. With that API set one can write test applications/cases but also automatically execute them on the hardware connected by iSYSTEM tools. testIDEA also includes an interactive GUI for generating test cases and test reports.
iSYSTEM testIDEA professional version now offers a feature to write the test specification at the same time you are writing your application's code.
For those that do not know about testIDEA: testIDEA provides realtime Unit Testing by execution of test cases and test vectorson the real customer hardware without code instrumentation. Primarily testIDEA is a free and open programming interface (API)that is completely integrated in winIDEA. With that API set one can write test applications/cases but also automatically execute them on customer’s hardware connected by iSYSTEM tools. testIDEA alsoincludes an interactive editor for generating test cases and test reports.
Additional to above you can now also write down a test specification in your source code and testIDEA will extract that test spec and execute it. Assume you are a developer or code reviewer: Already when you write (or read) the source code, you think about function parameters and their limits. So, wouldn't it be a good idea to write down a test specification already at that point?
In general unit tests are tightly related to specific sections of the source code. Usually each test belongs to one function. Therefore it makes sense to keep test specifications in the same file - in the source code. To add a test specification, mark it with the special C/C++ comment string/*# which is a standard C/C++ multi-line comment followed by #. Below you see two functions plus test specificatons (different parameters transfers are used to show possible syntax):
int Func1(int i) { int x=0; for (;x<10;++x) ++i; return i; }
long Func2(int i,char c, long l) { int x=0; long y=0; for (;x<10;++x) y+=x+i+c+l; return y; }
When you open in testIDEA the source code file including the test specifications, testIDEA will parse the code and find above test specs. When you change the test specification in testIDEA, the change will be written back to the source code file into the comment lines.
More information (including the syntax) can be found in testIDEA's online help file.
iSYSTEM testIDEA allows now to create test specifications from templates.
For those that do not know about testIDEA: testIDEA providesrealtime Unit Testing by execution of test cases and test vectorson the real customer hardware without code instrumentation. Primarily testIDEA is a free and open programming interface (API)that is completely integrated in winIDEA. With that API set one can write test applications/cases but also automatically execute them on customer’s hardware connected by iSYSTEM tools. testIDEA alsoincludes an interactive editor for generating test cases and test reports.
If test specifications are similar, we can now create new test specifications based on a template. Template files are *.iyaml files created with testIDEA. To use them as templates, we add them to the list of files in the Templates dialog. The same dialog is also used for adding new test specifications based on templates to the current project in testIDEA. Select option File | New Test From Template..., and the following dialog opens:
This dialog can be used to edit the list of template files - we can have more of them, if needed. Test specifications from the selected file are shown in the bottom pane. To add a new test specifiation to the current project, select the test specification in the dialog and click OK.
If you want it to be added as a derived test specification of the currently selected test specification in testIDEA, check also the Add as derived testcheck box.
More information can be found here and in testIDEA's online help file.
IEC61508, ISO26262, DO178B/C, IEC62304, EN5012x - They all and highly recommend testing. Some also include the qualification of tools used for the development of a safety application. So it is time to discuss this with you! Please register for our roadshow in Germany starting September 2011: Download the agenda for Germany
According to Beizer, 100% statement coverage at source code level might cover 75% or less statements at object code level (reference [1], reference [2]). That means if you only use source code coverage you have NOT tested 1/4 of your application's code!
But where does that come from...?
Remember the blogpost on Code Coverage: Relation between Source and Object Code? Every time you use complex instructions (e.g. for), complex conditions (e.g. if) or library or operating system functions object code is added that is not directly traceable to source code statements. Sometimes also the compiler generates object code to bridge the gap between what is possible in source code and on the processor e.g. a conversion from floating point to integer.
What impact has this?
DO-178/ED-12B Section 6.4.4.2b states that if "... the compiler generates object code that is not directly traceable to Source Code statements. Then, additional verification should be performed on the object code..." So if an application has to conform to standards like above then the object code has to be covered too. To get an application approved following three steps must be addressed (cite from [reference 3]):
detect and identify any added object code
establish the functionality of that code; and
verify the correctness of the added, untraceable code sequences
How can you achieve object code coverage?
You can easily achieve object code coverage using iSYSTEM's on-chip debug tools and analyzers. iSYSTEM's tools measure code coverage by recording all addressesexecuted on the processor. Code instrumentation is not used for that. Code coverage results displayed in source code are correlations of the low level binary data (that was used for code coverage measurement) to high level source code. Any object code insertations by the compiler or by library functions are easily visible in iSYSTEM's code coverage analysis window.
Code coverage is a measure used in software testing. Code coverage helps to find code that is never executed. This “dead code” represents undesired overhead when assigning memory resources and also a potential hazard if this code was not executed during testing. You may already know statement coverage that marks code (statements/instructions) as executed or not and decision coverage (also known as Branch coverage) that marks conditional instructions as evaluated to true or false or both (at least once).
But what is now offline and realtime execution coverage?
Let's start withexecutioncoverage. iSYSTEM tools measure code coverage by recording all addresses being executed on the processor without any code instrumentation. Code coverage results displayed in source code are correlations of the binary low level data (that was used for measurement) to the high level source code. In iSYSTEM nomenclature code coverage is called execution coverage in contrast to data or access coverage.
As the name already says offline execution coverage uses recorded data and analyzes them. Offline analysis allows statement as well as decision coverage.
Statement Coverage: each processor instruction is marked as executed or not executed. By correlating this information upwards to the high level source code, executed and not executed lines of source code are identified.
Decision Coverage: conditional instructions are analyzed to identify if a condition has been evaluated or not. For this analysis the coverage session must be stopped (therefore “off-line” execution coverage). By cumulatively merging the information for each condition four different states are possible:
the condition has never been evaluated,
the condition has been evaluated to true,
the condition has been evaluated to false,
the condition has been evaluated at least once to true and at least once to false.
Realtime execution coverage is based on hardware logic in the analyzer and is available for all in-circuit emulation systems with bus trace. The tool monitors in real-time what program addresses are executed and which records are updated. Real-time execution coverage can run indefinitely. Realtime execution coverage allows only statement coverage because there is no history information, and without that no decision coverage is possible.
Starting from the source code, there is a chain of transformations that produce the object code that will run in the actual target. These modifications directly affect code coverage analysis because it is very complex to establish which object code correlates to what source code line.
For C programs usually these steps are: preprocessing, compilation, optimization, assembly, linking and conversion. Details vary between systems (compilers, assemblers, linkers, converters). These steps are not bijective: information is added and removed in every step.
The following examples should be well understood when interpreting source code coverage information measured by iSYSTEM tools:
Complex Instructions The object code may not correspond to the source code as seen from the perspective of a source level programmer. Take this example: 1 int i; 2 for (i=0; i<10;++i) 3 { 4 somefunction(i); 5 } Line 2 is typically (no matter of the processor archtecture used) split in two or three blocks: one to initialize the loop, another the check the condition and incremeent the loop variable and a tail to cleanup ... (click here to read on...)
Complex Conditions For example: 1 if (a==3 || b==4) 2 { 3 somefunction(c); 4 } Line 1 of the example can be only partially tested, even if it is reached from the program. If this condition is reached during execution and evaluation short-circuit logic applies, then in the source code window the coverage information will show the line as not executed. If you drill down the assembler information for this line you will see that only part of the code was executed... (click here to read on...)
Compiler Optimizations Compilers may merge several source code lines, deem some unnecssary and/or reorder the execution flow. iSYSTEM tools take great care to show accurately what is actually going on. In the source code view of coverage information you can tell at once if a line of code generates any object code at all. Each source code line that generates object code has next to it a lowered square that is, depending on the coverage information available, red (if code was not executed), green (if code was executed) or has an arrow indicating that there is a conditional instruction... (click here to read on...)
Libraries As iSYSTEM tools gather coverage information directly from the hardware, analysis of library code isn´t different to analysis from code written by the user. If source code is not available, the information is given on the object and assembler levels only,... (click here to read on...)
Compiler Generated Code This is expecially true, but not limited, for smaller processors. Sometimes the compiler needs to introduce code to bridge the gap between what is possible in source code and on the processor. This is mostly the case for operations that don't translate directly to machine code... (click here to read on...)
Compiler Bugs Seldom, but still, compiler can have bugs. These bugs are rare on the code that is generated, as compilers are usually tested in this respect. Most of the time bugs are in the debug information generated by the compiler and used by other tools. It is important to remark that these happen seldom, but shouldn´t be ruled out.
Have you ever wondered what "(1t,6f,0b)/8 (44%)" means in winIDEA's Code Coverage window in the Conditionals column?
We changed the abbreviations for decision coverage to increase the accuracy of decision coverage results shown. We now use the term "Conditionals" (instead of the previously used "Branches") because not all branches are conditional statements. Now what is a conditional statement? A condition usually results in branch and jump instructions in assembler that lead to code that is not always run through (because of the jump...). When we use the opposite for explanation: an UNconditional statement contains only opcodes that are always executed and do not depend on the CPU condition register.
Now what is acutally shown in winIDEA's Code Coverage window?
The "Lines bar" shows a graphical representation of the percentage of lines that were covered for a certain item. The "Lines" column displays the amount of lines covered, the total amount of lines and a percentage figure. If the Lines column doesn´t show anything but the "Sizes" and "Conditionals" column are not empty, then there is no source code available for the assembler code executed (which is usually the case for library functions, e.g. operating system functions).
The "Sizes Bar" displays graphically the percentage of bytes that is covered or not for a given item. The "Sizes" column displays the amount of covered bytes, the total number of bytes and the percentage of bytes covered.
The "Conditionals" column provides information on how many conditional instructions have been evaluated to true (t) or false (f) or in both ways (b). Both ways means that the condition has been evaluated at least once to true and at least once to false. The number after the slash (/x) tells the total number of conditional instructions found. The percentage figure represents how many of all possible conditional outcomes have been covered.
With iSYSTEM testIDEA you can write test applications/cases but also automatically execute them on your hardware connected by iSYSTEM tools.
In the last weeks we already published several videos on the creation and running of tests, on the usage of variables and stubs, on the inheritance of test specifications and on test report generation.
Today we like to explain how you can use function and data profiler from inside testIDEA to optimize your application's performance.
You can profile every function inside testIDEA and collect stastistics on the results. A profiler helps identifying performance bottlenecks. You can determine how long a function ran in relation to the overall recording time.
In our example we profile a function that is calling 4 subroutines.
For that we specify in the Test Specification Editor - Function the top level function name to test - in our example Address_DifferentFunctionParameters(). In Test Specification Editor- Profiler window select run mode "Start" and set the name for the profiler output file. You could use "Trigger name" to specify a trigger when to start profiling - in our example we do not use it because we start profiling when function Address_DifferentFunctionParameters() starts.
We recommend to set "Save history" to "Yes" to use winIDEA´s timeline view to show which function was interrupted at what time for how long and what other functions were called how many times.
In "Profiler" - "Code areas" window list all function names that should get profiled - in our example the toplevel function Address_DifferentFunctionParameters() plus all its subroutines Func1() to Func4().
For every function set time and hit limits. If any limit gets exceeded, the test will fail.
In "Profiler" - "Data areas" window list all variables that should get profiled and specify time and hit limits. Profiler will only collect statistics from the variables in this list.
If we run now this test, we see in testIDEA´s "Profiler"- "Code areas" and "Data areas" window the results for every function and every variable profiled.
winIDEA displays a chart of the profiling test results in winIDEA´s profiler window. In timeline view every function currently running is shown in bright red. Light red color means that a subroutine is executing. The data window shows when and how long the variable iCounter is set to the destinct value.
Switching to statistics view gives an overview on how much time was spent in a destinct function.
With iSYSTEM testIDEA you can write test applications/cases but also automatically execute them on your hardware connected by iSYSTEM tools.
In the last weeks we already published several videos on the creation and running of tests, on the usage of variables and code coverage, on the inheritance of test specifications and on test report generation.
Today we like to explain how you can exclude functions from testing by using stubs.
When a function under test calls another function which we do not want to include in the current test, we can specify a stub.
The stubbed function will not be called, but parameters and return values will be set according to test specification.In our example we test Func4() which calls Func1(). We do not like to execute Func1(), therefore we stub it.
First we write a test specification for Func4(). Parameters and return value are set as always.
To stub Func1() specify in Test Specification Editor - Stubs the function name Func1, as Parameters 10 and a return value x. In the Assignments section set a value for x, e.g. 20.
When you now run your test, Func1() will not be executed and its return value will always be 20.
With iSYSTEM testIDEA you can write test applications/cases but also automatically execute them on your hardware connected by iSYSTEM tools.
In the last weeks we already published several videos on the creation and running of tests, on the usage of variables, on the inheritance of test specifications and on test report generation.
Today we like to explain how to collect code coverage information while tests are executed.
To collect Code Coverage information when running tests inside testIDEA set first all function parameters in Test Specification Editor - Function window.
Then open Test Specification Editor - Coverage window. Select "Start" as the Run mode, enter a file name and the Open mode for the file. Details on the different options you find in the tooltips when moving the mouse pointer over the items.
In our test we like to save the coverage statistics after test - therefore we select yes - and we do not close the file after test.
A rangeset specifies the functions on which code coverage statistics are collected. In Rangeset Name specify the rangeset name used by winIDEA. If you do not specify a Rangeset Name, the default name isystem.test is used. The content of the rangeset is set in the "Coverage" - "Statistics" window by adding functions to the "Function" list. Code Coverage information will only be collected on the functions in "Function" list.
In our example we like to cover Func1() and Func4(), although the test is specified on the toplevel function Address_DifferentFunctionParameters().
In the "Coverage" - "Statistics" window you can specify statistics requirements.The test will fail if coverage results are lower than specified. If you don't want to verify coverage, but only to record it, simply set expected coverage to 0 %.
If you like to export Code Coverage statistics specify the export format and a file name. Additional to statement coverage you can collect "Decision coverage" results (also called branch coverage) by selecting "Yes".
When you now run this test you see in Test Specification Editor the actual Code Coverage Results.
winIDEA will leave the code coverage window open after test. You see that Code Coverage statistics have only been collected on Func1() and Func4() and you can check Code Coverage results for every line of code.
With iSYSTEM testIDEA you can write test applications/cases but also automatically execute them on your hardware connected by iSYSTEM tools.
In the last weeks we already published several videos on how to create and run tests in testIDEA, how to generate test reports to document the test process and its results and how to use variables to pass values to a function in test. Today we like to explain how the test tree is structured and how you can use inheritance to simplify your test setup.
Often tests are grouped based on a common property. To provide this functionality, tests can be organized hierarchically in a tree structure.
In the base test we specify all properties that should be common/the same for all derived tests.
In our example we use a base test, where we test function Func4() but we stub function Func1() to always return 20 and we also specify Code Coverage in the base test.
We now derive a test specification by right clicking that base test and selection of "New derived test". We assume that we like to test Func4() again, but with different parameters, e.g. -1,0,0.
The derived tests inherits the complete test specification from it´s base test - in our example it will also have Func1() stubbed and Code Coverage specified.
Lets run the first derived test and look at the test report. You can see in the test results that for the derived test also Func1() was stubbed with return value 20 and that Code Coverage is set.
If a derived test specifies a property that is also specified in the parent base test, that property is NOT inherited, meaning that the derived test specification precedes.
Lets look at the second derived test. It has a Stub of its own which returns for Func1() 60. When we run this derived test, we can see in the Test Results that it uses as return value for Func1() 60, but still has Code Coverage inherited from its parent base test.
With iSYSTEM testIDEA you can write test applications/cases but also automatically execute them on your hardware connected by iSYSTEM tools.
In the last weeks we already published three videos that explain how to create and run tests in testIDEA and how to generate test reports to document the test process and its results. Today we like to show you how to use variables to pass values to a function that is tested.
Variables are created on the stack before test execution. They may be used as function parameters, when pointers or structures are needed or when you need parameters in your scripts. E.g. when you use testIDEA to create a test script in YAML and later expand this script to pass thousands of values to automatically test the function, you use TSE - Variables for the parameters in the script.
When we test in testIDEA a function that uses pointers or structures, you need to initialize those or your target will crash. Although the references could point to global variables, locals are usually preferred because they disappear after test and have no side effects.
To define a test for the function shown (with following definition: int move(int *mode, struct SCoordinate coord);)you specify in Test Specification Editor - Variables a structure and an integer with values initialized, e.g. structure type name is SCoordinate, thevariable name is Coordinate, structure members areCoordinate.x = 20, Coordinate.y=40 . The name of integer variable is mode with value 3.
To pass those variables to above function, you specify in Test Specification Editor/ Function in Params section: &mode, Coordinate
When we now run above test, the expected result in our test is the sum of Coordinate.x, Coordinate.y and mode, with above example values it will return 63.
iSYSTEM´s testIDEA can save test results to XML or YAML files.
To save test results, select menu "File" - "Save Test Report". Specify if you like to write a "Full report" or "Errors only" to the output file. Set the Output format.
In our example we create a full report in XML format.
Enter the output file name. To view test results in a web browser as HTML and print them, iSYSTEM´s testIDEA provides XML Style sheet templates - so called XSLT files. You can use one of the built-in XSLT templates or create a template of your own.
In our example we use the built-in itestResult.blue template and we also "include test specifications" in the output file. To automatically "open the default browser after save", we check this box. In Test Environment you can briefly describe what equipment you used for testing.
On OK the report is saved. Due to the check box "Open default browser after save" the default browser for XML files opens and shows the report.
In the Test Results file you find general test configuration information e.g. tester´s name, testSpecificationFile, winIDEAWorkspacePath plus every test with its name, specification and result.
In our demonstration we use again the sample workspace for the MPC5554 evaluation board included in winIDEA examples directory. The Blue Box used in an iTRACE GT.
When there is at least one test specification in testIDEA´s Test Tree, you can run
either all tests or
only the selected one or
the selected one and its derived tests.
To specify the different test types use
the main menu option "Run" and select the appropiate test or
use a toolbar icon.
In our example we select "Run all" to run all tests specified.
When the tests complete, iSYSTEM´s testIDEA marks each test and section with a marker. If a test passes with OK, a green check mark is shown, otherwise a red mark with cross inside is visible.
On all tests marked red you should check why the test failed.
With iSYSTEM testIDEA you can write test applications/cases but also automatically execute them on your hardware connected by iSYSTEM tools. iSYSTEM testIDEA is a free and open programming interface (API) that is completely integrated in winIDEA.Test applications, test cases and the accordant test reports can be written in many different programming and scripting languages (Python, Java, C/C++, C#, Perl, TCL…). testIDEA also includes an interactive editor for generating test cases and test reports.
To see how to specify your first test in testIDEA, run below video.
In our demonstration we use a sample workspace included in winIDEA examples directory for an MPC5554 evaluation board. The Blue Box used is an iTRACE GT.
To launch testIDEA use winIDEA´s "Test" menu and select "Launch testIDEA".
Verify that the connection to your target is working. To do so use menu "Run" - "Init Target".
You can see in the background winIDEA´s download window which shows that testIDEA is calling winIDEA.
This initialization also checks that testIDEA is connected to winIDEA - you should see a green CONNECTED in the bottom left corner - and it reviews that the workspace path in testIDEA and the workspace path in winIDEA fit. You can adjust testIDEA´s workspace path in "Tools" - "Options..." - "Workspace Path".
Most controls on the the initial screen are grayed out, which means you cannot enter any data until the first test case is created.
Create test cases either with the main menu "File" - "New base test" or with right mouse click in the Test Tree window and select "New base test".
The New test dialog opens. Use Refresh to retrieve a list of all functions available in the download file. Select the Function you like to get tested.
Type in values for the Parameters. If you can´t recall the function definition press Source and winIDEA´s document window will switch and show you the function you like to test.
If pointers are passed use variables in the test specification editor. How to use variables see in a later section.
In Verification specify an expression for the expected return value, e.g. rv==20 or you include global variables, e.g. rv==iCounter+20.
Ret.val. name is the variable name containing the return value after test.
To save your test specification press OK.
To test that function use "Run Selected".
More information on testIDEA and how to install it you find on iSYSTEM´s web page.
With iSYSTEM´s Blue Boxes and winIDEA you have the potential to run many test series and to accumulate the coverage results in one code coverage file.
E.g. developer A sits in Germany and tests module A. The results of this session are stored in a *.CCV file. After this test series the embedded system plus the *.CCV file are sent to developer B in France. B runs different tests because B has all the external hardware.
To prove that all code is covered developer B must continue his tests based on the *.CCV file of A. To do so B will load A´s *.CCV file into winIDEA and use the "Continue" button for his own tests.
This way B´s coverage results will be written into the same *.CCV file to document which functions have been covered.
This document can be used for certification purposes for safety critical systems (e.g. DO-178B, ISO 13485:2003, SC45A).
Today iSYSTEM AG released the new iSYSTEM testIDEA for embedded software tests. iSYSTEM testIDEA expands iSYSTEM’s debug and trace tools for software development and allows the execution of test cases and test vectors on the real customer hardware without code instrumentation – other tools usually instrument the code.
Primarily testIDEA is a free and open programming interface (API) that is completely integrated in winIDEA. With that API set one can write test applications/cases but also automatically execute them on customer’s hardware connected by iSYSTEM tools. Test applications, test cases and the accordant test reports can be written in many different programming and scripting languages (Python, Java, C/C++, C#, Perl, TCL…).
testIDEA also includes an interactive editor for generating test cases and test reports that can be directly called from winIDEA. With that GUI test cases can be created without any programming know-how. testIDEA stores the test cases as YAML (YAML Ain‘t Markup Language) files which is a description language similar to XML, but easier to read and to process in other languages. E.g. YAML files can be included in Python exactly as they are and therefore they are reusable in an executable mode.
The major advantage of non-instrumented unit tests is the fast turn-around-times because the tests can run without compilation, linking and download. In addition no test drivers are needed and therefore no additional resources are used on the target system. Functions like tracing, profiling and code coverage analysis can be integrated into the test cases. This expands the operation area of iSYSTEM test technology from unit testing to integration and system testing.
A paper that describes a basic workplace setup of an embedded software developer in the year 2011. It points out today’s challenges arising from low level functionality design as well as different ways of testing such software. It includes the aspect of test automation, how to test IO drivers and how to do functional testing (black box and white box, which is code coverage) within a developer's workplace [...]
Technical paper about the pros and cons of on-chip debug interfaces provided on a microcontroller today. What is the quality of data one gets out of such small interfaces? Are there design issues I do have to consider from the very beginning? What is the quality of trace, profiling and code coverage via on-chip debug interfaces? Is this possible at all?
We just updated our slideshare account. There you will find the latest powerpoint presentations about iSYSTEM Blue Box Technology for embedded development and test.
Most of the embedded development and test tools available in the embedded market today do include a profiler functionality. However, choosing a development and test tool that supports profiling, you should add additional check boxes according to your project requirements. This paper will help you to identify those. Beside that it is a nice document to learn about a profiler in general.
Extract of the content: What is a profiler?, What is an event? What is data and execution profiling? How does the debug interface of a MCU affects my profiling results? What is task profiling? Overview of MCU trace capabilities ...
You may now download libraries, examples and documentation for C# and Perl. Just extract the zip files into the SDK folder of your winIDEA installation.
The extensive (script) language support within winIDEA allows the users to write their own scripts and/or applications to remote control iSYSTEM tools. Remote controlling means actively steering or acquiring data from a target system. This
could be an Excel script filling a table of temperature readouts over time, a
production stage application which programs the microcontroller FLASH and
verifies RAM, application which, through data acquisition, realizes unit test,
a hardware-in-the-loop system test, etc.
At Embedded World 2010 Rapita Systems showed iSYSTEM's new hardware platform iC5000 together with their performance measurement and timing analysis tool RapiTime.
However, we were wondering why they put our competition on the same level ;-).
iSYSTEM finished its first series of workshops in south Germany.
"It was not just a workshop. We were able to address the right topics at the right time: Test automation with isystem.connect and some advanced basics of the use of high-end debug features such as tracing, profiling and coverage. The workshop showed that iSYSTEM is very well positioned along the complete embedded software development and test process and ready for future growth", said Erol Simsek Director Sales & Marketing of iSYSTEM AG.
Learn more about isystem.connect and download the presentations of our developers Marko and Primoz from our slideshare account:
isystem.connect for developers: isystem.connect is iSYSTEM´s generic API and basis to connect to winIDEA, to get full control over debugging and much more
isystem.test for testers: builds on isystem.connect and adds powerful unit testing capabilities
isystem.connect goes "mini"-HIL: builds on isystem.connect and adds control over external hardware
sepp.med
GmbH, iSYSTEM AG and the FZI (ResearchCenter for Information Technology in Karlsruhe) cooperate to define and enhance methods and tools for testing embedded
software. The project titled eMOTE ("*E*mbedded *Mo*del-based
*Te*sting") is funded by the BMWi (Federal Ministry of
Economics and Technology) and will last for two years. More ...
Yesterday the first Blue Box Workshop took place in Schwabhausen. The workshop was fully booked and it was a great success!
Topics of the workshop were
"Development and Test Process Automation using isystem.connect"
"Debug Interfaces of different MCU Architectures and their influence on Tracing, Profiling and Code Coverage"
Workshop participants were impressed by the large field of application offered by the API set isystem.connect. Not only test processes can be easily automated using standard scripting languages like Python, C++, Java..., the API set also eases unit tests by providing a wizard for writing the tests.
The second workshop part revealed the dependencies between debug options offered by the MCU itself and their influence on the debugging functionalities available to a developer in the debugging tool. Several participants called it a revelation to see in life demos how the amount and quality of debug and trace information changes between different MCUs.
Next workshop is planned in Stuttgart on 25th February 2010. And more workshops will follow all over Germany! Please check our event calender!
iSYSTEM’s winIDEA fully supports Python 2.6 since version 9.10.31. This allows developers to run Python scripts inside winIDEA and also to call winIDEA functions from Python scripts running at command line. In a debug and test environment this adds powerful debug and unit test capabilities and control over external hardware while running tests from script, e.g. when running HIL tests (Hardware In the Loop).
WinIDEA is the integrated development environment designed by iSYSTEM. WinIDEA contains all tools necessary for embedded software development (e.g. edit, build, debug and test) in one shell. Additionally WinIDEA interfaces and drives all Blue Boxes – the various On-Chip debuggers and In-circuit emulators manufactured by iSYSTEM.
Python is a popular scripting language with extensive programming capabilities available on Windows, Mac OS X and Unix operating systems and is much more flexible than e.g. shell scripts or batch files.
Today iSYSTEM AG based in Schwabhausen announces their new generation of development and test tools for embedded software and hardware.
The platform called iC5000 On-Chip Analyzer is the consequent product enhancement of well proven iSYSTEM technolgy into one hardware platform. iC5000 is a single hardware and software platform that is changed by software updates into a multifunctional development and test tool to support various processors and controllers.
The hardware of the iC5000 consists of a base module including USB2.0 and ETH100 for connection to the PC. On top of the base module the debug/trace module and an optional I/O module is located.
The debug/trace module connects to different debug interfaces (JTAG, BDM, SWD, ETM, NEXUS…).For that various adapters are available.
The I/O module controls measurement and stimulation tests.With it analog and digital signals can be generated, monitored and traced.
At the moment there are software licenses to support following controllers: ARM7, ARM9, XScale,CortexM3, Freescale Coldfire V1, Coldfire V2/V3/V4, MC9(S)08 and MC9S12(X). Others will follow in 2010. By purchase of a trace license the platform is expanded from a simple debugger to very powerful test tool with up to 256MB trace memory.
Here are the results of iSYSTEMs poll: 66,7% answered "yes, every developer should do unit testing". 25,0% said "no". 8,3% don´t know... This result reflects a trend for embedded software development.
Most of the developers know about the benefits of unit testing: - test as much and as early as possible in the development process. The earlier the cheaper to fix a bug. - test a single function without having implemented/or having access to all code - proove that a function behaves according to the specification/requirements - proove that there is no dead code (code coverage) - build a certain set of tests (test know-how) for regression testing if the function code has changed - ...
However, most of them don't do it because: - in most cases an additional tool is necessary - it is very time consuming because of additional compile-link-download-execute cycles - traditional unit test tools instrument the code to * generate a test harness for a single function * measure code coverage (e.g., to measure the execution of every single statement and/or decision) - it is not the orginal code - ...
So, this rises a big challenge for tool vendors such as iSYSTEM. In Q3/2009 iSYSTEM will publish a new version of isystem.connect with an add-on for unit testing. Stay tuned!
In the year 2010 an embedded software development and test workplace could look like this: In addition to a software development tool that would be a debugger with or without trace support, a stimulation hardware for the target system is used (e.g., sbRIO from National Instruments as a mini-HIL).
Thus, a developer may drive a lot of function tests already while developing software. All is done in a single tool environment that is as close as possible to the real hardware (and real-time) as well as in an early stage of the development and test process.
The connection between NI's sbRIO platform to iSYSTEM tools is realised using iSYSTEM's open API isystem.connect. This generic interface also includes a free LabVIEW VI library to drive the described test approach from within LabVIEW.
This week, October 1, 2009 the technology partners GÖPEL and iSYSTEM offer a free workshop: "Boundary Scan meets Microcontroller Emulation".
To cope with challenges of future hardware components tests, boundary scan and JTAG emulation must be combined. Boundary Scan and JTAG emulation are two methods that complement each other perfectly.
The workshop takes place October 1, 2009 from 10:00am to 4:00pm at iSYSTEM AG, Carl-Zeiss-Str. 1, 85247 Schwabhausen (close to Munich).
For detailed information on the workshop see this whitepaper.
If you like to participate in this Workshop, please contact us
iSYSTEM developed a stand-alone on-chip emulator solution for the so called Pictus and Bolero Controllers of Freescale/STMicroelectronic. Everything is integrated in one box in order to avoid too many adapters/connectors between a target and the development tool (see picture below). The blue box includes a proper emulation device (208BGA Bolero, 144QFP Pictus). Only these devices do have a NEXUS debug and trace interface. Development and test may be done with and/or without a target system. Due to the use of the internal clock of the on-chip emulator, high frequencies are no challenge anymore. An optional measurement board enables developers to measure or set every single pin of the controller.
Hardware manual and supported devices: ActiveGT POD
The question "Should a developer do Unit Testing?" is not meant as a rhetorical question.
Acting on impulse most people would answer immediately "yes".
But when you thoroughly think about the question what would it mean for the development process itself, for development times and also for costs? Wouldn´t development time and costs dramatically increase when every developer accomplishes unit tests? Would the development process change by testing procedures involved in early stages? Will testers be necessary or new software that creates automated test cases? On the other hand how does a failing product affect customer/market acceptance? And what legal liabilities are tied to a product?
Now, what is unit testing? Wikipedia states following: "In computer programming, unit testing is a software verification and validation method in which a programmer tests that individual units of source code are fit for use. A unit is the smallest testable part of an application." [5]
Articles discussing the pro's and contra's and hands-on experiences on unit testing show different aspects [1],[2],[3]. Legal liabilities that arise from standards e.g. IEEE Std 1012-1998, IEEE Std 829-1998 and the resulting requirements on software tests are discussed in [4].
Now what do you think? Do you think that every developer should do unit testing? Feel free to comment your opinion using the comment link below this blog post.
