Reviewing code coverage reports and dashboards

This lab guides you how to review code coverage reports and dashboards as Tammy, the test manager.

Complete the following steps to examine the code coverage reports and dashboards.

Navigate your browser to https://healthcare.example.com:9443/adi/web/projects/HealthCare4All.
Log in as the tester manager with Tammy as user ID and Tammy as password. The Workbook page appears. For this tutorial, you will see the code coverage report for Perfect Calculator and SAM Application.

You can see that both applications have the Aggregated Code Coverage over 70% but Perfect Calculator is in warning state while SAM Application is in normal state. The code coverage report is a roll-up percentage of code coverage from all builds within one workbook. ADI allows users to define the code coverage threshold for a workbook to customize based on the quality level that application required.

For example in this lab, the code coverage for Perfect Calculator which is lower than 80% will be shown in yellow to indicate the warning status and the code coverage which is lower than 50% will be shown in red to indicate that it is not in acceptable number. For the SAM Application, the Aggregated Code Coverage which is lower than 70% will be shown in yellow.

These threshold values can be set when you create the workbook. See Setting up Manual Builds for Code Coverage Analysis how to set up the threshold values.
Click the name of Perfect Calculator card to deep dive to the code coverage reports of Perfect Calculator workbook. The summary charts for Perfect Calculator loads.

Currently there is only one build, Perfect Calculator Manual Builds for the Perfect Calculator workbook. The reports show the code coverage analysis for the latest build of Perfect Calculator Manual Builds.

The first report shows that the Perfect Calculator has 75% Code Coverage which is in the warning area. Looking at the next report, the Code Coverage Percentage Trend is starting to go down from Nov 12. On the next report, the number of total executable lines in the next report is going up from 64 to 76 on 11/13/2015 and then to 87 on 11/17/2015. However, the executable lines hit remains quite stable. There seems to be some code coverage issue with the last two builds of the Perfect Calculator Manual Build.

The Coverage Changed vs Unchanged Code report indicates that there are fifteen executable lines changed within a build and none of them have been tested. This means the changed code has not been tested properly.
Click on the name of Perfect Calculator Manual Build: 2015Nov17Build to drill down further.
The Files Analysis view appears. By default, this view shows the files that the code coverage is below the insufficient threshold value for the current build, which is 2015Nov17Build. You can immediately see the files that have code coverage issues from this view.

On the top row of this view, you can see the summary of the entire build: code coverage percentage, number of modified files, number of new files, number of historical tests files ran against this build, and number of minimal tests to run.

You can see the same information as in the Workbook view that the code coverage drops by almost 11%. There is one file updated for this build.
Below the top row, you can view the detail of file level code coverage. For each file, you can see the following information.
- Name of the file
- Warning status
- Code coverage percentage of the file
- Percentage of code coverage being changed
- Number of executable lines that are tested
- Number of total executable lines
- Number of executable lines added
- Number of executable lines updated
- Percentage of the executable lines that are added and updated
- Number of executable lines deleted
- Number of Historical Tests hitting the file
- Number of Minimal Tests to run
Note: The percentage of executable line changes is the percentage of executable lines being changed with in a build calculated from number of total executable lines within a build and number of lines added and updated within a build. The executable lines represent the number of executable lines for a source file or program. The executable line is defined as the line of code that the compiler marks as executable. For COBOL, the executable line may not directly correspond to the exact source line, as COBOL is not debugged by using source but rather the expanded source.
You can notice that CLCTPCTB.cbl status is in red warning and the percentage of code coverage change goes down by 18%. From the total of 39 executable lines, there are 11 executable lines added and 4 lines updated to the build. You suspect that these added and updated executable lines that didn't get tested cause the code coverage of the file to go down 18%.
Click the Expand icon in front of CLCTPCTB.cbl name to expand the flowpoints analysis section. The flowpoints analysis section displays 3 flowpoints but there are total of 5 flowpoints within the file.
Click Show All 5 Flowpoints on the bottom of the expanded section to see the code coverage detail of all flowpoints within the file. The Flowpoints Analysis view appears.

Similar to the Files Analysis view, this view shows the code coverage detail for all flowpoints within a file. You can see that the code coverage of COMBINE-AND-WRITE is showing 0% and 15 executable lines from the total of 16 executable lines are either added or updated. This confirms Tammy suspicion that the executable lines that are modified within the CLCTPCTB.cbl are not tested.
Select the Filters dropdown list on the top menu and check Changed Flowpoints Only option to filter the view to show only the flowpoints that are modified.
The view updates with only COMBINE-AND-WRITE flowpoint which confirms that this is the only flowpoint that is modified and it has not yet been tested.
Click the Back icon on the top menu to go back to the Build Analysis view for 2015Nov17Build page.
Click the Close icon (X) in the filter dialog box to clear the filter.
The view updates to show all files. In this case it shows 2 source files: CLCTPCTB.cbl and PERFCALC.cbl.
Click the Build dropdown list on the top menu and select 2015Nov13Build to go to the build on Nov 13, 2015.

On the page for 2015Nov13Build, you can observe that CLCTPCTB.cbl status is also in warning status for this build and the percentage of code coverage change goes down by 36%. 10 executable lines are added. There is still an issue with code coverage for this build as well.
Click the Build dropdown list again and select 2015Nov12Build to go to build on Nov 12, 2015.
On the page for 2015Nov12Build, the code coverage for overall build indicates 100% code coverage. This build has been fully tested. The issues seem to be arisen starting in 2015Nov13Build. Tammy wants to compare the builds to get a better understanding of what has changed.
Click the Compare Builds on the top menu.

The Compare Builds page appears. By default, 2015Nov12Build is preselected.
Select 2015Nov13Build and 2015Nov17Build.
Click Compare to compare the three builds.

On the Compare Build view, you can compare the detail of three builds in one view. This view helps confirm that your observation earlier that file CLCTPCTB.cbl has brought the code coverage percentage of the entire build down from 100% to 86% and 75%. The cause could be that the executable lines that are added to the CLCTPCTB.clb in the past 2 builds have not been properly tested. You can also see that the code coverage percentage of PERFCALC.cbl also drops 2% on 2015Nov13Build. There are 2 execution lines added but it seems not tested either.
Click the Expand icon in front of CLCTPCTB.cbl from all 3 builds to show the flowpoints analysis.

You can see that for 2015Nov13Build, P-INPUT-AZUE0000 and P-OUTPUT-AZUE0000 are newly added to the file but they have not been tested. P-INPUT-AZUE0000 is removed from 2015Nov17Build and COMBINE-AND-WRITE is added to the build. Again the new flowpoint has not been tested either. These 2 flowpoints are areas that require testing.

Tammy is now satisfied with her investigation on the code coverage issue for Perfect Calculator application. She communicates the results to her team and asks Trish to update the tests to cover P-OUT-AZUE0000 and COMBINE-AND-WRITE flowpoints. She also suggested her team to use ADI to always check the code coverage reports to ensure that the application is fully tested.

Now you have explored how you can use the test analysis results of the latest build to investigate the testing issues of your build. Typically you can use this information to perform the following analysis.

Determine if the coverage has improved from the previous builds.
Determine if the changed files were indeed covered during the testing.
Compare the results of the current build with one or more older builds to see differences between coverage results of these builds.
Determine how much the set of files have actually changed from build to build.
Review the trends of changes and coverage over time to determine the quality of the tests performed.