Reviewing code coverage reports and dashboards

This lab guides you how to review code coverage reports and dashboards as Tammy, the test manager.

Complete the following steps to examine the code coverage reports and dashboards.
  1. Navigate your browser to https://healthcare.example.com:9753/adi/web/projects/HealthCare4All.
  2. Log in as the tester manager with Tammy as the user ID and Tammy as the password. The Workbook page appears. For this tutorial, you will see the code coverage report for Perfect Calculator and SAM Application.
    Figure shows the page of Workbooks.

    You can see that both applications have the Aggregated Code Coverage over 70% but Perfect Calculator is in warning state while SAM Application is in normal state. The code coverage report is a roll-up percentage of code coverage from all builds within one workbook. ADI allows users to define the code coverage threshold for a workbook to customize based on the quality level that application required.

    For example in this lab, the code coverage for Perfect Calculator which is lower than 80% will be shown in yellow to indicate the warning status and the code coverage which is lower than 50% will be shown in red to indicate that it is not in acceptable number. For the SAM Application, the Aggregated Code Coverage which is lower than 70% will be shown in yellow.

    These threshold values can be set when you create the workbook. See Setting up Manual Builds for Code Coverage Analysis how to set up the threshold values.

  3. Click the name of Perfect Calculator card to deep dive to the code coverage reports of Perfect Calculator workbook. The summary charts for Perfect Calculator loads.
    Figure shows the summary charts for Perfect Calculator workbook.

    The reports show the code coverage analysis for the latest build of Perfect Calculator Manual Builds. The first report shows that the Perfect Calculator has 75% Code Coverage which is in the warning area. The code coverage is dropped from the previous build by 10.813%. From the next report, you can see that 23 executable lines of code are modified on this build but only 1 line is tested. There seems to be some issues here. The cause could be that the modified code has not been tested properly.

    Under the Trends section, the Code Coverage Percentage Trend is starting to go down from Nov 12. On the next report, the number of total executable lines in the next report is going up from 64 to 76 on 11/13/2015 and then to 87 on 11/17/2015. However, the executable lines hit remains quite stable. The issue seems to happen since the previous build.

  4. Click View Files to drill down further.
    Figure shows the location of View Files icon.
  5. On the top row of the Build Analysis view that appears, you can see the summary of the entire build: code coverage percentage, number of modified files, number of new files, number of historical tests files ran against this build, and number of minimal tests to run. You can see the same information as in the Workbook view that the code coverage drops by almost 11%.

    You can notice that CLCTPCTB.cbl status is in red warning and the percentage of code coverage change goes down by 18.286%. From the total of 39 executable lines, there are 11 executable lines added and 4 lines updated to the build. You suspect that these added and updated executable lines that didn't get tested causes the code coverage of the file to go down by 18.286%.

    Figure shows the Build Analysis view.
    Note: The "Change%" of executable lines is the percentage of executable lines that are changed with in a build, which is calculated from the number of total executable lines within a build and the number of lines that are added and updated within a build. The executable lines represent the number of executable lines for a source file or program. The executable line is defined as the line of code that the compiler marks as executable. For COBOL, the executable line may not directly correspond to the exact source line, as COBOL is not debugged by using source but rather the expanded source.
  6. Click the Expand icon in front of CLCTPCTB.cbl name to expand the flowpoints analysis section. The flowpoints analysis section displays 3 flowpoints but there are total of 5 flowpoints within the file.
    Figure shows the expanded flowpoints analysis section.
  7. Click Show All 5 Flowpoints on the bottom of the expanded section to see the code coverage details of all flowpoints within the file. The Flowpoints Analysis view appears.
    Figure shows the Flowpoints Analysis view.

    Similar to the Files Analysis view, this view shows the code coverage details for all flowpoints within a file. You can see that the code coverage of COMBINE-AND-WRITE is showing 0% and 15 executable lines from the total of 16 executable lines are either added or updated. This confirms Tammy suspicion that the executable lines that are modified within the CLCTPCTB.cbl are not tested.

  8. Select the Filters dropdown list on the top menu and check Changed Flowpoints Only option to filter the view to show only the flowpoints that are modified.
    Figure shows the drop-down box of Filters and the Changed Flowpoints Only option that is checked.
  9. The view updates with only COMBINE-AND-WRITE flowpoint which confirms that this is the only flowpoint that is modified and it has not yet been tested.
    Figure shows the Flowpoint Analysis view with changed flowpoints only.
  10. Click the Back icon on the top menu to go back to the Build Analysis view for 2015Nov17Build page.
  11. Click the Build dropdown list on the top menu and select 2015Nov13Build to go to the build on Nov 13, 2015. You suspect that the code coverage issue started from this build..
    Figure shows the drop-down list of Build.
    On the page for 2015Nov13Build, you can see that the overall code coverage is dropped by 14.5% comparing with the previous build. For 2015Nov13Build, the CLCTPCTB.cbl status is also in warning status for this build and the percentage of code coverage change goes down by 36% comparing with the previous build. 10 executable lines are added to be build. Again, you suspect that these 10 added lines probably weren't tested well.
    Figure shows the Build Analysis view for Perfect Calculator.
  12. Click the Build dropdown list again and select 2015Nov12Build to go to the build on Nov 12, 2015.

    On the page for 2015Nov12Build, the code coverage for overall build indicates 100% code coverage. This build has been fully tested. The issues seem to be arisen starting in 2015Nov13Build. Tammy wants to compare the builds to get a better understanding of what has changed.

  13. Click the File Trends on the top menu.
    Figure shows the location of Files Trends icon.

    The File Trends page appears. By default, 2015Nov12Build is preselected.

  14. Select 2015Nov13Build and 2015Nov17Build.
    Figure shows the page of File Trends.
  15. Click Compare to compare the three builds. The page of code coverage details for three builds appears.

    On this page, you can compare the details of three builds in one view. This view helps confirm that your observation earlier that file CLCTPCTB.cbl has brought the code coverage percentage of the entire build down from 100% to 86% and 75%. The cause could be that the executable lines that are added to the CLCTPCTB.clb in the past 2 builds have not been properly tested. You can also see that the code coverage percentage of PERFCALC.cbl also drops 2% on 2015Nov13Build. There are 2 execution lines added but it seems not tested either.

    Figure shows the page of code coverage details for three builds for comparison.
  16. Click the Expand icon in front of CLCTPCTB.cbl from all three builds to show the flowpoints analysis.
    Figure shows the expanded flowpoints analysis for CLCTPCTB.cbl from all three builds.

    You can see that for 2015Nov13Build, P-INPUT-AZUE0000 and P-OUTPUT-AZUE0000 are newly added to the file but they have not been tested. P-INPUT-AZUE0000 is removed from 2015Nov17Build and COMBINE-AND-WRITE is added to the build. Again the new flowpoint has not been tested either. These 2 flowpoints are areas that require testing.

    Now as Tammy, you are satisfied with your investigation on the code coverage issue for Perfect Calculator application. You communicates the results to your team and ask Trish to update the tests to cover P-OUT-AZUE0000 and COMBINE-AND-WRITE flowpoints.

    Next, you learn that 2015Nov12Build is one of the milestone builds and it is fully tested. You want to make sure that in the future your team can use this build as baseline for comparing the code coverage results of the next builds.

  17. Open the menu pane and select Data Providers to go to Data Providers page.
    Figure shows the Data Provider link on the menu pane.
  18. On the Perfect Calculator Manual Builds data provider, click the Options Menu icon and then select View Builds. The Builds page appears.
    Figure shows the options menu for Perfect Calculator Manual Builds data provider.
  19. Select the Star icon in front of the 2015Nov12Build. The Star icon in front of a build name is the indication of a baseline build.
    Figure shows the page of Builds on which the Star icon before 2015Nov12Build is selected.
  20. Click Save baseline builds to save your changes.
  21. Click Start data collection on the window for ADI to perform the data collection of the latest build and generate the code coverage reports against the new baseline build. When the data collection is completed, you will see a new notification message on the Notifications icon.
  22. Click Notifications icon to view the data collection status.
    Figure shows the notification message that the data collection is finished.
  23. Click the Menu icon next to the IBM® ADI header to open the menu pane.
  24. Select Perfect Calculator workbook to view the code coverage reports of Perfect Calculator.
  25. On the header of the Charts section, select Compare With drop-down menu. By default, the code coverage information, such as the percentage of code coverage changes and the percentage of executable line changes, is calculated against the previous build. After you define one or more baseline builds, ADI also calculates these information against the baseline builds that you define.
    Figure shows the drop-down menu of Compare With.
  26. Select Most Recent Baseline Build. With the reports that load, you can see the updated information, such as the percentage of code coverage changes is updated from 10.81% to 25.287% and the code coverage of CLCTPCTB.cbl is dropped by 54%.
    Under the Trends section, you can see the baseline build is marked on the trends line as well.
    Figure shows the summary charts for Perfect Calculator Analysis workbook.
  27. Select View Files icon on the upper right to go to the Build Analysis view.
  28. Observe the information against the most recent baseline build on the Build Analysis page.
    Figure shows the page of Build Analysis

Now you are satisfied with your investigation. You suggest your team using ADI to always check the code coverage reports and check against the baseline build to ensure that the application is fully tested.

Now you have explored how you can use the test analysis results of the latest build to investigate the testing issues of your build. Typically you can use this information to perform the following analysis.
Parent topic: Analyzing code coverage