Validating TRS feeds and the LQE index

Validate tracked resource set (TRS) feeds and the Lifecycle Query Engine (LQE) index when Report Builder reports don’t show the artifacts that you expect. Validating these items can help you diagnose and troubleshoot problems and avoid reindexing data sources.

About this task

You might validate a TRS feed and the LQE index in these situations:
  • A report is missing artifacts.
  • A report includes outdated (stale) artifacts.
  • A report shows the same artifact more than once, possibly with different attribute values or with its old and new name.
  • An application view and a corresponding report in Jazz® Reporting Service show a different number of artifacts.
  • A report includes deleted artifacts.
  • LQE shows skipped resources.
Validating TRS feeds and the LQE index can increase your confidence in the quality of your reports and help you troubleshoot missing, stale or extra artifacts:
  • Validating a TRS feed compares the current TRS feed to the data in the application repository.
  • Validating the LQE index compares the current TRS feed from a Rational® solution for Collaborative Lifecycle Management (CLM) application with the LQE index.

Validate these items when server usage is low, such as during a scheduled maintenance window.

For more information, go to this Jazz.net article.

The steps to validate TRS feeds are different across CLM applications. Complete the steps for the application whose feed you want to validate.

For a description of TRS feeds in CLM, see Adding Tracked Resource Set feeds for lifecycle management applications.

Jazz Team Server, Rational Team Concert, Rational Quality Manager, and Global Configuration Management

Use the TRS Feed Validation page in the administration section to validate TRS feeds.

Before you begin

For Rational Quality Manager:
  • When you run validation, you must have at least 1 GB of free disk space for each set of 1 million test artifacts in your repository (including planning and execution artifacts, such as test plans, test cases, test scripts, test script steps, test execution records, test execution results, test execution result steps, test suites, test suite execution records, and test suite execution results).
  • If you want the validation tool to automatically fix any discrepancies, go to the Advanced Properties administration page at https://host:port/qm/admin, and change the Automatically resolve discrepancies option for the com.ibm.rqm.oslc.trs.validator.TRSValidationWatchdog property to true.

    Ensure that you configure this advanced property before you start the validation process.

    If you don't configure this property, you can compare and resolve discrepancies by using the cleanDifferences option in the TRS Validator.

About this task

Examples of TRS feeds:
  • JTS User Resources: This feed contains information about Jazz Team Server (JTS) users. If a report about system users isn't showing the expected results, and you confirm that 10 users are registered in the repository, the validation task checks whether the TRS feed has 10 distinct references to users.
  • RTC Process Resources: If a report doesn’t show the project areas, team areas, or iterations that you expect, the validation task can help you determine whether they are missing from this TRS feed. If a user runs a report that doesn't include artifacts from a project that the user belongs to, TRS feed validation can fix the permission problem. After the validation, the report shows all the expected data.
  • GCM Resources: This feed contains information about changes to global configurations, for example, when they were created, updated, or deleted.

Procedure

  1. Open the application administration page for the TRS feed to validate:
    • Jazz Team Server: https://host:port/jts/admin
    • Rational Team Concert®: https://host:port/ccm/admin
    • Rational Quality Manager: https://host:port/qm/admin
    • Global Configuration Management (GCM): https://host:port/gc/admin
  2. On the toolbar, click TRS Feed Validation.
  3. Select a TRS feed and click Validate > Validate now. To validate the RQM Resources feed go to step 4.

    Validation examines the changes in the TRS feed. The change logs are recorded and stored since the first validation or since the last time that incremental data was reset. The validation results show errors from all the saved change records.

    If you no longer need the logged information, you can discard the old data, and validate the TRS feed from scratch. To remove all saved change records, select the Reset incremental data check box. You might want to do this if, for example, errors in the TRS feed are resolved and the LQE index was updated accordingly.

    If the validation finds discrepancies, you see a list of missing or extra artifacts. Click the links and examine the information for why the artifacts are missing or added to the TRS feed. If you can't find the cause of the problem, contact IBM Support.

    If the data in the repository matches the TRS feed but you still see discrepancies in reports after validating the feed, consider running a full validation. This operation takes longer because it examines every entry in the TRS feed.

  4. To validate the RQM Resources feed, select the TRS feed, and click Open RQM TRS Validator. Then, follow this procedure:
    1. For a full comparison of the TRS feed artifacts with those in the repository, click compare > runCompare. When the comparison starts, you can check the progress or stop the process.
    2. To see the log of all validations that ran on the system, or any validation that it’s still in progress, click trsLogs.
    3. When the comparison ends, check the logs for messages, such as Items match exactly or Discrepancies were found.
    4. If discrepancies are found, they are automatically fixed if you set the Automatically resolve discrepancies option for the com.ibm.rqm.oslc.trs.validator.TRSValidationWatchdog property to true.

      If you didn't change the advanced property, you can start cleanup events at https://host:port/qm/TRSValidator by clicking compare > cleanDifferences.

What to do next

If the problem persists, ask an LQE administrator to validate the LQE index.

Rational DOORS Next Generation

Use the Rational DOORS Next Generation Reporting Tool to validate the DNG Resources TRS feed.

Before you begin

Procedure

  1. Start the validation task from a command line: runRT.bat TRS validateLQE

    This command finds missing or extra artifacts but doesn't look at the content of the artifacts.

    To view detailed output while the task runs, see the rmLQE.log log file. On the product banner, click Debug. Then, click Logging, and search for rmLQE.log in the list.

  2. Check the results at the end of the rmLQE.log file.
    Discrepancies between the Rational DOORS Next Generation repository and data in the LQE index are written to an archive (.zip) file in a subfolder where you ran the validation command.
    Note: The absence of an archive file doesn't mean discrepancies weren't found. The reporting tool doesn’t wait indefinitely for the Rational DOORS Next Generation server to respond to the validation request. Check the rmLQE.log file for the status of the task.
  3. Fix the LQE data problems and validate the repairs by using the generated report (.zip file) as input to the runRT.bat FIX repairLQE command. For details, see the Repairing the LQE Data section of the wiki page.
  4. In the LQE application, check the status of the DNG Resources data source on the Data Sources page.

Lifecycle Query Engine

Validate the LQE index to compare it with a TRS feed from a CLM application. This task finds missing, stale (outdated), or extra artifacts.

Before you begin

  • If you upgrade from 6.0.6 or an earlier version, a migration task creates a new partition that is used for validation and shortens reindexing time. This process might take some time.
  • If you upgrade from any earlier version, add the following properties to the JazzInstallDir/server/conf/log4j.properties file to ensure that your validation logs use the latest format. 
    # Validation Task
log4j.logger.com.ibm.team.integration.lqe.lib.indexer.tasks.ValidationTask = info, lqeValidationTaskLog
log4j.additivity.com.ibm.team.integration.lqe.lib.indexer.tasks.ValidationTask=false
log4j.appender.lqeValidationTaskLog=org.apache.log4j.RollingFileAppender
log4j.appender.lqeValidationTaskLog.MaxFileSize=10MB
log4j.appender.lqeValidationTaskLog.MaxBackupIndex=5
log4j.appender.lqeValidationTaskLog.File=logs/lqe-validation.log
log4j.appender.lqeValidationTaskLog.layout=org.apache.log4j.PatternLayout
log4j.appender.lqeValidationTaskLog.layout.ConversionPattern=%d{ISO8601} [%30t] %5p %-50.50c  - %m%n

About this task

CLM application administrators might ask you to validate the LQE index after they validate TRS feeds.

Consider the following points before you validate the LQE index:
  • Run this task when system load is low. The time required for validation depends on the size of the index.
  • Don’t validate all the TRS data sources at the same time. Validate large data sources on separate days.
  • Validate only one TRS feed for one data source (CLM application) at a time. For CLM applications that have more than one TRS feed, run a separate validation for each feed.
  • Administrators can run LQE validation on a regular basis to confirm whether the LQE data has been kept in sync with the TRS feed from the data source. For example, if you run weekly validation and resources of the same type are logged as missing, you should check the data-providing application. You might need to change or configure a setting in that application. If the validation task runs for a few months without reporting any issues, you can extend the validation schedule. For example, you can run the task monthly.

To stop a validation that is running, restart the LQE server.

Procedure

  1. Open the Data Sources page: https://host:port/lqe/web/admin/data-sources
  2. Click the name of the TRS feed to validate.
  3. Click Validate near the upper right of the page.
    Validate button on the data source page
  4. Optional: Choose when to validate the data source by selecting a date and time.
    Important: Don't set your schedule too far in advance. No further updates are made to the index from the moment you set the schedule until the validation is finished.
  5. Click Validate and follow the prompts.

    While LQE reads the TRS feed, you see this message: Waiting for a response from the data source. This step can take a while, depending on the number of resources in the feed. This is an expected behavior, and no action is needed.

Results

When the validation finishes, the results are published on the Data Sources page, at https://host:port/lqe/web/admin/data-sources. Click the status link of a data source:
Data source status link
On the Data Source Status page, you can see how many resources were updated (because they were either missing or stale), and how many extra resources were removed.
Validation results on the data source status page
On the Overview tab, click Download Logs to retrieve all the LQE logs for this data source.
Download logs button
The validation log files are named lqe-validation-<encoded_data_source_URL>.log. The logs list resource discrepancies in the following categories:
  • Missing resources: These resources are included in the TRS feed but not in the LQE index. The validation process will fetch these resources again to resolve the discrepancy.
  • Stale resources: The resources in the TRS feed are newer than the resources in the LQE index. The validation process will fetch these resources again to resolve the discrepancy.
  • Extra resources: These resources are included in the LQE index, but not in the TRS feed. The validation process will delete these resources to resolve the discrepancy.

You can enable email notification for validation events. For details see Setting up email notification for Lifecycle Query Engine events.

If you still see discrepancies in the content of a report after validation, consider reindexing.

After you click Validate, no further changes are indexed for that data source. If you scheduled validation to run later, no changes will be processed from the moment you click the validation button, until the validation finishes.