Validating TRS feeds and the LQE or LDX index

Edit online

Validate all the tracked resource set (TRS) feeds registered in the Lifecycle Query Engine (LQE) or LDX index when Report Builder reports don’t show the artifacts that you expect. Validating these items can help you diagnose and troubleshoot problems and avoid reindexing data providers.

About this task

You might validate only TRS feeds, for which, validation is supported by applications, in these situations:

A report is missing artifacts.
A report includes outdated (stale) artifacts.
A report shows the same artifact more than once, possibly with different attribute values or with its old and new name.
An application view and a corresponding report in Jazz® Reporting Service (JRS) show a different number of artifacts.
A report includes deleted artifacts.
LQE or LDX shows skipped resources.

Validating all the TRS feeds that are registered in the LQE or LDX index can increase your confidence in the quality of your reports and help you troubleshoot missing, stale, or extra artifacts:

TRS Validation works by itself, without accessing LQE or LDX. It compares only the actual and expected content in TRS feed.

Note:

Do not run more than one TRS validation operation at a time for a given TRS.
If the application server is stopped during a TRS Validation operation, then when the server is restarted, the TRS validation is not automatically restarted and it is in an unknown state. The admin user must set the option to clear the cache when starting a new validation.

The steps to validate TRS feeds are different across Engineering Lifecycle Management applications. Complete the steps in this topic for the application whose feed you want to validate.

Note: Some TRS validation and repair options might not be available if 7.0.2 iFix23 or higher is not installed.

To learn more about validation before you do it, see the following resources:

Reliable reporting with Jazz Reporting Service – how to resolve data problems (Jazz.net article)

For a description of TRS feeds in Engineering Lifecycle Management, see Adding Tracked Resource Set feeds for lifecycle management applications.

Jazz Team Server, Engineering Workflow Management, Rhapsody Model Manager, Engineering Requirements Management DOORS Next, Engineering Test Management, and Global Configuration Management

Edit online

Validate TRS feeds on the TRS Feed Validation page in the administration section of the application.

About this task

The following are some examples of TRS feeds:

JTS User Resources (TRS 2.0): This feed contains information about Jazz Team Server (JTS) users. If a report about system users isn't showing the expected results, and you confirm that 10 users are registered in the repository, the validation task checks whether the TRS feed has 10 distinct references to users.
Engineering Workflow Management Process Resources (TRS 2.0): If a report doesn’t show the project areas, team areas, or iterations that you expect, the validation task can help you determine whether they are missing from this TRS feed. If a user runs a report that doesn't include artifacts from a project that the user belongs to, TRS feed validation can fix the permission problem. After the validation, the report shows all the expected data.
GCM Resources (TRS 2.0): This feed contains information about changes to global configurations, for example, when they were created, updated, or deleted.

Tip: It is recommended to run validation when the server is not under load. Maintenance windows, when nobody makes changes on the server are preferable. Validation is potentially heavy operations, consuming a lot of memory and processor time.

Procedure

Open the application administration page for the TRS feed to validate:
- Jazz Team Server: https://host:port/jts/admin
- Engineering Workflow Management or Rhapsody® Model Manager: https://host:port/ccm/admin
- Engineering Requirements Management DOORS® Next: https://host:port/rm/admin
- Engineering Test Management: https://host:port/qm/admin
- Global Configuration Management (GCM): https://host:port/gc/admin
On the toolbar, click TRS Feed Diagnostics.
Select the TRS that you want to validate and click Validate artifacts and configurations. A TRS validation dialog box opens. Validation options in this TRS validation dialog depend upon which validation options are provided by the current application for the chosen TRS.
- Validate configurations option:
  Available when the application supports validation of configurations in the repository. Use this option to check that configurations have the correct members, and that those members are present in the TRS.
  - When Validate all configurations option is selected, the following happens:
    
    TRS Validation performs a more thorough analysis of all configurations in the repository. Configuration membership is verified to agree with current state in the repository, and each configuration member is checked to ensure that it is present in the TRS.
    
    This option must be used consistently whenever a validation is done. Otherwise, the validation cache may become stale and validation discrepancies will result. Those discrepancies can be resolved by using the Clear cached validation data option.
    Note: It is acceptable to run validation for all configurations, then validate specific configurations, then again validate all configurations. However, running validation for all configurations, then running validation without validating configurations, and then validating all configurations will probably result in validation failures in configuration content. To resume validating all configurations in such case, you must clear cached validation data.
  - When Validate specific configurations option is selected, validation focuses only on the chosen configurations to validate, rather than all configurations and all resources in the TRS. It may be quicker to validate a few configurations than all configurations. Further, when you do not care about the validity of other configurations:
    
    You can select any combination of local and global configurations, and validation will check the union of all local configurations, which are contributed to each chosen global configuration.
    Note: Notice that only configurations from the current application are checked.
    
    The membership of each local configuration is checked against the repository, and the TRS is checked to ensure that all configuration members are present.
- Validate artifacts option:
  The Validate artifacts option can be used to verify that resources in the TRS can be fetched from the current application. Use this option if LQE is detecting 'skipped' resources.
  
  This option is mandatory when validating all configurations in the TRS.
  
  When validating specific configurations, only configurations and their members are validated.
- Resolve problems option:
  Sometimes, applications might choose not to provide any repair capabilities in TRS Validation. So, this option is available only when application supports TRS feed repair.
  
  You can use the Resolve problems option in the Validate dialog box to automatically resolve discrepancies, if any. The checkbox is selected by default. This option is displayed only for the TRS feeds that support repair operation.
  
  When there are unresolved discrepancies, you can see a list of missing or extra artifacts, unknown resource for Selections patch, wrong patch eTag, unexpected artifact added by a patch, etc. The Resolve problems option ensures that the artifacts in the TRS feed match the current set of artifacts in the repository. With the option enabled, it automatically resolves discrepancies in the TRS feed by adding missing artifacts or removing extra artifacts, or by using some other similar options to fix the feed.
  
  The validation results are appended to the existing validation summary. You can view the number of repaired, missing, and extra artifacts in the feed. These artifacts are listed in separate sections. If all issues are resolved, then the validation result shows a green checkmark and shows a red cross if there are discrepancies.
  
  If you see discrepancies in reports after you validate the feed, consider running a full validation. This operation takes longer because it examines every entry in the TRS feed, discards all caches and starts with the beginning.
  
  The artifacts that cannot be repaired are listed as missing or extra artifacts. If you still see discrepancies, even after running the full validation, contact IBM Support.
- Clear cached validation data option:
  When you are validating TRS feed, selecting this option, erases cached data that is recorded during previous validations that pertains to the current validation. (This cached data helps to improve TRS Validation performance.)
  
  Tip: As you can validate either the TRS (with or without configurations) or specific configurations, it is important to use the same options each time. When validating specific configurations, options do not matter for the correctness of the next TRS validation.
  Use this option in the following cases:
  - After the TRS has been rebased or truncated, either when done manually, or as part of a scheduled periodic task.
  - Following a validation run with incompatible options, such as, after you accidentally cleared the Validate configurations checkbox but still want to keep validating configurations.
  - If you suspect that previous validations reported incorrect failures due to inconsistent validation options, such as when you tried validating a configuration with an outdated configuration cache entry. (This might happen when you stop validating all configurations when you are validating TRS and then decide to validate specific.)
In the Validate dialog box, click Validate artifacts and configurations. The TRS is read and validated.

For Engineering Test Management, you can also go to https://host:port/qm/TRSValidator and complete the following steps.
1. For a full comparison of the TRS feed artifacts with those in the repository, click compare > runCompare. When the comparison starts, you can check the progress or stop the process.
2. To see the log of all validations that ran on the system or any validation that is still in progress, click trsLogs.
3. When the comparison ends, check the logs for messages, such as Items match exactly or Discrepancies were found.
4. If discrepancies are found, they are automatically fixed if you set the Automatically resolve discrepancies option for the com.ibm.rqm.oslc.trs.validator.TRSValidationWatchdog property to true.
  
  If you did not change the advanced property, you can start cleanup events at https://host:port/qm/TRSValidator by clicking compare > cleanDifferences.
Note:
You can use an alternative Engineering Test Management validation tool to automatically fix any discrepancies. The tool requires an advanced property before you start the validation process.

To set the property, go to the administration page at https://host:port/qm/admin and find the Advanced Properties section. Then, change the Automatically resolve discrepancies option for the com.ibm.rqm.oslc.trs.validator.TRSValidationWatchdog property to true.

DOORS Next TRS Validation:

For running DOORS Next TRS Validation, it requires the Selections Validation disk cache directory advanced property to be set before validation, pointing to a directory on the application server. To set the property, go to the administration page at https://host:port/rm/admin and find the Advanced Properties section.

The Selections Validation disk cache directory contains configuration Selections cache data, which may be large. If you need to relocate the cache directory, it can be done by changing the property value. After changing the property value, you must select the Clear cached data option during the next validation.

Note: The system can estimate the disk space that is required for Selections Validation disk cache directory. The estimation is done when you conduct a full TRS rebase and the results can be seen on the Required file system space estimate tab on the TRS Feed Diagnostics page for DOORS Next resources. It is recommended that the Selections Validation disk cache directory is placed on a different file system than the DOORS Next server to prevent issues if the disk becomes full.

What to do next

If the problem persists, ask an LQE or LDX administrator to validate the LQE or LDX index.

Lifecycle Query Engine or LDX

Edit online

Validate the LQE or LDX index to compare it with a TRS feed from an Engineering Lifecycle Management application. This task finds missing, stale (outdated), or extra artifacts.

Before you begin

If you upgrade from 6.0.6 or an earlier version, a migration task creates a new partition that is used for validation and shortens reindexing time. This process might take some time.

If you upgrade from any earlier version, add the following properties to the JazzInstallDir/server/conf/log4j2.xml file to ensure that your validation logs use the latest format.

# Validation Task
<Configuration name="ApplicationLogging" status="error" monitorInterval="10">
    <Properties>
        <Property name="pattern">%d{ISO8601} [%30t] %5p %-50.50c %X{Tid} - %m%n</Property>
        <Property name="ansipattern">%d{ISO8601} [%-5.5t] %5p %c{1} %X{Tid} %highlight{%m}%n</Property>
        <Property name="app">lqe</Property>
        <Property name="dir">logs</Property>
    </Properties>
	<Appenders>
	<!-- Validation task log file -->
		<RollingFile name="lqeValidationTaskLog" filename="${dir}/${app}-validation.log" filePattern="${dir}/${app}-validation-%i.log">
			<PatternLayout pattern="${pattern}"/>
			<Policies>
				<SizeBasedTriggeringPolicy size="10 MB"/>
			</Policies>
			<DefaultRolloverStrategy max="5"/>
		</RollingFile>
	</Appenders>
	<Loggers>
		<!-- Validation logging -->
		<Logger name="VALIDATION" level="INFO" additivity="false">
			<AppenderRef ref="lqeValidationTaskLog"/>
		</Logger>
	</Loggers>
</Configuration>

About this task

Engineering Lifecycle Management application administrators might ask you to validate the LQE or LDX index after they validate TRS feeds.

Consider the following points before you validate the LQE index:

Run this task when system load is low. The time that is required for validation depends on the size of the index.
Don’t validate all the TRS data providers at the same time. Validate large data providers on separate days.
Validate only one TRS feed for one data provider (Engineering Lifecycle Management application) at a time. For Engineering Lifecycle Management applications that have more than one TRS feed, run a separate validation for each feed.
Administrators can run LQE or LDX validation regularly to confirm whether the LQE or LDX data is in sync with the TRS feed from the data provider. For example, if you run weekly validation and resources of the same type are logged as missing, you must check the data-providing application. You might need to change or configure a setting in that application. If the validation task runs for a few months without reporting any issues, you can extend the validation schedule. For example, you can run the task monthly.

To stop a validation that is running, restart the LQE or LDX server.

Procedure

Open the Data Providers page: https://host:port/lqe/web/admin/data-sources
Click the name of the TRS feed to validate.
Click Validate artifacts and configurations in the header of the page.
Optional: Choose when to validate the data provider by selecting a date and time.
Click Validate artifacts and configurations and follow the prompts.

While LQE or LDX reads the TRS feed, you see this message: Waiting for a response from the data provider. This step can take a while, depending on the number of resources in the feed. This is an expected behavior, and no action is needed.
Optional: Schedule LQE or LDX validation activities regularly.
1. Click Edit Schedule in the Validation Schedule section.
2. Select the Enable Schedule checkbox.
3. Specify the time and days of the week when you want to run the validation.
4. Save your changes.
When you enable a validation schedule for a TRS feed and it is the next scheduled task, the scheduled validation time is displayed in the Scheduled Tasks column of the Data Providers page.

Results

When the validation finishes, the results are published on the Data Providers page, at https://host:port/lqe/web/admin/data-sources. Click the status link of a data provider:

On the Data Provider Status page, you can see how many resources were updated (because they were either missing or stale), and how many extra resources were removed.

On the Overview tab, click Download Logs to retrieve all the LQE or LDX logs for this data provider.

The validation log files are named lqe-validation-<encoded_data_source_URL>.log. The logs list resource discrepancies in the following categories:

Missing resources: These resources are included in the TRS feed but not in the LQE or LDX index. The validation process fetches these resources again to resolve the discrepancy.
Stale resources: The resources in the TRS feed are newer than the resources in the LQE or LDX index. The validation process fetches these resources again to resolve the discrepancy.
Extra resources: These resources are included in the LQE or LDX index, but not in the TRS feed. The validation process deletes these resources to resolve the discrepancy.

You can enable email notification for validation events. For more information, see Setting up email notification for Lifecycle Query Engine events.

If you still see discrepancies in the content of a report after validation, consider reindexing.

After you click Validate artifacts and configurations, no further changes are indexed for that data provider. If you scheduled validation to run later, no changes are processed from the moment you click the validation button, until the validation finishes.

For more information, watch the following video: