How to collect data

There are three approaches that you can take to collect data for Transformation Advsior

Using the data collector

The data collector is a tool that gathers information about middleware deployments in your environment to help IBM Cloud Transformation Advisor provide you with a migration analysis of Java™ EE applications running on IBM WebSphere Application Server, Apache Tomcat, Red Hat JBoss, or Oracle WebLogic application servers. The tool generates one .zip folder per profile/domain and places analysis results within that directory.

For Java applications, the tool leverages the Migration Toolkit for Application Binaries. You may use the Migration Toolkit for Application Binaries (AKA the "binary scanner") directly. This may be particularly useful in some troubleshooting scenarios. For information on how to download and use the Migration Toolkit for Application Binaries please click here. If these tools are already in use, then the same security requirements which are required to run them are also required to run the data collector.

The data collector will collect configuration information on WebSphere Application Server installations at Version 6.1 or later.
The exact configuration information that is collected can be found here

To analyze your applications, the data collector requires the following access:

  • Read access to the application server installation directory and all subdirectories
  • Read access to profile directories
  • File and directory creation and write access to the current directory

The data collector does not gather any data related to the following:

  • Message content or data processed by workload
  • Logs or log data
  • Password information

Downloading the data collector

To get started, complete the following steps from the Transformation Advisor UI:

  1. Create a workspace.
  2. On the new workspace page, click the data collector button and follow the instructions to download.

Installing the data collector

To install the data collector, log on to your application server with the application owner’s user credentials and complete the following steps:

  1. Copy the downloaded file to your application system in a directory where it has read-write-execute access.
  2. Decompress the downloaded file by issuing the command for your operating system:

Linux: tar xvfz transformationadvisor-Linux_<WORKSPACE_NAME>.tgz

AIX: gunzip -c transformationadvisor-AIX_<WORKSPACE_NAME>.tgz | tar xf -

Solaris: tar xvfz transformationadvisor-Solaris_<WORKSPACE_NAME>.tgz

z/OS: gunzip -c transformationadvisor-zOS_<WORKSPACE_NAME>.tgz | tar xf -

Windows: unzip transformationadvisor-Windows_<WORKSPACE_NAME>.zip

  1. Go to the data collector directory:

cd transformationadvisor*

Required resources

 System         | Memory (GB) | CPU (cores) | Disk space (GB)
 ---------------| ----------- | ----------- | ---------------
 Data Collector | 2           | 2           | 0.5

Running the data collector

When you run the data collector for the first time, the license will be displayed. Accept the terms to continue.

WARNING: The data collector should not be run on production servers.

To analyze both your applications and their configuration, run the command for your domain:

IBM WebSphere: ./bin/transformationadvisor -w <WEBSPHERE_HOME_DIR> -p <PROFILE_NAME> [ --ignore-missing-binary --ignore-missing-shared-library --applications --applications-file --skip-applications --skip-applications-file --no-upload] ([] denotes optional arguments)

Oracle WebLogic: ./bin/transformationadvisor --web-logic-config-file <Path to the WebLogic domain config.xml file> [--applications --applications-file --skip-applications --skip-applications-file --no-upload] ([] denotes optional arguments)

JBoss: ./bin/transformationadvisor --jboss-config-dir <Path to the JBoss server configuration directory> [--applications --applications-file --skip-applications --skip-applications-file --no-upload] ([] denotes optional arguments)

Apache Tomcat: ./bin/transformationadvisor --tomcat-home-dir <TOMCAT_HOME_DIR> --tomcat-config-dir <TOMCAT_CONFIG_DIR> [--applications --applications-file --skip-applications --skip-applications-file --no-upload] ([] denotes optional arguments)

To view command line options that are available for the data collector, use the --help option.

Viewing your data

Depending on the number, size, and complexity of your applications, the data collector may take some time to execute and upload the results. During this process, you can keep track of its progress by checking your command line.

If there is a connection between your system and your new collection, the data collector automatically uploads the results to Transformation Advisor. A detailed analysis that includes several reports is provided to help you understand issues and where code changes might be required.

If there is no connection, the data collector will return a .zip file containing your application data. You will need to manually upload the zip file using the Upload data button on the Workspace page.

Transformation Advisor data collector and Java

The data collector requires version 1.7+ to run. In the first instance, the data collector will look for a compatible version of Java that is used by WebSphere (if this is a WebSphere collection). If it cannot find the WebSphere Java version, or if the WebSphere version is not compatible (i.e. < 1.7), or this is not a WebSphere collection, then the data collector will attempt to use the Java specified in the -–java-home data collector argument. If the –java-home argument is not provided, it will use the Java specified with the JAVA_HOME environment variable. If none of those options result in finding a compatible version of Java, the data collector will use the Java that is packaged with the data collector.

Replacing version of Java in the data collector

In some circumstances, a compatible version of Java may not pre-exist on the system that you want to run the data collector on, and the Java that is packaged with the data collector is also not compatible with the system on which it is being run. For example, this can arise with the Solaris data collector which contains Java for AMD64 architecture. If your Solaris is running on SPARC, then the packaged Java will not work. In these cases, you can build your own version of the data collector, with an appropriate Java version.

Take the following steps:

  • Download a version of that data collector via the Transformation Advisor UI
  • Unpack the compressed data collector
  • Locate the “jre” folder in the unpacked data collector and replace it with a jre from a desired Java version
  • OPTIONAL: There is a file in the unpacked data collector called uploadEndpoint.json. This contains the location of the Transformation Advisor server, and a unique key for uploading to a specific workspace. You can remove the uploadEndpoint.json file. This means that the data collector will not be able to automatically upload the collection to Transformation Advisor, but rather will produce a zip that you can manually upload using the Transformation Advisor UI. Removing the uploadEndpoint.json has the advantage of allowing you to create a custom data collector that can be readily copied between environments. In effect, it breaks the link between the data collector and a specific workspace in Transformation Advisor.
  • Compress the data collector

The compressed data collector can now be copied to whatever environment you wish to perform the collection on.

Migrating From SunOS and Solaris architectures

Transformation Advisor provides a data collector for SunOS/Solaris environments. It has been tested on SunOS 5.10, 5.11 (Solaris 10, 11), but may work on older versions. The data collector provided for SunOS/Solaris packages a JRE for use on SunOS/Solaris on AMD64 architecture. If you want to run the data collector on SunOS/Solaris SPARC architecture, then you need to have a compatible version of Java available on the System where you are running the data collector.

Known Issues:

  • Before running the data collector on SunOS/Solaris, please check the version of bash on the environment. If the version is < 4.x, please upgrade to 4.x to ensure correct functioning of the data collector.

Migrating from old versions of WebSphere

The Transformation Advisor data collector supports collecting from WebSphere v6.1+. For WebSphere migrations from versions older than 6.1, you need to manually find all of the application binaries deployed on the system and put them into some single location. You can then run the data collector using the “-o” option to point to that location and perform an analysis on the applications. When you run the data collector with the -o options, it will ask you to choose the source WebSphere and Java version. If it does not show your exact versions, select the values that are closest to your actual versions to get the most accurate results. In this situation, the analysis may not catch all issues, issues from version of WebSphere and Java older than the supported rules will not be flagged.

Migrating from supported application servers

Transformation Advisor supports collection from the following application servers:

  • WebSphere
  • Red Hat JBoss
  • Weblogic
  • Tomcat.

Migrating from application servers that are not supported

If you want to run an analysis on applications from application servers that are not supported (e.g. Glassfish), you can do the following:

  • Manually collect the application binaries from the system, and place in a directory.
  • Run the data collector and point it at the applications location using the “-c” option. This runs a base set of Java rules plus some Tomcat specific rules. The Tomcat specific rules may or may not apply to the application server that you are analyzing, and need further investigation. In future versions of the data collector, there will be the option to just run the generally applicable Java rules. The Tomcat rules that need further investigation are as follows:
Set the sharing scope on resource references
Spring applications might fail to run from a non-expanded WAR file
Stub classes must be included when using remote Enterprise JavaBeans (EJB) 2.x
The getRealPath method previously returned null for files that do not exist
The OSGI remote bundle repository service API is unavailable
The OSGI Remote Service Admin API is unavailable
Transaction propagation is not supported for Enterprise JavaBeans (EJB) remote interfaces
Use correct case for tag attribute names
Use Java EE deployment descriptors and WebSphere bindings to define resource link references
Use Java EE deployment descriptors and WebSphere bindings to define resource references
Use Java EE deployment descriptors to define context lifecycle listeners
Use Java EE deployment descriptors to define context parameters
Use Java EE deployment descriptors to define environment references
Use Java EE deployment descriptors to define missing security roles
Validate the result of concatenation with getRealPath("")
Validate the result of concatenation with getRealPath("/")
Web Services Notification (WS-Notification) is unavailable

Customize the scan options used to generate report files

The customCmd.properties file under the /conf directory is used to config the scan options used to generate the report file during scanning the application. User can edit this file to customize the scan options.

Defining your own rules

Since the release of Transformation Advisor v2.5 you can define your own custom analysis rules to detect scenarios specific to your application migration. The user-defined rules can be easily created as described in detail here. The results will display in Transformation Advisor UI in the same manner as the pre-written rules that come out of the box.

data collector command line tool options

General options:

--version

Displays the version of the data collector.

--help

Displays all options for the data collector.

--java-home

Specifies the version of Java used for the data collector.

-J-X

JVM options that are passed to the data collector. -J will be stripped and -X and remaining options will be passed to the Java runtime. For example, -J-Xmx4G will set the max heap size to 4GB for the data collector runtime.

--no-upload

Flag to indicate that no upload to Transformation Advisor server will take place once the collection completes

Options for IBM WebSphere:

-w, --was-home

The location of the WebSphere installation directory.

-p, --profile-config

If provided, the data collector will connect to the WebSphere JVM associated with that profile and collect the configuration information. If you have multiple profiles within your WebSphere installation directory, you can supply this option multiple times. The syntax for this option is: --profile-config <Profile Name>

Example: --profile-config AppSrv01 --profile-config Dmgr01

-a, --applications

By default, the data collector collects configuration information for all applications deployed on a profile. If this option is specified, the tool will only collect configuration information for the applications that are listed. The syntax to specify multiple applications is: --applications <application_1 Name>...<application_n name>

Example: --applications app1 app2 app3

-f, --applications-file

Similar to the --applications option, this option allows the data collector to obtain application names from the provided file, and only collect configuration data for applications that are defined in that file.

Example: --applications-file /tmp/applicationsToScan.txt

The file can be a comma separated and/or line separated list of applications to include, e.g.

app1.ear,app2.ear
app3.ear

-s, --skip-applications

By default, the data collector collects configuration information for all applications deployed on a profile. If this option is specified, the tool will skip the applications that are listed. The syntax to specify multiple applications is: --skip-applications <application_1 Name>...<application_n name>

Example: --skip-applications app1 app2 app3

--skip-applications-file

Similar to the --skip-applications option, this option allows the data collector to obtain application names from the provided file, and skip those applications that are defined in that file.

Example: --skip-applications-file /tmp/applicationsToScan.txt

The file can be a comma separated and/or line separated list of applications to skip, e.g.

app1.ear,app2.ear
app3.ear

--ignore-missing-binary

Signals the scan to skip applications that do not have binary files.

-i, --ignore-missing-shared-library

Signals the scan to run on applications that have a nonexistent shared library.

--scan-binary-location

Signals the scan to run against the expanded directory of the deployed application. By default, the data collector will scan the binary file which is uploaded to the WAS server during deployment time. If this option is specified, the data collector will scan the expanded directory of the deployed application on the WAS server.

-s, --scan-node

Signals the scan to run for managed profiles on the node.

-o, --outside-location

Location of a directory outside of the WAS home location that contains application binary files that are to be scanned. The syntax is: ./transformationadvisor -o <location of applications Outside WAS>

Note: If this option is specified, all previous arguments are ignored. The data collector will ask you to select the WebSphere version number and Java version.

Options for Oracle WebLogic:

-l, --web-logic-config-file

Path of the WebLogic domain config.xml file. If you have multiple domains, you can specify this option multiple times. The syntax is: --web-logic-config-file <Path of the config.xml file>

Example: --web-logic-config-file /home/oracle/Oracle/Middleware/Oracle_Home/user_projects/domains/base_domain/config/config.xml

-g, --web-logic-apps-location

Location of the directory that contains WebLogic application binary files. The syntax is: --web-logic-apps-location <WebLogicapps location>

Note: If this option is specified, all previous arguments are ignored. The data collector will ask you to select the Java version used for the Weblogic server.

Options for JBoss:

-j, --jboss-config-dir

Path of the JBoss server configuration directory. If you have multiple servers, you can specify this option multiple times. The syntax is: --jboss-config-dir <Path of server configuration directory>

Example: --jboss-config-dir /root/EAP-7.1.0/standalone/configuration

-b, --jboss-apps-location

Location of the directory that contains JBoss application binary files. The syntax is: --jboss-apps-location <JBoss applications location>

Note: If this option is specified, all previous arguments are ignored. The data collector will ask you to select the Java version used for the JBoss server.

Options for Apache Tomcat:

-t, --tomcat-home-dir

Path of the Tomcat CATALINA_HOME directory. If this argument is specified and the --tomcat-config-dir argument is not, then it is assumed that --tomcat-config-dir is the same as --tomcat-home-dir. If one or more --tomcat-config-dir options are specified in addition to --tomcat-home-dir, the --tomcat-home-dir will not be treated as a config directory, unless it's explicitly listed in one of the --tomcat-config-dir options. The syntax is: --tomcat-home-dir <Directory of Tomcat home>

-d, --tomcat-config-dir

Path of the Tomcat CATALINA_BASE directory. The --tomcat-home-dir (CATALINA_HOME) must be specified in addition to this option. If you have multiple CATALINA_BASE directories for a multi-instance Tomcat installation, you can supply this option multiple times. The syntax is: --tomcat-config-dir <Directory of Tomcat Configuration>

-c, --tomcat-apps-location

Location of the directory that contains Tomcat application binary files. The syntax is: --tomcat-apps-location <Tomcat appslocation>

Note: If this option is specified, all previous arguments are ignored. The data collector will ask you to select the Java version used for the Tomcat server.

Using the built-in WSADMIN command for WebSphere Application Server

The wsadmin command available with WebSphere has a built-in option for scanning your installed applications and generating a set of zip files that can be uploaded directly into Transformation Advisor.
The availability of this command depends on the WebSphere version and Fix Pack you are running:

  • WebSphere 8: Version 8.5.5.23 or later
  • WebSphere 9: Version 9.0.5.14 or later

The full details of this and all instructions can be found here

Using the Migration Toolkit for Application Binaries

An alternative to using the Transformation Advisor Data Collector is to use the Migration Toolkit for Application Binaries to generate a data collection that can be uploaded to IBM Cloud Transformation Advisor. Details of how to do this are outlined here.

.