IBM Data Reduction Estimator Tool (DRET) for SVC, Storwize and FlashSystem products

Download Description

Overview

The Data Reduction Estimator utility uses advanced mathematical and statistical algorithms to perform an analysis with a low memory footprint. The utility runs on a host that has access to the devices to be analyzed. It performs read operations, so has no effect on the data stored on the device. The following sections provide information on installing Data Reduction Estimator on a host and by using it to analyze, devices on it. Depending on the environment configuration, in many cases Data Reduction Estimator can be used on more than one host in order to analyze more data types.

It is important to understand block device behavior, when analyzing traditional (fully allocated) volumes. Traditional volumes that were created without initial zeroing the device might contain traces of old data on the block device level. Such data is not accessible or viewable on the file system level. When using Data Reduction Estimator to analyze such volumes, the expected reduction results reflect the saving rate to be achieved for all the data on the block device level, including the traces of old data.

Regardless of the block device type being scanned, it is also important to understand a few principles of common file system space management. When files are deleted from a file system, the space they occupied before deletion becomes free and available to the file system. This happens even though the data on disk was not removed, but rather the file system index and pointers were updated to reflect this change. When using Data Reduction Estimator to analyze a block device used by a file system, all underlying data in the device is analyzed, regardless of whether this data belongs to files that were already deleted from the file system. For example, you can fill a 100GB file system and make it 100% used. You can then delete all the files in the file system making it 0% used. When scanning the block device used for storing the file system in this example, Data Reduction Estimator (or any other utility for that matter) accesses the data that belongs to the files that are already deleted.

In order to reduce the impact of block device and file system behavior previously mentioned, it is recommended to use Data Reduction Estimator to analyze volumes that contain as much active data as possible rather than volumes that are mostly empty of data. This increases accuracy level and reduces the risk of analyzing old data that is already deleted, but might still have traces on the device.

Using Data Reduction Estimator tool

There are 3 steps that need to be performed to get the reduction ratio.

Step 1: Obtain the device list.

Step 2: Estimate the reduction ratio on each of the devices independently.

Step 3: In case there are 2 or more scanned devices, merge the results from all independent volume scans together to view the data reduction estimation across all scanned volumes.

The next section will describe how to perform each of the steps.

See the Syntax section for a detailed description of the syntax and command-line options.

Step 1: Obtain the device list

This step is performed differently on different platforms.

Linux, ESX, AIX, Solaris, and HP-UX server:

1. Log in to the host by using the root account.
2. Obtain the list of device names, by using the following commands:
   • Linux: "fdisk –l"
   • ESX: "esxcli storage core device list | grep Dev"
   • AIX: "lsdev –Cc disk"
   • Solaris: "format"
   • HP-UX: "ioscan –kfnC disk"

Windows server:

1. Log in to the server, by using an account with Administrator privileges.
2. Open an elevated command prompt with Administrator rights (Run as Administrator).
3. Run “wmic DISKDRIVE list brief”
   C:\>wmic DISKDRIVE list brief
   Caption DeviceID Model Partition Size
   IBM 2145 Multi-Path Disk Device \\.\PHYSICALDRIVE0 IBM 2145 Multi-Path Disk Device 1          256052966400

Step 2: Estimate the reduction ratio on each of the devices independently

Run Data Reduction Estimator tool with the following parameters:

• –d <device> the device to analyze, according to the DeviceID output
  • --command scan|partialscan the scan command performs a full scan, and reports total data reduction saving estimations (deduplication + compression), while partial scan performs a short scan, estimating compression savings only but not dedup. The default is “scan”
• –o outputFileName to keep the output results for the next step.
  
  For example:
  
  Linux:
  Data-Reduction-Estimator -d /dev/sda1 -o scan_Linux_RHEL7
  
  Windows:
  Data-Reduction-Estimator.exe -d \\.\PHYSICALDRIVE0 -o scan_Win7
   

  Data reduction estimator tool can scan one device at a time in a single CLI command. Using the --batchfile parameter, can handle several devices in parallel, when each line represents a different device.

  An example of several devices in a single batch file:

  In this example, devices /dev/sda and /dev/sdb are fully scanned, while device /dev/sdc are partially scanned.

  During the scan process, checkpoints are stored in a file (with *.rdb extension)

  .dat sketches are for completed scanned devices.

  If the scan is terminated before completion, on the next execution the scan will continue from the checkpoint.

  • -d /dev/sda
  • -d /dev/sdb
  • -d /dev/sdc –-command partialscan

Step 3: Merge the results from the independent scans together

1. Collect the output files of all scanned devices from step 2 and place them in the same directory, where the >Data-Reduction-Estimator tool is located;
2. To calculate the total data reduction saving, use the –-command merge option. Separate output file with a comma “,”:
   
   For example:
   Data-Reduction-Estimator --command merge --mergefiles scan_freebsd91_1024,scan_vendlist_1024,scan_win2008_1024,scan_Win7,scan_Linux_RHEL7

   After the data reduction analysis is completed, the overall data reduction estimation for all devices is displayed.

   • Volumes with overall data reduction ratio threshold of 90% or lower by default (equivalent to 10% savings or higher).
   • --mergeall – Overrides the data reduction threshold. All volumes are merged.

   Note:

   • Merge is applied to data files generated by the same binary build. Otherwise, the “Build mismatch” error is generated;
   • Merge cannot be applied on *.rdb sketches files. *.rdb are intermediate files created by unfinished scans.

Syntax

Linux, ESX, AIX, Solaris, and HP-UX:
Data-Reduction-Estimator –d <device> [-x Max MBps] [-o result data filename] [-s Update interval] [--command scan|merge|load|partialscan] [--mergefiles Files to merge] [--loglevel Log Level] [--batchfile batch file to process] [-h]

Windows:
Data-Reduction-Estimator.exe –d <device> [-x Max MBps] [-o result data filename] [-s Update interval] [--command scan|merge|load|partialscan] [--mergefiles Files to merge] [--loglevel Log Level] [--batchfile batch file to process] [-h]

Examples:

Data Reduction Estimator output examples

root@swfc120:/tmp# ./Data-Reduction-Estimator -d /dev/dm-10
Result data filename not given, auto-generating: file_C8F50050.dat
200.00 GB | 55.60 MBps: 0% [####################] 100%%
Estimated Dedupe Savings:         11.659%
Estimated Compression Savings:    65.797%
Data Reduction Savings:           69.784%
---------------------------------------
Zeroes Detected Savings:          0.227%
Total Data efficiency Savings:    69.853%
Time Consumed:                    00:11:01

Analyzing the results of the above example:
Volume size: 200GB
Auto-generated output file: file_C8F50050.dat

To get the total data on disk after reduction, take the data reduction saving of the disk size:
(100% - 69.784%) of 200GB = 60.432GB
Data on disk after reduction (dedup and compression) is 60.432GB

The Zeroes Detected saving refers to large sequences of zeros that were detected on the device. It is not an inherent part of the data reduction saving, as some systems consider this as thin provisioning. The total data efficiency is the total savings, including deduplication, compression, and the large zero sequences combined.

First data is deduped, and then it is compressed.
Dedup saving:
0.11659 * 200GB = 23.318GB
Compression saving (after dedup)
0.65797 * (200 - 23.318GB) = 116.210GB

Total saving: 23.318GB + 116.210GB = 139.528GB (~70% of 200GB)

Next example illustrates the total data saving on two volumes: RHEL7 and win2008.

Data-Reduction-Estimator --command merge --mergefiles scan_Linux_RHEL7,scan_win2008_1024
Result data filename not given, auto-generating: merge_out
Estimated Dedup Savings:         97.8%