IBM Support

IBM FileNet Image Services: Added Privileged Delete option for Centera or ECS storage

Fix Readme


Abstract

Image Services added an option to the 'deldocs' command called the Privileged Delete (-p). This new option is useful if you use Centera or ECS storage with integral SDS documents. Privileged Delete can be used to delete ECS or Centera content even when the object retention has not expired or when the retention is set to infinite.

Content

The Image Services tools, deldocs and DOC_tool are enhanced to support Privileged Delete for Centera or ECS storage platforms.
The new privileged delete feature is especially relevant to General Data Protection Regulation (GDPR) - data minimization and storage limitation.
Note: Only the deldocs tool supports privileged delete. Other methods of delete such as 'IDMdesktop delete' do not use privileged delete. Other delete methods when used with SDS Centera or ECS continue to use standard Centera or ECS delete and only object when the retention has expired.

To enable privileged delete, set the following features:
  1. Enable Content delete.
    Content delete is only done if the SDS has content delete is turned on through fn_edit (Single Document Storage > SDS Content Delete column).
  2. When you run the deldocs, set the -i option so that retention is ignored. Deldocs must be run on the primary storage server for multiple server systems fn_edit (Set the Server Attributes > Server Type column for server row to "Combined", "Root/OSAR" or "DocLocator").
  3. If you are using Image Services 4.2.0 releases, disable the SDS consistent delete feature. To do that, create the /fnsw/local/trigger/consistent_delete_mask with “off” content (echo off > /fnsw/local/trigger/consistent_delete_mask) and recycle Image Services.
  4. Enable the 'privileged delete' capability in the ECS or Centera device.
    The Compliance Edition Plus ECS or Centera model does not support privileged delete.
To confirm that privileged delete capability is enabled for an SDS unit and content delete, run the DOC_tool SDS command and Summary option ‘S’.
Example: The following example is the case of Image Services 4.2.0. Image Services 4.1.2 displays less information.
<DOC_tool:lib a>sds
 Select an SDS option:
 'l' - List option. List all SDS units (ID, name, System state, User State, Raw State)
 's' - Summary option. Summary info for SDS unit (ID, name, states, Worker settings, Connection string, priority, DEBUG level, content delete, SDSw library, SDS supported options)
 'd' - Detailed option. Includes Summary info (same as 'S' option) and total accumulated performance info per SDS unit (read, write, background)
 'w' - Worker option. Includes Summary info (same as 'S' option) and performance per SDS worker process for an SDS unit (read, write, background)
 'a' - All option. Includes Summary report info (same as 'S' option), followed by Detailed total accumulated report (same as 'D' option), followed by Worker process reports (same as 'W' option)
 'f' - Find SDS object. Finds an SDS object/document, reads and displays header info. Displays retention setting of the object.
 'reset_thresholds' - Reset SDS threshold option. This causes the SDS threshold trigger to be re-read: /fnsw/local/trigger/SDS_thresholds
 'q' Quit SDS command

 ('l', 's', 'd', 'w', 'a', 'f', 'reset_thresholds', 'q') : s
The current time is Wed May 26 13:42:10 2021
  
SDS info: SUMMARY option
All SDS units mode (y/n) [y]: : 
==>SDS unit = ecs_infinite (2)

       SYSTEM state = SYSTEM ENABLED (0x0)
       USER state = USER ENABLED  (0x0)
       Worker = 'SDS_worker' Number Instances = 3
       info = '10.30.254.71?/fnsw/local/sd/1/ecs.pea'
       SDS priority = high
       DEBUG Setting = MAX
       SDS content delete setting=YES
       dynamic repository lib = 'SDSw_centera'
       retention infinite
       SDS delete content setting=YES
       SDS supports: EBR=NO  HOLDS=NO  Retention Extension=YES Priv delete=YES
       dedicated SDS_workers to reads: 1
       SDS retry=YES Read_after_write=YES SDS FBC_breakup=NO
       Delete mask: OFF
Both SDS content delete and privileged delete must be enabled (“YES”).
If the DOC_tool displays “Priv delete=NO”, the privileged delete needs to be enabled by your ECS or Centera administrator.
Deldocs tool options for Privileged Delete
There are two new deldocs options for privileged delete:
  • -p for privileged delete
  • -r for the reason
The required -r option is used to convey the reason for using the privileged delete. The ECS or Centera objects are deleted regardless of the retention setting (infinite retention, chronological retention, event base retention, expired retention, no retention).
The other deldocs options can be used along with privileged delete (-p) such as document list file “-f <doclist file>”.

Example: This program deletes documents from Index and Document services. To run this program, enter the following:
 
deldocs { -f <filename> | -s <surfaceid> [-b] | -d <doc_id>} [-i] [-v] [-V] [-p -r "<reason>"]
Where the parameters are as follows:
-f <filename> Specifies the name of a text file of document IDs, one ID per line
-s <surfaceid> Specifies to delete all documents on this surface
-b Specifies to delete a document from both sides of the disk specified by <surfaceid>
-d <doc_id> Specifies a single document delete
-i Ignore SDS retention settings
-v Verbose mode - Print to standard output document IDs that are successfully deleted. The verbose mode is less efficient but the counters are more exact because the documents are deleted one at a time, and are easier to keep track of. If you need the exact counters for statistics, use the verbose mode. In addition, if the file name option is used with the verbose mode, documents that are skipped over because they do not exist in the DOCS table are also printed.
-V
Expanded Verbose mode - Operates similar to the -v option and must be used with the -f option. When the -V option displays the deleted document IDs, the storage location is also displayed: surface IDs and SDS unit number.
The individual deleted documents are also displayed to standard output because generating an elog per document might cause a disk space issue. Standard out might be redirected to an output file so that records of the deleted documents and their location might be saved. In addition, a summary report is displayed upon completion that will include all surfaces and SDS units deleted from. The number of documents deleted from each surface and SDS unit or both will be displayed to standard output and elog. The list of surfaces is useful to determine which surfaces to consolidate in order to purge the document content of the documents that are deleted from the databases. The consolidation must be invoked from the Background Library Control GUI window.
New privileged delete parameters
-p
  1. Privileged delete option - Must be used with the -r option.
  2. Can be run only with ECS or Centera documents.
  3. Can be run with any existing deldocs parameters.
  4. To perform privileged delete, the following are required:
    1. Enabling content delete for SDS unit using fn_edit
    2. Enabling content delete for ECS or Centera devices
    3. Running deldocs on the primary Storage Server
    4. Using the ignore retention (-i) option
    5. For Image Services 4.2.0: Setting conistent_delete_mask to 'off'
-r Reason for privileged delete - In order to run privileged delete, you must specify a reason using the -r parameter. The reason uses inverted commas (" ") to delineate a reason string. The character limit to specify the reason is 2,000. For example, -r "Need to delete because the employee left the company"

IMPORTANT:
  • Note that deldocs does not delete a document in the following scenario:
    • If the document is stored on an SDS device and the 'SDS delete' option is turned on for SDS unit, and
    • If the document retention setting is set to one of these options: infinite, EBR, or the chronological
      retention.
      Workaround: To ignore the SDS retention setting, use the -i option and proceed with deleting.
  • When using privileged delete ('-p' and '-r' option), log files are created with the following naming convention:
    • Unix: /fnsw/local/logs/sds_logs/priv_delete_<YYYYMMDD>
    • Windows: <drive>:\fnsw_loc\logs\sds_logs\ priv_delete_<YYYYMMDD>
  • The following is an example of a log generated after successfully deleting an ECS or Centera object:
           2020/11/28 14:01:52.809 <Administrator> deldocs (6680.5108 0x1a18.13f4) SDS_priv_delete() Success; doc_id=104204;
           reason=’Need to delete’;   clip_id=3ONL26FCKUJPGe2GQ3816CIHHQNG41BOLABF6F0D7BO54C713CN2M
  • To generate the above log, the following command was run:
      deldocs -i -p -r “Need to delete” -d 104204
Privileged Delete examples
The following are some examples of running deldocs with the new privileged delete option.
  1. Deleting a list of documents with doclist file (-f) that contains a document ID per line:
    deldocs -i -p -r “need to delete group of documents” -f doclist
    Note: When you use a document list file (-f) with privileged delete, the same reason (-r) is used for all the documents in the list.
  2. Deleting a list of documents with doclist file (-f) and expanded verbose expanded (-V) which results in displaying to standard output each document deleted and storage location (surface_id, SDS unit):
    deldocs -V -i -p -r “need to delete group of documents” -f doclist
    Note: The parameters can be used in any order.
Tombstones and JCASS tool
After a deletion is done, the ECS or Centera creates queryable metadata for each document known as tombstones. It includes the reason specified in the -r option. The JCASS tool provided by Dell or EMC can be used to access the tombstones.
Here are some examples of using the JCASS tool to display tombstones:
CASScript>qsd
Query Expression Properties:
        Start Time: UNBOUNDED
        End Time: UNBOUNDED
        Type: DELETED
CASScript>qia name
Query Expression Properties:
        Start Time: UNBOUNDED
        End Time: UNBOUNDED
        Type: DELETED
        Attributes: name
CASScript>qia creation.date
Query Expression Properties:
        Start Time: UNBOUNDED
        End Time: UNBOUNDED
        Type: DELETED
        Attributes: name, creation.date
CASScript>qia modification.date
Query Expression Properties:
        Start Time: UNBOUNDED
        End Time: UNBOUNDED
        Type: DELETED
        Attributes: name, creation.date, modification.date
CASScript>qia retention.period
Query Expression Properties:
        Start Time: UNBOUNDED
        End Time: UNBOUNDED
        Type: DELETED
        Attributes: name, creation.date, modification.date, retention.period
CASScript>qia reason
Query Expression Properties:
        Start Time: UNBOUNDED
        End Time: UNBOUNDED
        Type: DELETED
        Attributes: name, creation.date, modification.date, retention.period, reason
CASScript>querysetlowerbound "2020.08.02 00:00:00"
Query Expression Properties:
        Start Time: 2020.08.02 00:00:00
        End Time: UNBOUNDED
        Type: DELETED
        Attributes: name, creation.date, modification.date, retention.period, reason
CASScript>query
Performing clip query
Query Expression Properties:
        Start Time: 2020.12.04 00:00:00
        End Time: UNBOUNDED
        Type: DELETED
        Attributes: name, creation.date, modification.date, retention.period, reason
77CMQTE2AU9I4e89I8C3DNT5QQPG41BNOOJ31U07AT70IOINK49TR 2019.08.17 06:53:30 GMT name/P8IM_93947201_104153
creation.date/2020.12.03 18:51:50 GMT
modification.date/2020.08.17 18:51:50 GMT
retention.period/4294967295
reason/Test delete with new tool
Number of clips returned: 1
All results returned: True
 
When Image Services is restarted the ECS or Centera 'privileged delete' setting is displayed in the elog startup information as illustrated in the following:
2021/04/12 11:41:24.377 238,0,1 HGPGJ0001I <fnsw> SDS_worker 2 1 (942254.1.252 0xe60ae.1) [INFO]
Log on to: 'ecs_server?/fnsw/local/sd/1/ecs.pea' Repository
Dynamic Connector library: 'SDSw_Centera'
Device-specific info: 'ECS (version 3-0000-0-0) libSDSw Ver=1.0,
Mode=ce, SDK v3.2.705 (Apr  2 2021 21:27:09) priv_delete=true'
Capacity: 490.374999GB (514195455KB) 
Used: 436.541250GB (457746678KB) (89.02%)
Free: 53.833749GB (56448777KB) (10.98%)
Supports EBR=YES holds=YES retention_extend=YES
Feature availability
The feature is available from the following releases of Image Services on the IBM Fix Central:
  • Image Services 4.1.2.27 interim fix 6104342 and later
  • Image Services 4.2.0 fix pack 16 (4.2.0.16) and later

[{"Type":"SW","Line of Business":{"code":"LOB45","label":"Automation"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSNVUD","label":"FileNet Image Services"},"ARM Category":[{"code":"a8m50000000Cdy8AAC","label":"FileNet Image Services"},{"code":"a8m50000000CdrDAAS","label":"FileNet Image Services-\u003ESDS"},{"code":"a8m50000000CdzgAAC","label":"FileNet Image Services-\u003EStorage Devices"},{"code":"a8m50000000Ce0gAAC","label":"FileNet Image Services-\u003EStorage Devices-\u003ECSAR (EMC Centera)"}],"Platform":[{"code":"PF002","label":"AIX"},{"code":"PF027","label":"Solaris"},{"code":"PF033","label":"Windows"},{"code":"PF057","label":"HP"}],"Version":"4.1.2;4.2.0"}]

Document Information

Modified date:
22 December 2021

UID

ibm16455615

Page Feedback