IBM Support

Scrubbing, Shredding or Sanitizing your Disks to Remove All Data via a small C program

How To


Summary

If you are repurposing a computer or giving it to some one else, then you want to ensure you remove all data from the disks. Removing the files does not remove the file content and special scanning devices can even detect previous data recorded on the disks.

Objective

nigels banner

Environment

An AIX computer with internal or remote disk drawer "real brown spinning" disks. 
Updated: This tool does not apply to Fibre Channel SAN disk LUNs, high function disk subsystem, SSP, or solid-state drives (SSD). These devices might not destroy your data that uses multiple writes due to advanced techniques like snapshots, de-duplication, remote asynchronous mirrors, remapping on write for wear-leveling and blocks are scattered across many drives.

IBM takes no responsibility that this program conforms to any standards - you have to decide for yourself.

Warning: This program is for root users as it needs access to the raw disks.
Warning: This program might need you to boost your ulimit (memory) settings - if you do not know how to set these limits then you are probably the wrong person to be running this command.

Steps

We are not covering deleting the contents of files - only the entire whole disk content.
So what are your options
1) One approach is to hammer a nail through the disk drive = 100% sure it is unreadable. You can purchase a crude device that does this damage for you but the disks are mechanically destroyed.  This method is the only approved method for data classified as SECRET. You might be required to break the disk into small chunks!
2) If you want Department of Defense standard level disk scrubbing or shredding and large numbers of disks then you could purchase specialized software.  But that might not work on some operating system and hardware.  For more information about this topic, see the details at the end of the article. There are many disk formats, disk technology, and disk-connecting buses, which complicate software and various versions to cover all the bases. The second problem with this approach is that you are not sure what the software program does - you might want some sort of certification to build trust that is works as expected.
3) Some people use the AIX "diag" command option "Format" to remove data.  While this software is free, it might not remove the data and you can run one at a time (a diag limitation) plus it takes hours.  I could find no reference to diag overwriting the disk - just a low-level format, which is a vague term.  This formatting does not conform to the recommended processes.
4) If you want a "cheap and cheerful" scrubbing tool and AIX / Linux then the "dd" command can be used to write zeros or random numbers all over the disk:
 
    dd if=dev/zero of=/dev/rhdisk99 bs=4m
    or
    dd if=dev/random of=/dev/rhdisk99 bs=4m
But many have concerns: writing zeros is not good enough and probably the easiest overwrite to "see" through.  Some data recovery companies can read such a disk. Their disk recovery experts use at high-cost magnetic scanning devices and advanced techniques to find ghost patterns of previous values on the disk.
5) To avoid that you need to write multiple times and multiple bit patterns all across the disk and then confirm that the writing took place.
To over write the whole disks, we don't want the disk to be included in any AIX LVM Volume Group or include any Logical Volumes.  Overwriting them while AIX thinks it has access confuses AIX and it can assume the Disk Failed that the contents is invalid and then you have to force it out of AIX control. This operation can prove tricky and takes considerable time with AIX trying to recover the situation.

Make sure that lspv command does not report the disk is active. For example,
# lspv
hdisk0          00f6db0a6c7aece5                    None
hdisk1          00c0493045dbeb1e                    rootvg          active
#
Notes:
  • Here, hdisk0 is not active - this disk can safely be scrubbed or shredded.
  • If you accidentally scrub or shred hdisk1, in this case, you destroy your running AIX in a faction of a second.

Warning: Using this program is at your own risk.

On AIX to read and write the whole disks, we access it using the raw disk interface. AIX devices are found in /dev and the raw device starts with an "r".  For example, regular device on AIX is hdisk42, and the raw device is /dev/rhdisk42. This approach is fast and avoids AIX disk file cache.
Using a small C program has many attractive features:
  • A small C program can be checked for errors and no unexpected security issues
  • It is fast
  • It is freely available
  • It can be compiled for any operating system like AIX or Linux.
  • Many copies of the program can be concurrently run to scrub many disks at the same time and started with a simple shell script.
Using the DoD standard model for multiple passes of write and verify program:
  • Writes and reads 64 MB blocks for high performance.
  • Writes the buffer full of Hexadecimal 0x55 characters which in binary 10101010 to the whole disk.
  • Writes the buffer full of Hexadecimal 0xAA characters which in binary 01010101 to the whole disk.
  • Writes the buffer full of random characters to the whole disk.
  • Reads the whole disk back to check the random numbers were successfully written.
  • It leaves the disk full of random numbers.
Here is a small C program that I call: "nshred64.c"
This program has 100 lines of code. It is a code sample with no copyright, no warranty. See the download details in the Additional Information section:
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <strings.h>
#include <fcntl.h>

/* #define CORRUPT 1 */
#define BUFSIZE 64*1024*1024
char *buffer;
char *buffer2;

void write_file( char * filename)
{
    long writes=0;
    int fd;

    printf("write file %s blocksize=%d\n",filename, BUFSIZE);
    if( (fd = open(filename, O_WRONLY, 0)) == -1 ) {
        perror("failed to open file");
        exit(2);
    }
    while (    write(fd, buffer, BUFSIZE) == (size_t)BUFSIZE ) {
        writes++;
    }
    close(fd);
    printf("wrote %ld blocks\n",writes);
    sync();
}

void verify_file( char * filename)
{
    long reads=0;
    int fd;
    int j;

    printf("verify file %s blocksize=%d\n",filename, BUFSIZE);
    if( (fd = open(filename, O_RDONLY, 0)) == -1 ) {
        perror("failed to open file");
        exit(2);
    }
    while (    read(fd, buffer2, BUFSIZE) == (size_t)BUFSIZE ) {
#ifdef CORRUPT
        if(reads==4) buffer2[42]++; /* at block=4 byte=42 corrupt the data just read */
#endif
        for(j=0;j<BUFSIZE;j++) {
            if(buffer[j] != buffer2[j]) {
                printf("VERIFY FAILED at block=%d byte=%d as 0x%2x!=0x%2x\n",
                        reads, j, buffer[j], buffer2[j]);
                exit(24);
            }
        }
        reads++;
    }
    close(fd);
    printf("read %ld blocks\n",reads);
    sync();
}

void randomise()
{
int i;
    printf("prepare for phase random\n");
    for(i=0;i<BUFSIZE;i++)
        buffer[i]=(char)random();
}

void prepare(char ch)
{
int i;
    printf("prepare for phase 0x%02x\n",ch);
    for(i=0;i<BUFSIZE;i++)
        buffer[i]=ch;
}

int main(int argc, char** argv)
{
    buffer  = malloc(BUFSIZE);
    buffer2 = malloc(BUFSIZE);
    if(buffer == NULL || buffer2 == NULL) {
        printf("nshred malloc() failed\n");
        return 1;
    }
    srandom(getpid());
    if(argc != 2) {
        printf("nshred <disk-name>\n like nshred /dev/rhdisk999\n");
        printf("You must be the root user\n All disk data will be destroyed\n");
        exit(42);
    }
    if(strncmp(argv[1], "/dev/r",6)) {
        printf("nshred <disk-name> - disk name must start with /dev/r\n");
        exit(43);
    }
    prepare(0x55);
    write_file(argv[1]);
    prepare(0xAA);
    write_file(argv[1]);
    randomise();
    write_file(argv[1]);
    verify_file(argv[1]);

    return 0;
}
To compile is simple enough:
cc -O3 -o nshred64 nshred64.c
It takes under a second to compile. Do not forget to compile with optimization (-O3).

Note:
     To compile for the Linux operating system, remove the strncmp() check in the main() function. Linux has many weird disk names.

Here is an example run:
# lspv
hdisk0          00f6db0a6c7aece5                    None
hdisk1          00c0493045dbeb1e                    rootvg          active
#

# ./nshred64
nshred <disk-name>
 like nshred /dev/rhdisk999
 You must be the root user
 All disk data will be destroyed
# 

# time ./nshred64 /dev/rhdisk0
prepare for phase 0x55
write file /dev/rhdisk0 blocksize=67108864
wrote 512 blocks
prepare for phase 0xaa
write file /dev/rhdisk0 blocksize=67108864
wrote 512 blocks
prepare for phase random
write file /dev/rhdisk0 blocksize=67108864
wrote 512 blocks
verify file /dev/rhdisk0 blocksize=67108864
read 512 blocks

real    9m33.82s
user    0m7.72s
sys     0m4.47s
#
This example test used a 32 GB disk - so it is quick to complete. If your disks are much larger, it takes considerably longer to complete.
Small example of a failure:
prepare for phase 0x55
write file /dev/r1GBFILE blocksize=67108864
wrote 512 blocks
prepare for phase 0xaa
write file /dev/r1GBFILE blocksize=67108864
wrote 512 blocks
prepare for phase random
write file /dev/r1GBFILE blocksize=67108864
wrote 512 blocks
verify file /dev/r1GBFILE blocksize=67108864
VERIFY FAILED at block=4 byte=42 as 0x70!=0x71

 
The four phases require the reading or writing of 128 GB of data in total at an average of 228 MB per second.  So that would be about 75 minutes for a 1 TB disk.  This was 100% disk bound and uses less than 2% of a single POWER8 CPU.
Your disk speeds might be different - specially on older computers and disks.

Common Standards for overwriting the disk data
  1. NNSN 
    • 4-pass NNSA Policy Letter NAP-14.1-C (XVI-8) for sanitizing removable and nonremovable hard disks, which require overwriting all locations with a pseudo-random pattern twice and then with a known pattern: random(x2), 0x00, verify.
  2.  DoD   
    • 4-pass DoD 5220.22-M section 8-306 procedure (d) for sanitizing removable and nonremovable rigid disks, which require overwriting all addressable locations with a character, its complement, a random character, then verify.  NOTE: some performs the random pass first to make verification easier: random, 0x00, 0xff, verify.
  3. BSI   
    • 9-pass method recommended by the German Center of Security in Information Technologies (http://www.bsi.bund.de):  0xff, 0xfe, 0xfd, 0xfb, 0xf7, 0xef, 0xdf, 0xbf, 0x7f.
  4. Gutmann

Additional Information

Download

  • Source code and Makefile
  • Pre-compiled binary for 7.3 called nshred64
  • In .tar of .zip format
  • Files:  The files were moved to https://github.com/nigelargriffiths in the nshred repository.

Other places to find content from Nigel Griffiths IBM (retired)

Document Location

Worldwide

[{"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"SWG10","label":"AIX"},"Component":"","Platform":[{"code":"PF002","label":"AIX"}],"Version":"All Versions","Edition":"","Line of Business":{"code":"LOB08","label":"Cognitive Systems"}}]

Document Information

Modified date:
31 December 2023

UID

ibm11136086